01:16 <elibrokeit> Anubis by default assumes that all user agents which don't mention "Mozilla" aren't scrapers because scrapers always try to masquerade as interactive users :p

01:18 <elibrokeit> (and because many things need to work with wget/curl, like downloading raw/patch files in a distro integration, and git clone needs to work, etc etc)

01:26 <pinchartl> how long until scrapers maskerade as git or curl or something else? :(

07:16 <immibis> pinchartl: most sites block curl

07:20 <dwfreed> many sites that scrapers would genuinely be interested in tend to block curl so it's probably unlikely a scraper would use a curl user agent normally, but it may start using it for specific sites to bypass blocks

07:21 <dwfreed> especially if anubis continues its meteoric popularity rise as a defense against the scrapers

08:33 <Mithrandir> the logical conclusion is a payment system where you pay to download a page and the resources involved in generating and creating it.

09:54 <martink> heya. Getting an "gitlab.exceptions.GitlabHttpError: 403: insufficient_scope" while trying to bin/ci/ci_run_n_monitor.sh --pipeline --target here. What would be a good token scope for that?

09:55 <daniels> martink: api

09:57 <martink> daniels: works like a charm, thanks!

09:57 <daniels> martink: np :)

11:41 <mripard> n/bu21

18:34 <svuorela> are 'we' dos'ing canonical servers or is someone standing on the internet hose?

18:34 <svuorela> connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::102). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::101). - connect (101: Network is unreachable) Cannot

18:34 <svuorela> initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.81 80]

19:13 <pinchartl> Mithrandir: food for thoughts on the subject. I read a recent study that focussed on how aggressive blocking of aggressive crawlers meant that LLMs are increasingly trained on lower quality and predatory (as in being biased or entirely false for political goals) data, leading to increased dissemination of incorrect information

19:14 <pinchartl> this probably doesn't apply much to blocking access to gitlab.fdo (or at least not in a way that would hurt beyond wasting the time of LLM users), but is an issue with scientific papers

19:22 <elibrokeit> I have no problem with this lol

19:34 <pinchartl> with LLMs increasingly pushing conspiracy theories ? again that won't come from deploying anubis on gitlab.fdo, but it's an important side effect of blocking crawlers globally

19:35 <karolherbst> cloudflare now sells access for crawlers 🙃

19:35 <karolherbst> and blocks the ones not paying, which isn't a terrible idea as long as the actual websites get most of the money

19:36 <elibrokeit> I think that we're long past the point where LLMs will ever not be pushing conspiracy theories, at least making them more blatant might either cause people to smell something stinky or cause any resulting documents to be more easily dismissed

19:37 <karolherbst> twitter's LLM is already actively tempered with to push nonsense, soooo..

19:38 <karolherbst> side-effect of LLMs being pushed by the biggest assholes of the tech industry is that it's unavoidable that it's being tempered with

19:38 <elibrokeit> cloudflare a while back was majorly bragging about how instead of deploying mazes of junk data (iocaine, nepenthes, etc), they were explicitly deploying "mazes of useful data to protect your site without running the risk of poisoning the AI"

19:39 <karolherbst> and cloudflare is a good judge of what's "useful data" because?

19:39 <elibrokeit> "give the scraper something healthy to munch on"

19:39 <elibrokeit> well I think their theory was to just put random pages of generic facts

19:40 <karolherbst> right... it's a neat idea if you are willing to trust them

19:40 <karolherbst> the bigger issue here isn't about what "wrong" data they could show, but what "correct" data they won't

19:40 <Consolatis> <karolherbst> cloudflare now sells access for crawlers and blocks the ones not paying

19:41 <Consolatis> sounds more like an extortion scheme to me

19:41 <karolherbst> as long as it's extorting the AI crawlers I don't care 🙃

19:42 <karolherbst> the big issue is, that they were externalizing their costs by forcing everybody to pay more for servers

19:42 <karolherbst> if they now have to pay up for their access that only sounds fair

19:43 <pinchartl> karolherbst: "as long as the actual websites get most of the money" <- that would surprise me

19:43 <karolherbst> instead of deploying anubis, we could also just get 5 times the hardware

19:44 <karolherbst> pinchartl: doesn't even matter much, because it's cloudflare providing the hardware anyway

19:44 <karolherbst> maybe customers just pay less for traffic in the end

19:44 <karolherbst> or they choose smaller plans

19:44 <karolherbst> in the end doesn't really matter if it's a pay out or just spending less

19:44 <elibrokeit> > This pre-generated content is seamlessly integrated as hidden links on existing pages via our custom HTML transformation process, without disrupting the original structure or content of the page. Each generated page includes appropriate meta directives to protect SEO by preventing search engine indexing. We also ensured that these links remain invisible to human visitors

19:44 <elibrokeit> through carefully implemented attributes and styling. To further minimize the impact to regular visitors, we ensured that these links are presented only to suspected AI scrapers, while allowing legitimate users and verified crawlers to browse normally.

19:45 <elibrokeit> ^^ cloudflare's "irrelevant facts" labyrinth

19:45 <karolherbst> it does sound like something easy to get around tho

19:45 <elibrokeit> yeah I'm full of questions about this

19:46 <karolherbst> maybe they underestimate how much the companies are willing to do to get their crawlers to crawl

19:50 <karolherbst> maybe it's the new way of earning money instead of ads lol

21:37 <DemiMarie> The CI situation is really annoying, especially when PipeWire tests fail locally so I have no idea if my changes broke something or not.

21:38 <DemiMarie> that said, having the CI create an ephemeral VM for each job (which is what github.com and gitlab.com do) might take more resources than fd.o can afford

21:38 <DemiMarie> also, anything involving HW accell would still need to be gated

21:48 <daniels> DemiMarie: I’m not sure where you mean by ‘the CI situation’

21:49 <DemiMarie> daniels: I have a PipeWire MR (https://gitlab.freedesktop.org/pipewire/pipewire/-/merge_requests/2423) that I can't test locally because the test suite fails on my machine (on both my branch and git master)

21:50 <DemiMarie> usually what I do in this situation is push and let upstream CI run the tests in a known-working environment, but that only works when CI runs happen automatically

22:01 <whot> DemiMarie: you can pull down the container the CI runs in and run your tests in that

22:04 <DemiMarie> whot: what is the easiest way to find it?

22:04 <whot> DemiMarie: line 7 of the job output :)

22:05 <whot> Where it says "using docker image...", you can e.g. podman run -it registry.freedesktop.org/demimarie/pipewire/fedora/41:2025-05-10.0

22:06 <DemiMarie> whot: makes sense

22:13 <whot> DemiMarie: if you look at the libinput .gitlab-ci.yml, search for the build-in-vng@template, you can run the containers as a vm too

22:46 <DemiMarie> whot: how many people use this workflow?

22:48 <whot> virtme-ng is used elsewhere (gstreamer?) but I don't recall which projects specifically

23:07 <pinchartl> libcamera and linux-media both use virtme-ng in CI

23:08 <DemiMarie> Curious: is the reason that fd.o CI doesn't automatically spin up a VM for each run the need to use devices, resource constraints, or something else?

23:08 <DemiMarie> Or just that nobody has implemented it yet?

23:08 <whot> it takes a lot more resouces and for the vast majority of jobs its not needed

23:09 <DemiMarie> is this because the vast majority of jobs are by trusted people?

23:10 <whot> spinning up a container to run e.g. ruff on the code base is a matter of seconds, spinning up a vm a matter of minutes. add to that that log collection etc. from a VM is a lot more difficult because gitlab CI is designed for containers and you have an immediate bias towards containers

23:12 <whot> the *only* reason why libinput uses vms is becuase the test suite is {un}fortunately designed to use uinput devices so we need it. if I could get rid of that requirement I'd drop the vms instantly

23:15 <DemiMarie> whot: Does GitLab.com use VMs? I presume it does because it is a multi-tenant service with mutually distrusting users. If your VMs take minutes to start you have a misconfiguration somewhere.

23:30 <whot> tbh I don't know the exact setup gitlab.com uses internally but runners can be configured at various levels of distrust and ours is relatively permissive

23:36 <daniels> gitlab.com charges you for use

23:40 <elibrokeit> I expect that gitlab.com is running a fleet of VMs that you don't actually have insight into, and they restart them as fast as they can and all you see is when your job gets picked up by one

23:40 <elibrokeit> it's not like they assign you a VM and then boot the VM just for you