08:45 <eric_engestrom> bentiss, daniels, and anyone with an internet-accessible server: turns out there's a market of malware turning everyone's phone into a botnet, which app developers are *voluntarily* adding to their apps in exchange for money, literally selling their users' phones: https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/

08:46 <eric_engestrom> this explains the way we get attacked by tons of regular residential IPs

08:53 <daniels> yeah, and you can opt in to extensions or apps which add your home connection to AI scraping

08:54 <daniels> so using IP as an indicator of actual data source is pretty meaningless these days

08:58 <eric_engestrom> yeah, but to me it's not just about knowing the source, but the fact it has become literally impossible to block attacks (at a network level) without blocking all users

08:59 <eric_engestrom> app-level blocks like Anubis are now becoming mandatory for any internet-accessible service

09:00 <eric_engestrom> and that makes me angry & sad

09:00 <daniels> ah, there we go https://proxyway.com/reviews/massive-proxies

09:01 <daniels> yeah, it’s pretty fucked up

13:45 <karolherbst> one could try to block their command&control stuff, but.. one would need to do it in every network...

15:57 <pinchartl> eric_engestrom: isn't it legitimate to block users who install those apps though ?

15:58 <pinchartl> or who install such appliances

15:59 <eric_engestrom> pinchartl: IMO the user is a victim, having their devices hijacked for malicious purposes; the guilty ones are the devs

16:00 <pinchartl> to some extent yes, but the users need to understand that they have a role to play

16:01 <pinchartl> certainly not ideal of course

16:01 <pinchartl> but these days, I think I'd consider legitimate to block an residential IP if it for instance an ring or alexa device

16:02 <pinchartl> s/an ring/a ring/

16:02 <pinchartl> s/an residential/a residential/

16:02 <pinchartl> (there seem to be a big supply of free N's today)

16:04 <eric_engestrom> I agree to some extent, but banning them will not educate them about installing these apps, they'll never know

16:04 <eric_engestrom> at most (if they even notice) they'll be annoyed at the service admins for banning them

16:05 <pinchartl> yes, it's not ideal

16:05 <pinchartl> they should at least be redirected to a page that explains the problem

16:07 <alanc> do such users even have static IP addresses or just whatever was next in the DHCP pool when their phone or cable/dsl modem connected?

16:07 <eric_engestrom> alanc: +1

16:08 <pinchartl> it won't solve our immediate problem, but long term someone will need to do something about educating people

16:08 <pinchartl> (I know, that's a daunting task, and it's not getting better)

16:08 <eric_engestrom> pinchartl: you'll never know for sure if the user has one of those apps, and unless you can give them a list of known ones, the most that page can have is a generic "don't install apps you don't _need_" which will likely just be ignored (but maybe I'm just too jaded :)

16:11 <pinchartl> so the alternative is to send someone at their address who will confiscate their phone and computers, ... ok, maybe not :-)

16:12 <pinchartl> I wonder how law enforcement agencies and courts handle this situation, now that everybody effectively gets plausible deniability for their online actions

16:14 <eric_engestrom> +1 for educating people though, but IMO what's really needed is consequences for bad behaviour, and right now they're doing it out in the open (as shown in the links above) because there are no consequences for them (the botnet devs and the app devs)

16:15 <eric_engestrom> (I'm on a train, my connection is very flaky, I see you brought up law enforcement as well, before my message sent ^^)

16:16 <daniels> yeah, you either have dynamic IPs or CGNAT, so you can’t just wipe the IPs out

16:16 <daniels> else you’re nailing everyone who shares a provider with people who have installed bad mobile games, which is most people

16:16 <pinchartl> sometimes it's hard to resist the temptation to educate people with a baseball bat :(

16:23 <__tim> Not sure apache2 on annarchy is well, getting 'connection refused' for http(s)

16:47 <jrayhawk> Something pushes kemper's kernel into a bad state where I/O scheduling dies, which then gets a bunch of annarchy and molly process stuck in D-state

16:55 <jrayhawk> in the pathological state, iostat -x 1 rkB/s maxes out at, like, 200

16:59 <__tim> I'm currently doing some uploads to S3 fwiw, but you hope that wouldn't be overly problematic

16:59 <__tim> (and it didn't seem to recover when I stopped it earlier)

17:04 <jrayhawk> i am guessing a filesystem data structure has grown unmanageable in the 12 years of that filesystem's existence and the filesystem should be recreated

18:32 <karolherbst> pinchartl: how do you determine such users? Also their IPs are random

18:32 <karolherbst> residential and mobile IPs aren't fixed, so what good would it do to block them?

18:32 <karolherbst> could just as well block all ISPs

18:32 <karolherbst> or random IPs, same result

18:32 <pinchartl> block 0.0.0.0/0, problem solved :-)

18:33 <karolherbst> well that was your suggestion :P

18:33 <pinchartl> it wasn't a fully serious suggestion

18:33 <karolherbst> the business model works, because you can't do anything about it really

18:34 <pinchartl> (although, when it comes to ring or alexa devices... but that's a different story)

18:34 <karolherbst> I think what would help is to create a list of all those command and control servers and just ask ISPs to block those, or have a shared blocklist for those everybody is free to use

21:33 <DemiMarie> karolherbst: personally I prefer an enforced ban on running a residential proxy system. The motivation is financial, and as soon as money is changing hands, regulation is much easier.

21:34 <karolherbst> DemiMarie: they use IPs from random users

21:34 <karolherbst> the entire point of the business model is that you can't ban them without causing fallout

21:34 <DemiMarie> karolherbst: I was referring to governments prosecuting those who run the residential proxy networks

21:35 <karolherbst> well. then they move countries or something funky

21:35 <karolherbst> also.. try to proof that they are indeed causing problems

21:35 <DemiMarie> karolherbst: kick them out of the financial system

21:35 <DemiMarie> the reason app developers use them is that they get paid, so stop them from paying the app developers

21:35 <karolherbst> need a judge for that normally or regulation

21:35 <DemiMarie> exactly

21:36 <karolherbst> and even with regulation you have to proof wrong doing

21:36 <DemiMarie> this needs to be dealt with by governments

21:36 <karolherbst> you can't prove anything here is the point

21:36 <karolherbst> they can just say "nah, we don't cause the AI scrapping overloading everybody's system"

21:37 <DemiMarie> you could assume that any app making requests to something the user didn’t ask it to, and which is not related to the legitimate publisher of the app, is malicious

21:37 <karolherbst> of course you could outlaw this specific business model, but then you need to discuss this with politicians so they udnerstand

21:37 <karolherbst> hah

21:37 <karolherbst> can't prove that either

21:37 <karolherbst> given that all those apps are normally full of ad stuff

21:38 <DemiMarie> Put “All ad requests must be proxied through a host belonging to the legitimate publisher,” in the app store policies.

21:38 <karolherbst> discuss it with google and apple then :P

21:38 <DemiMarie> Or governments could just ban digital advertising (only half-joking)

21:39 <karolherbst> but yeah....

21:39 <karolherbst> apps having to declare what IPs they are accessing might be one solution

21:39 <karolherbst> but that's tough for certain kind of apps to solve

21:39 <DemiMarie> have the OS prompt the user when the app wants to establish a connection to something it hasn’t connected to before

21:40 <DemiMarie> unless it is a browser/SSH client/etc

21:40 <karolherbst> but who says people don't do it in desktop games as well :P

21:40 <karolherbst> or random other applications

21:40 <DemiMarie> and then require those apps to not include advertising

21:40 <karolherbst> or random websites

21:40 <DemiMarie> but really this needs to be dealt with by government action

21:40 <karolherbst> sure

21:40 <DemiMarie> if using these proxy networks was considered a CFAA violation they would go away pretty quickly

21:40 <karolherbst> usually need big players to lobby for that otherwise not much will happen or it will take years

21:41 <karolherbst> yeah well..

21:41 <karolherbst> I consider the CFAA to be pretty much toast these days :P

21:42 <karolherbst> but then again.. maybe it's illegal in the US, maybe you get it outlawed in europe.. and then you have random other places where it's not

22:04 <DemiMarie> I really wish that proof of work could be made into a native browser feature, so that JavaScript would not be needed.

22:16 <karolherbst> I think the ship has sailed and given the mess authentication has become in HTTP, it's probably for the best not adding any more stuff as headers

22:57 <Consolatis> POW wouldn't really solve the issue with residential botnets. i would also look at it from another perspective, if something is on the internet and accessible without some kind of authorization like an accounts you have to expect to get requests for it. if that causes issues like with heavy dynamic pages then those pages should require a user account or session token from some captcha (preferably not some unsolvable google one) or POW system

23:40 <kode54> POW forces whatever is fetching the page to process commands before they get the actual data

23:41 <dwfreed> and also requires them to have functional js

23:41 <dwfreed> which is less of a barrier these days than it used to be, but a barrier nonetheless

23:41 <kode54> it's currently only doing it to devices that appear to be browsers that would have js

23:42 <dwfreed> a POW that takes a second per page load is only a minor annoyance to a human, but a huge slowdown to a bot

23:42 <kode54> that's the whole benefit

23:42 <kode54> which is a good one at slowing down scrapers

23:42 <dwfreed> aye, I'm agreeing with you :)

23:42 <kode54> I know :]

23:43 <kode54> making the browser automate it will just add another native way to bypass the fix

23:44 <dwfreed> imo the "js should not be required" ship sailed a long time ago

23:44 <dwfreed> most "modern" sites on the web are not useful without js

23:45 <dwfreed> (for better or worse)