AI scrapers everywhere

submitted by
[any] edited

There’s a nice article: The Day I Logged 1 In Every 2000 Public IPv4: Visualizing The AI Scraper DDoS which someone kindly dropped in the fuck_ai community. It’s worth reading.

After reading it, I decided to have another look at my logfiles and I’m pretty sure I see the same thing. A good while back I had my small 1-3 people instance nearly melt because of Tencent and Alibaba. I managed to address that by blocking some of their IP ranges. Proceedingly rimu implemented a Honeypot, which currently has 14.272 IPs blocked on my instance. And a few database requests got more efficient. But I think the overall situation deteriorated. These days the AI scrapers seem to come from everywhere on the internet.

My logfiles just scroll by again. It’s not as bad as when I was hit by the really nasty crawlers. The server keeps up with it, I’m just constantly sending out loads of data. There’s a crawler getting caught up in the honeypot almost every minute. But there’s also still a near infinite amount of addresses left, querying all kinds of posts, communities and user profiles. And I notice since it’s a dysproportionate amount of traffic for a small instance.

I think the honeypot is great. But the AI scrapers have way to many addresses. And in the mid-term we probably need to come up with some more mitigations. At least to cater to smaller servers.

Just wanted to draw some attention to how things change. I’ve switched my instance to “private” for now, but I’ll continue to investigate.

I’ve made two images myself (for palaver.p3x.de). First is all IPv4 addresses in my nginx log. That includes all Fedi instances and users. But I’m just a small instance and I guess most of the reddish areas are caused by crawlers.

AJvKYRbM1j1amZm.png

Second image is what’s in my honeypot, (dots visible after zooming in):

1y7biwvTHxw9Q8q.png

6
8

Log in to comment

Hot Top New Old
he/him edited

For single-user instances, where the only user will be logged in continually, setting it to private is really easy and comes with basically no downside. Even if you want to provide anonymous access, often going private for a day or two is all it takes to shake the scrapers so you can open up again later. I did this on piefed.social last week.

Also I found many many scrapers are using quite old Chrome versions (or their user agents use randomly generated version numbers) so blocking those at the web server level pwns 90% of that family of scrapers. e.g. on nginx you’d put this in your server block: 

    # Scrapers often use old user-agents  
    if ($http_user_agent ~* "^(?!.*(Googlebot|bingbot)).*(Chrome/([0-9]{1,2}|10[0-9]|11[0-9])\b|Firefox/([0-9]{1,2}|10[0-9]|11[0-4])\b)") {  
        return 403;  
    }  
    # Block dead browsers / engines  
    if ($http_user_agent ~* "(Opera/[0-9]|Presto/|MSIE|Trident/)") {  
        return 403;  
    }  

    if ($http_user_agent ~* "(Sogou web spider|SERankingBacklinksBot|WF search)") {  
        return 403;  
    }

If using Cloudflare you can have it present a “are you a human” challenge to big chunks of the address space using AS numbers. I have AS45899 blocked, which gets rid of most of Vietnam and I’ve also got Country = China as a criteria. That helps a lot.

 reply
3
[any] OP edited

Thanks. I already had an older version of that in my config. I’d say from looking at the logs, at times it catches every 10th crawler or so.

I blocked most nasty address ranges from AS45899 yesterday, as well.

I’m living the Cloudflare -free lifestyle 😅. I got geoip2 running, though. Not sure if I want to hand out some collective punishment against the Chinese. Looking at my honeypot with Tencent and Alibaba blocked, they don’t even make the cut with the single digit amount of addresses blocked. Main offenders are: BR (2120), US (864), IQ (805), IN (555), BD (535)… At least as of right now. I guess in a few weeks we might be looking at a different situation anyway.

Edit: And I’m still being bombarded 12h after switching the instance to ‘private’. They seem to move towards the login page now, with the next argument set as in the 302 forwards. But they seem to also be still working on a large backlog of regular URLs to crawl. And it’s still multiple requests a second. I’ll look at how they’re going to adapt.

 reply
1
[any] OP

Update: It’s calm again. I’ve reverted to public instance. But I’ll send a 402 “Payment required” to browser agents who claim to use Windows or Mac OS X. People should be using Linux anyway 🙃

It’s hard to tell what’s going on here. I thought maybe it’s a good sign the scrapers adapt in some way, we might be able to trigger that behaviour in some other ways. But it’s completely unclear to me if something like a more aggressive honeypot will be enough to pull it off.

 reply
1

Requiring a login to view the site like you have done is extremely effective, but it is a big tradeoff.

in the mid-term we probably need to come up with some more mitigations

I know that some PieFed instances have deployed Anubis successfully (like quokk.au), but, as effective as Anubis might be for most scrapers…it has never really felt like a great solution to me. The whole proof of work model is basically just burning CPU cycles to prove you are human, which doesn’t feel great.

However, I don’t have a better solution either. Maybe the easiest path forward for now would be to work on documenting Anubis setup instructions in the official docs?

Are there other solutions out there to pull from? Web dev isn’t really my jam, so it isn’t something I keep up with new developments, and stuff can move quickly.

 reply
2
[any] OP edited

The whole proof of work model is basically just burning CPU cycles to prove you are human, which doesn’t feel great.

Hmmh. I mean the AI industry doesn’t care at all about burning CPU cycles. Or wasting massive network resources to scrape some random garbage and come back a few minutes later. So I don’t really know if a proof of work is the correct approach here. I don’t think that’s a long-term deterrent against someone like them who doesn’t even care about the CPU cycles.

 reply
2

Yeah, agreed. I think the main way that Anubis has been effective is that it requires the ability to run JS, which a lot of scrapers don’t have (yet).

 reply
2
Insert image