Piero Bosio Social Web Site Personale

@ansuz @oblomov Looking at my logs, most RSS readers are unaffected: they either use their own user agent, and don't try to pretend to be Firefox or Chrome, or they are running within a real browser, in which case the expected headers will be there.

Quick look at my logs from yesterday:

• 2582 total requests against atom.xml on my blog.

• 105 unique user agents

• Only 24 of those user agents had Chrome/ or Firefox/ in their user agent

• These 24 made 165 requests total.

• Out of that 165, 54 did not have sec-fetch-mode.

• Out of those 54, the majority came from either Cloudflare or Amazon, or another cloud provider.

• Still out of those 54, 21 pretended to be Firefox, but the user agent wasn't what a real browser sends: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0 - in real browsers, rv and the Firefox/ version match. (All 21 were from Cloudflare IPs too)

• Still out of the 54, 27 pretended to be Chrome, but did not send a sec-ch-ua header, nor sec-fetch-mode, and they said they're Chrome/84.0.4147.105 from 2020 - coming from Amazon AWS. I don't believe for a second these would be real browsers.

This leaves us with 6 requests that may have come from legit browsers. Five of those were Chrome on Android, coming from a DigitalOcean IP, without sec-fetch-mode or sec-ch-ua. I don't think those were legit.

There was one Firefox/, coming from an American residential IP, without sec-fetch-mode... that might have been legit, maybe?

But out of 2.5k requests, 1 false positive¹ is, imo, acceptable.

Of course, what's acceptable varies a lot, and the people who visit (or rather, subscribe to) my blog are likely a bit atypical.

What I'm trying to convey here is that the majority of RSS readers don't pretend to be Firefox or Chrome, or - because they're running in one - send the appropriate headers anyway.

@oblomov @ansuz I'm not an Apache person, but this module might do the trick.

I also have about a week's worth of logs from mid-April this year, iirc, with full headers, but I'll have to double check. The bots haven't changed much since. If that'd be useful for you, I'll go and figure out where I put them... they're somewhere on my storage server, just gotta find which bucket.

@oblomov @ansuz It's even easier than that, and most bots can be caught on the first request: if the user-agent contains Firefox/ or Chrome/, and you're serving on HTTPS, the request will¹ contain a sec-fetch-mode header too, when coming from a real browser. Bots don't send it.

Pair it with blocking agents listed in ai.robots.txt, and ~90% of your bot traffic is gone. If you can afford to block Huawei's and Alibaba's ASNs, you pretty much got rid of all of them.

Many of the bots do download CSS, and some even fetch the JS too, by the way. And images? Some of them love 'em.