@algernon @oblomov this aligns closely with my experience

So far I don't block on the absence of those specific headers because I want RSS readers to be able to get through. For the most part they should mainly be fetching the feed URL and maybe the site's favicon, but there are exceptions as you noted.

Some reader software will fetch arbitrary pages (at the user's request) and check for the existence of a <link rel="alternate"> tag. Since I strongly encourage readers to follow via RSS I'd hate to ban them when they try to do so 😅

@algernon @ansuz thanks, that looks exactly like what I needed. I think I have enough scrapers attacking me these days that I hopefully won't need other people's logs ;-)

@oblomov @ansuz I'm not an Apache person, but this module might do the trick.

I also have about a week's worth of logs from mid-April this year, iirc, with full headers, but I'll have to double check. The bots haven't changed much since. If that'd be useful for you, I'll go and figure out where I put them... they're somewhere on my storage server, just gotta find which bucket.

One of the custom emojis I throw out in work chats...

Only eight?

Sometimes, there's even gradation. Both the magnetic compass and invention of precision ship-grade clocks lurched us straight into the Modern World, on the same axis which is open sea navigation, just at different degrees.

And centuries later, GPS lurched us into an even moderner world. For now, the pre-GPS world is within the living memory, but soon, it will be just a bunch of quaint old-time problems that people used to work hard to wrestle with, like dead reckoning and the Longitude Problem used to be.

@algernon @ansuz that's useful information too, thanks. I'm actually considering collecting more information about the request headers in general to see if there's other subtle hints about them. Is there a way to tell apache to log all request headers for every request? At least while debugging it'd come in handy.

@oblomov @ansuz It's even easier than that, and most bots can be caught on the first request: if the user-agent contains Firefox/ or Chrome/, and you're serving on HTTPS, the request will1 contain a sec-fetch-mode header too, when coming from a real browser. Bots don't send it.

Pair it with blocking agents listed in ai.robots.txt, and ~90% of your bot traffic is gone. If you can afford to block Huawei's and Alibaba's ASNs, you pretty much got rid of all of them.

Many of the bots do download CSS, and some even fetch the JS too, by the way. And images? Some of them love 'em.

Exceptions apply: if you put a page in Reader Mode in Firefox, and reload while in reader mode, no sec-fetch-mode is sent. There are also some applications like gnome-podcasts that uses a Firefox user-agent, but doesn't send sec-fetch-mode. While there will be false positives, most of them can be worked around, and the gain of catching all the lame bots far outweights the cons, imo. ↩︎