Snapshot of the load ap.space sees from AI crawlers
Meta
1
Posts
1
Posters
3
Views
-
Can you guess when I turned Anubis back on?
- Grey line (left-hand; y-axis) tracks page views
- Blue line (right-hand; y-axis) tracks unique users
You can even see the spike in traffic that brought down the site hard enough that I got my butt in gear to tune Anubis and turn it back on.
Based on the numbers here, there is a thirteen-fold decrease in activity (or a ~92% drop in traffic), all identified by Anubis as bots and blocked.
Selective adjustments were made to the nginx config and anubis bot policy to allow certain bits of traffic through (for unimpeded federation, etc.), but otherwise the site is now quite stable, on a small potato server. :potato:
Default bot policy does let search engine crawlers though (I think), so that is win-win.
Thank you so much @cadey@pony.social, my next stop is your GitHub Sponsors page.
-
undefined nodebb@fosstodon.org shared this topic on
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Can you guess when I turned Anubis back on?
Grey line (left-hand; y-axis) tracks page views Blue line (right-hand; y-axis) tracks unique usersYou can even see the spike in traffic that brought down the site hard enough that I got my butt in gear to tune Anubis and turn it back on.
Based on the numbers here, there is a thirteen-fold decrease in activity (or a ~92% drop in traffic), all identified by Anubis as bots and blocked.
Selective adjustments were made to the nginx config and anubis bot policy to allow certain bits of traffic through (for unimpeded federation, etc.), but otherwise the site is now quite stable, on a small potato server. :potato:
Default bot policy does let search engine crawlers though (I think), so that is win-win.
Thank you so much @cadey@pony.social, my next stop is your GitHub Sponsors page.
-
16 January 2026
Re-enabled Anubis after adjusting some rules. After the last update 17 Nov, I actually disabled Anubis again because it was interfering with logins (perhaps this is a known issue? Anyhow, proxying POST requests directly to NodeBB fixed the issue)
-
17 November 2025
Re-enabled Anubis with a blanket rule that allows requests to /inbox; hopefully that should resolve any issues.
-
12 November 2025
Per report trwnh@mastodon.social about some content missing from mastodon.social, I have disabled anubis temporarily as it may have been inadvertently blocked by it.
-
7 November 2025
Enabled Anubis in response to AI scrapers overwhelming the server multiple times a dayPlease let me know if you experience any federation delays or other oddities!
-
21 October 2025
Added fedimemes@feddit.uk to the Related Communities section :smiling_imp:
-
Thank you so much for the kind words 😎 it's been a busy year and a half but integrating ActivityPub has definitely been a step in the right direction.
There are still some hairy bits here and there but "it just works" is definitely the goal I'm shooting for.
-
... I just have to say how much I am enjoying the NodeBB user interface. This is a really pleasant piece of software, and it seems to Just Work on the Fediverse.
