Any protection against AI Crawlers and AI learning bots?
-
The forum has the robot.txt which is a basic protection against search bots, but AI bots are still a threat, so maybe someone can make a plugin or devs can integrated that into the next versions of the Forum? NodeBB joining Fediverse is excellent and I'm using NodeBB Forums for the exactly reason, I understand that project is like in pre-alpha but suggesting new things to be implemented is also a good thing.
-
The forum has the robot.txt which is a basic protection against search bots, but AI bots are still a threat, so maybe someone can make a plugin or devs can integrated that into the next versions of the Forum? NodeBB joining Fediverse is excellent and I'm using NodeBB Forums for the exactly reason, I understand that project is like in pre-alpha but suggesting new things to be implemented is also a good thing.
Shaknunic here's the catch-22... If you want to stop AI crawlers, you can stop the good ones using
robots.txthttps://github.com/ai-robots-txt/ai.robots.txt
If you want to stop the bad ones (which you probably do)... you can't
They masquerade as legitimate users, and they have more funding behind them than you have time. Every time you try to block them someone on the other end will adapt.
Unfortunately right now the only reliable solution is a hard challenge from CloudFlare for nearly all requests.
-
As a solution, the link you suggested me kinda helped, but not really... My alternative solution for this was to implement Anubis anti-crawler, since I don't trust Cloudflare that much... My Forum is not on Fediverse yet and I didn't share with anyone and still I was receiving high numbers of clicks and inconsistent unique visitors... I told robots.txt to block every bot since well I'm still building the Forum and still I received one bot in there...
Hopefully with Anubis the problem can be at least minimized, but for anyone having a Forum... I suggest search for how to implement an anti-bot yourself since otherwise your bandwidth is going to cry for help. I implemented Anubis just today successfully, but I'll be monitoring the traffic of the site either way...
Gli ultimi otto messaggi ricevuti dalla Federazione
-
As a solution, the link you suggested me kinda helped, but not really... My alternative solution for this was to implement Anubis anti-crawler, since I don't trust Cloudflare that much... My Forum is not on Fediverse yet and I didn't share with anyone and still I was receiving high numbers of clicks and inconsistent unique visitors... I told robots.txt to block every bot since well I'm still building the Forum and still I received one bot in there...
Hopefully with Anubis the problem can be at least minimized, but for anyone having a Forum... I suggest search for how to implement an anti-bot yourself since otherwise your bandwidth is going to cry for help. I implemented Anubis just today successfully, but I'll be monitoring the traffic of the site either way...
-
Shaknunic here's the catch-22... If you want to stop AI crawlers, you can stop the good ones using robots.txt
https://github.com/ai-robots-txt/ai.robots.txt
If you want to stop the bad ones (which you probably do)... you can't
They masquerade as legitimate users, and they have more funding behind them than you have time. Every time you try to block them someone on the other end will adapt.
Unfortunately right now the only reliable solution is a hard challenge from CloudFlare for nearly all requests.
-
The forum has the robot.txt which is a basic protection against search bots, but AI bots are still a threat, so maybe someone can make a plugin or devs can integrated that into the next versions of the Forum? NodeBB joining Fediverse is excellent and I'm using NodeBB Forums for the exactly reason, I understand that project is like in pre-alpha but suggesting new things to be implemented is also a good thing.