sysadmins/webmasters of fedi:
Uncategorized
38
Posts
13
Posters
60
Views
-
sysadmins/webmasters of fedi:
I am looking for suggestions of which search engine crawlers I should consider permitting in my robots.txt file.
There can definitely be value in having a site indexed by a search engine, but I would like to deliberately exclude all of those which are using the same data to train LLMs and other genAI. More specifically, I would only like to allow those which have an explicit stance against training on others data in this fashion.
Currently I reject everything other than Marginalia (https://marginalia-search.com/). Are there any others I should consider?
@ansuz If Marginalia is already there then:
https://alpha.mwmbl.org
https://stract.com -
@ColinTheMathmo oh yea, a lot of them have very clearly had way too much silicon-valley-kool-aid to drink and were basically incoherent.
I was actually pleasantly surprised at a few others, though, I can't help but feel that the more sensible ones might be wasting their time commenting on HN and should just join fedi
@ansuz @ColinTheMathmo this has been my experience also in the rare cases when my writing gets shared on HN (or lobste.rs or reddit, FWIW): there a bunch of ignorami who are full-in on the corporate propaganda, and a few sane voices that try to highlight how bad that is for everyone. 1/2
-
@ansuz @ColinTheMathmo this has been my experience also in the rare cases when my writing gets shared on HN (or lobste.rs or reddit, FWIW): there a bunch of ignorami who are full-in on the corporate propaganda, and a few sane voices that try to highlight how bad that is for everyone. 1/2
On the one hand, I agree that they're mostly wasting their time and would do better to spend it elsewhere; OTOH, I am glad for the work they do, and who knows, maybe their words reach others that can then do better.
2/2
-
@ColinTheMathmo oh yea, a lot of them have very clearly had way too much silicon-valley-kool-aid to drink and were basically incoherent.
I was actually pleasantly surprised at a few others, though, I can't help but feel that the more sensible ones might be wasting their time commenting on HN and should just join fedi
@ansuz responding to someone on hacker news that they're too good for HN and should join fedi would be absolutely hilarious to no one but me
-
@ansuz responding to someone on hacker news that they're too good for HN and should join fedi would be absolutely hilarious to no one but me
@rune I might have tried it if I had an account there, but then imagine if I was wrong and that was their one good take. fedi equivalent of buyer's remorse
-
I just published a blog post summing up my most pertinent thoughts about dealing with badly-behaved web-scraping bots:
https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/
It isn't exactly a Hallowe'en-themed article, but today is the 31st and the topic is concerned with pranking people who come knocking on my website's ports, so it's somewhat appropriate.
#infosec #bots #halloween #scrapers #AI #someMoreHashtagsHere
One of the people that read my article on algorithmic sabotage set up an infinite source of nonsense for LLM scrapers to ingest:
It uses txtgen (https://ndaidong.github.io/txtgen/) to respond to every subdomain and URL with garbage.
There are already other projects to do the same, but it's nice to see more people trying their hand at addressing the problem.
-
looks like someone reshared my article to Hacker News where someone has (predictably) already commented on the headline without reading the article 😅
I wrote up some commentary about the response to my last article:
"Algorithmic sabotage debrief"
-
I wrote up some commentary about the response to my last article:
"Algorithmic sabotage debrief"
@ansuz I love this, thank you for writing it!
You talk about how people just block IP addresses. Yup, that's me.
I'm blocking /16s at this point, as well as a couple /8s, and many /13s and /14s.
1.5 billion requests over a 24 hour period is not okay. I'd already blocked one offender subnet, when they stepped up their attempts to the billion range. They tried for a solid week. Then they spent 3 months at a rate of 30K per 24 hours, before wandering off to bug someone else. 🙄 -
One of the people that read my article on algorithmic sabotage set up an infinite source of nonsense for LLM scrapers to ingest:
It uses txtgen (https://ndaidong.github.io/txtgen/) to respond to every subdomain and URL with garbage.
There are already other projects to do the same, but it's nice to see more people trying their hand at addressing the problem.
Somebody posted a Korean-language summary of my "AI scrapers request commented scripts" with some commentary:
https://aisparkup.com/posts/6165
An excerpt, translated via Firefox's local translation models because I can't read Korean:
> Is it self-defense against unauthorized collection of data under the guise of ignoring robots.txt, or is it excessive retaliation? The fact that an AI model that costs billions of dollars can be disabled with a few hundred documents is a pleasant reversal for some, but a worrying situation for others.
They also mention the WWE/Kropotkin tarpit described by @jonny:
> Another is said to have created a contaminant data with text created by mixing the WWE script with Kropotkin’s “inter-infrastructure theory.”
-
Somebody posted a Korean-language summary of my "AI scrapers request commented scripts" with some commentary:
https://aisparkup.com/posts/6165
An excerpt, translated via Firefox's local translation models because I can't read Korean:
> Is it self-defense against unauthorized collection of data under the guise of ignoring robots.txt, or is it excessive retaliation? The fact that an AI model that costs billions of dollars can be disabled with a few hundred documents is a pleasant reversal for some, but a worrying situation for others.
They also mention the WWE/Kropotkin tarpit described by @jonny:
> Another is said to have created a contaminant data with text created by mixing the WWE script with Kropotkin’s “inter-infrastructure theory.”
I'm trying to work on the fluconf website, but I keep getting nerd-sniped by the behaviour of the scrapers that are already hitting the not-yet-officially-public site.
In my follow-up to the AI-scrapers article I mentioned that there are actors who monitor certificate transparency logs for new domains to crawl/probe, but I've never really paid close attention to the resources that get targeted first.
It's hard to know for sure because they spoof their user-agent strings and do their best to look like legitimate traffic, but it looks like OpenAI, Nomic.ai, and some other minor AI companies are using this technique.
-
Somebody posted a Korean-language summary of my "AI scrapers request commented scripts" with some commentary:
https://aisparkup.com/posts/6165
An excerpt, translated via Firefox's local translation models because I can't read Korean:
> Is it self-defense against unauthorized collection of data under the guise of ignoring robots.txt, or is it excessive retaliation? The fact that an AI model that costs billions of dollars can be disabled with a few hundred documents is a pleasant reversal for some, but a worrying situation for others.
They also mention the WWE/Kropotkin tarpit described by @jonny:
> Another is said to have created a contaminant data with text created by mixing the WWE script with Kropotkin’s “inter-infrastructure theory.”
I'm still seeing little waves of traffic from my AI scrapers article, now via this Polish tech newsletter: https://news.mrugalski.pl/post/800814241829568512/21-listopada-2025
Feed RSS
Gli ultimi otto messaggi ricevuti dalla Federazione
-
Credo di aver incontrato la cugina di #Sphalman: #AscellPuzz ! 😜🤪😅
-
@lorenzofb I’ll bet you could spot them if you just go to Barcelona and walk around. Follow the buzzcuts and the hoodies 😉
-
Ho appena pubblicato il quinto episodio dell'audiolibro di Assalto alle piattaforme!
https://podcast.kenobit.it/@assaltoallepiattaforme
In questa puntata si parla diffusamente di Mastodon e Fediverso.
-
@lorenzofb It may be a “we don’t have a venue yet so how do we announce the location” situation. (Knowing nothing at all about that particular conference, but being familiar with the issues many conferences have.)
-
@DigiDavidex Here is your #FediTree for the year 2025.
3 accounts were omitted due to #nobot tags or discoverability settings.
-
-
Guido Crosetto denuncia i pacifisti. Più facile e meno rischioso che ribellarsi al regime di Netanyahu.
S01E18: Guido Crosetto denuncia i pacifisti. Più facile e meno rischioso che ribellarsi al regime di Netanyahu.
Una puntata in cui un ministro della repubblica italiana, invece di rispettare i tribunali, i cittadini e il diritto internazionale, si mette a chiamare "schifosi" da una festa di partito i cittadini che hanno sporto denuncia alla Corte Penale Internazionale per non aver ostacolato il genocidio della popolazione palestinese certificato dalle Nazioni Unite.
-
@fosdembsd hello!!!
