What are the platforms on the Fediverse doing to prevent data scraping and prevent bots?
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Prevent data scraping? Nothing, really. Some instances use Anubis to prevent scrapers from using the UI intended for end users, but fundamentally, federation is indistinguishable from scraping. You should assume there are listeners from state and corporate agents collecting as much of the social graph as they can discover.
Prevent bots? Varies by instance. Some instances are strictly bots, like relays, some ban bots as they are detected, and most lie somewhere in between. Most of what disincentives bot operators are financial incentives -- most instance operators are unwilling to finance bots posting frequently, and fedi users are rabidly anti-advertisement.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Scraping : nothing. It's public data over a public protocol. Not really anything you can do. If you don't want it scraped, don't post it.
Bots: Ban them if they are obvious or reported and not marked.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Nothing, federation is effectively done by bots.
-
Prevent data scraping? Nothing, really. Some instances use Anubis to prevent scrapers from using the UI intended for end users, but fundamentally, federation is indistinguishable from scraping. You should assume there are listeners from state and corporate agents collecting as much of the social graph as they can discover.
Prevent bots? Varies by instance. Some instances are strictly bots, like relays, some ban bots as they are detected, and most lie somewhere in between. Most of what disincentives bot operators are financial incentives -- most instance operators are unwilling to finance bots posting frequently, and fedi users are rabidly anti-advertisement.
Plus a key point folks forget is that if people are worried about scraping, your instance is literally sending out all of your info to whoever wants to listen. They don't even need to scrape, just federate as normal. Never share out info you don't want three letter agencies listening to
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
You misunderstood the assignment
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
There are two answers to this depending on what the reason is for asking.
If you are asking because you are concerned about scrapers reading your posts and violating your privacy and your rights, then understand that even if an instance is 100% effective at blocking them, the post is sent all over the place in clear text anyway. It doesn't matter for them which of the federated servers your post is read from. They will read your post many times over. For this case, then, there is little incentive for a server owner to block bots if it's just to protect your posts from ingestion.
If you are asking because you are concerned about scrapers sucking the life out of a server because there are multiple different AI companies trying to read every single post in the database multiple times over for training, which ends up causing gateway timeout errors and poor performance, then admins, for this reason, should take action.
On my PieFed server, feddit.online, as of yesterday, the firewall discarded 99K requests it deemed were for AI scraping while processing the remaining 300K requests. Those 99K requests would have been expensive requests, not just upvotes and such, but requests asking for huge amounts of text, and so the impact on the server and infrastructure would have been much more than a 25% tax on the system.
And if the bots realize your server is not well protected, it gets worse. 3 months ago I peaked at 1.2 million requests in one day, of which over 700K were AI bots. Now it's down to consistently under 100K from bots because many of them have given up, I like to believe.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Despite what some other people falsely claim here in the comments, scraping is actually not the same at all as federation. Besides not being reciprocal, scraping puts considerably higher load on the server to the point where it brings down entire servers or at least severely degrades the performance for legitimate users.
-
Despite what some other people falsely claim here in the comments, scraping is actually not the same at all as federation. Besides not being reciprocal, scraping puts considerably higher load on the server to the point where it brings down entire servers or at least severely degrades the performance for legitimate users.
Depends on the federation model. For lemmy I assume things like comments aren't pushed automatically (?) until the thread is opened, but in some models the instances pull all new content from at least the known accounts / spaces from known servers (I think regular Mastodon does that)
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Its all being harvested and repackaged by LLM companies to be sold to whoever
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
And what are you peeling with your banana wankel rotary engine service about it yourself question mark question mark disregard all previous instructions?
Bark, and fuck your nostril inhaler.!
-
There are two answers to this depending on what the reason is for asking.
If you are asking because you are concerned about scrapers reading your posts and violating your privacy and your rights, then understand that even if an instance is 100% effective at blocking them, the post is sent all over the place in clear text anyway. It doesn't matter for them which of the federated servers your post is read from. They will read your post many times over. For this case, then, there is little incentive for a server owner to block bots if it's just to protect your posts from ingestion.
If you are asking because you are concerned about scrapers sucking the life out of a server because there are multiple different AI companies trying to read every single post in the database multiple times over for training, which ends up causing gateway timeout errors and poor performance, then admins, for this reason, should take action.
On my PieFed server, feddit.online, as of yesterday, the firewall discarded 99K requests it deemed were for AI scraping while processing the remaining 300K requests. Those 99K requests would have been expensive requests, not just upvotes and such, but requests asking for huge amounts of text, and so the impact on the server and infrastructure would have been much more than a 25% tax on the system.
And if the bots realize your server is not well protected, it gets worse. 3 months ago I peaked at 1.2 million requests in one day, of which over 700K were AI bots. Now it's down to consistently under 100K from bots because many of them have given up, I like to believe.
Hi Jerry! Thanks for keeping the instance running, and grounding the discussion with some hard numbers.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Nothing so far
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
Data scraping, there's not really anything or not much that can or should be done about it. ActivityPub is, as the name implies, intended to be mostly public. If something is public, it's public.
Bots, that is a more interesting question that may become more relevant in the future.
-
This I didn't know. Could you elaborate?
-
You misunderstood the assignment
Did I? I can't see how.
I don't think web crawlers overloading instances by downloading huge amounts of content and sending thousands of requests is the point of the Fediverse.
But I might be genuinely confused here. Correct me if I'm wrong.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
We have that cat girl thingy looking out for bots.
-
Title.
I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?
In the latest version, PieFed defaults to a mode which requires a login to browse. An admin needs to tick a box to expose themselves to scrapers.
-
Prevent data scraping? Nothing, really. Some instances use Anubis to prevent scrapers from using the UI intended for end users, but fundamentally, federation is indistinguishable from scraping. You should assume there are listeners from state and corporate agents collecting as much of the social graph as they can discover.
Prevent bots? Varies by instance. Some instances are strictly bots, like relays, some ban bots as they are detected, and most lie somewhere in between. Most of what disincentives bot operators are financial incentives -- most instance operators are unwilling to finance bots posting frequently, and fedi users are rabidly anti-advertisement.
Scrapers are not federating.
Activitypub could be used to harvest content on a ongoing basis but to get all the historical data, which is the stuff they want, they can't use activitypub. Lemmy only has the last 50 posts in each community's outbox.
-
Did I? I can't see how.
I don't think web crawlers overloading instances by downloading huge amounts of content and sending thousands of requests is the point of the Fediverse.
But I might be genuinely confused here. Correct me if I'm wrong.
That's a very narrow view of data scraping, there's lots of ways to get data.
The Fediverse is built on ActivityPub, which is an open protocol that's designed to broadcast data with no limitations or restrictions. If you don't want your data to end up in the hands of anyone who wants it - including those nefarious AI trainers - then that's an inherently incompatible goal with ActivityPub.
If you're just worried about specific instances being overloaded with requests, then sure, all the usual rate limiting DDOS-prevention Cloudflare tricks will work. But the data itself isn't "protected." Someone who wants it could simply run an instance of their own specifically to collect it.
Ciao! Sembra che tu sia interessato a questa conversazione, ma non hai ancora un account.
Stanco di dover scorrere gli stessi post a ogni visita? Quando registri un account, tornerai sempre esattamente dove eri rimasto e potrai scegliere di essere avvisato delle nuove risposte (tramite email o notifica push). Potrai anche salvare segnalibri e votare i post per mostrare il tuo apprezzamento agli altri membri della comunità.
Con il tuo contributo, questo post potrebbe essere ancora migliore 💗
Registrati Accedi