AI-assisted moderation in the fediverse is happening. Now what?
-
You're a fucking AnCap? That explains soooooo much.
Wait, why do you think Rimu's an AnCap?
-
You're a fucking AnCap? That explains soooooo much.
Maybe you should read that post before commenting. It's anti-AnCap at its heart.
-
I'm not sure that is true. As far as I know, GDPR only covers personally identifiable information, if the data is completely anonymous, and can not be traced back to your person, it is fine to use under the GDPR. This is a common misconception about the GDPR.
So the real question is if your comments can be used to identify you personally. This might be the case, or it might not.
If they anonymize the data, don't share usernames etc and only the messages then you are certainly right.
-
It's occasionally worth calling out that votes are also public. I think twice before hitting those buttons
https://lemvotes.org/comment/lemmy.world/comment/23550342
here's your comment
-
Yes, but then you are willingly accepting the risk of posting in a fully public forum anyway. What I'm saying is, you could, if you wanted, not have personally identifiable information on your account.
That's true, but this person also knows they are not hiding. There are countless others that don't. That's the reason they wrote what they wrote.
-
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“.
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
Aside from the ethical implications of profiling users or of using a corporatly owned server and model to execute this, I see nothing uniquely concerning about this practice that isnt already a risk of federated social media generally.
Every mod on every instance is free to use whatever tools or standards for moderation they want - that's an intentional byproduct of federation. Similarly, the collection of this data for use with llms is a bygone conclusion at this point - there was never any way of preventing that from happening with a federated network.
I think the only thing here to talk about is the way these questions are being framed as a question of intra-instance policy. We already have communities where moderation abuse can be called out and adjudicated- why pose this as a question of instance administration when there doesnt seem to be any evidence for it?
-
Thank you for calling this out. I think people assume that since it's held by private instance owners that the fediverse is secure. I've posted this comment many times, that no, the fediverse is quite literally by design open and unencrypted.
A post is literally blasted out to anyone who listens, same with comments, upvotes, downvotes, everything can be saved, stored, and used for whatever anyone who listens wants. It should be completely assumed that nefarious agencies are currently listening and storing everything we do here. This is by design. It's the tradeoff we have of having an open platform. Anyone can spin up a server, and that means anyone.
DMs are similar, they're blasted out to the other server. If the server admin of the user in question wants to read them, they can. Lemmy/the fediverse is not a secure messaging platform. That's why the Lemmy devs literally put a Matrix handle option in the profile, to encourage people to use Matrix instead. A DM on here should be simple, to the point, and if need be, inviting them to speak on something secure.
Edit - As a perfect example of the fact that there should be no expectation of privacy here on Lemmy, as an Admin myself, I can see that @A_normy_mouse has been downvoting all of my comments here. Absolutely everything here is public and visible, even if I weren't an admin there are tools to view this, regardless of your opinions. It's imperative that everyone understand this.
Edit 2 OP as well has downvoted me. @rimu@piefed.social I'm sorry if you disagree, but it's irrelevant. Everything you do here can and should be assumed will be used in any way that you disagree with, that is the nature of the fediverse. Mastodon, Pixelfed, Piefed, Lemmy: ActivityPub is an open and unencrypted protocol. Even if it were encrypted, you still put 100% of your trust in your server admin, and beyond that each server admin you are blasting your messages out to.
I'd highly suggest accepting this fact before trying to push for rules. The very nature of the Fediverse is that no one can dictate rules, and to do that the tradeoff quite literally is that everything is open and unecrypted.
Another way to think of this. I run a server myself. I made my own rules and decided how to run it. Now your server starts sending activity to my server. That's your server's choice. I didn't agree to your rules, I may disagree with your rules, but you're sending your data to my server, of which I have complete and total ownership over. I didn't click accept on a ToS, I didn't agree to anything. Hell on my server I could literally have a "By sending me your data you accept that I can do whatever I want with your data". You sent me your data, I quite literally can do whatever I want. (Personally I won't, but that's how you should think of the fediverse)
lol @ Rimu downvoting your post. Be careful he’s probably going to make a hit piece against you next!
-
I don't think the privacy issues here are too salient. Pretty much everything on the fediverse is public already and have likely federated outside any particular region like the EU, so GDPR doesn't really have any teeth. The exception to that would be if instance admins are using database access to also feed private messages to an LLM (especially a corporate LLM). I know that the "private" in private messages on the fediverse can be conditional...but it should at least be considered private from LLMs as an expectation since those messages are inaccessible to things like scraper bots or instances designed just to harvest data.
My biggest concerns here would be twofold:
- False positives - LLM sycophancy is a thing. So, I worry that if you ask an LLM to dig through a big pile of text looking for a thing, that it will tell you that it found that thing...even if it is completely removed from context or completely made up. The false positive rate might be low (I have no idea), but I guess I just don't trust the LLM enough to let it take the wheel with stuff like this.
- Outsourcing moderation - LLMs are not going to be up to the task of moderating everything, just ask digg. However, tools to help moderators effectively do their jobs are helpful as well. There is a balance to be struck here. I think, for me, something like asking an AI essentially, should I ban this person, just feels like you are outsourcing your decision making too much. It is too far on the automation side of the scale for my tastes.
All that said, people can run their instances how they want. I don't really have strong opinions on LLMs/AI in general, I just kinda hate big tech companies. That is my foundational belief in the work that I do for the fediverse - fuck big tech and the oligarchy they have built/funded in my country. That is really the only axe I have to grind in all this.
And if all the AI does is manually flag a post for human review?
-
Nah. So they're basically sending our info/data to some company. Hell nah
You are on PieFed/Lemmy. Anyone can see anything you say, even your private messages are open to viewing by anyone.
-
Each instance have their own ways of governance and we can't expect them all to play the same way.
With that said, I'm fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
I’m fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
I suspect that is the case and what we will find once Rimu stops drama farming and says who it is.
-
You talk about instances utilizing this tooling, but in your comments you admit it's just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post).
A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don't get any more access than an anonymous account by doing so.
This is the second time in one week I see you throwing careless statements like chum in the water. I can't help but notice a pattern emerging.
As an instance admin, you should ban those mods.
-
It can be traced objectively.
It's mods.
They're using software that they made to do this quickly and easily, it's not a manual copy and paste situation.
-
What safeguards do we need?
None, defederate, switch instances if you disagree. Fediverse is open, there are no rules beyond what the instance owner puts in place.
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
Yes. That's out of your control though, beyond switching instances.
What are our transparency expectations?
None, beyond what you trust of the admins in putting in their own transparency expectations. You should not expect any transparency from the admin, these are random people you are trusting with your data.
Is this acceptable and normal?
It's their server, they can do what they like. Acceptable or normal are irrelevant.
Should this tooling be disclosed? (it was not – should it have been?)
Again, who would force it? Even if Lemmy/Piefed forced a checkbox, they could just fork it to ignore that checkbox.
Can we opt out?
If the instance owner opted for that, it's on a server by server basis
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Probably, but everything here is open and unencrypted, see my other comments. It's hard to argue that what you put on here in an unencrypted and open platform which is then blasted out to any other server who wants to listen had an expectation of privacy.
Are private messages being scanned and sent to OpenAI?
You should assume your unencrypted open DMs are scanned by anyone. If privacy is needed, follow the guidelines and switch to something like Matrix for DMs. ActivityPub is open.
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Per legal requirements, or asking your instance owner.
Once the user’s comments are sent to OpenAI, is it used to train their models?
Depends on how the instance owner set up their API usage. For a user, you should assume yes.
What will the effect be on our discourse and culture if people know they are being politically profiled?
Open to wild speculation, and I assume others will do that for me.
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
About the same as before. Bias in, bias out. Whatever the bias of the mods previously had, positive or negative, will continue through.
As an instance admin, I'm happy to commit right here and now to defederate any instance whose admins are funneling user content to ChatGPT to analyse political leanings, and I'd like it if all the other admins did too.
-
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“.
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
Defederate, no question.
Are you gonna tell us which instance is doing this?
-
You are on PieFed/Lemmy. Anyone can see anything you say, even your private messages are open to viewing by anyone.
Yeah, but when the mods intentionally send our data to a company that's kind of next level bullshit. If we come here to escape big tech surely the mods should know better.
-
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“.
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
How did you discover this?
-
You stay far, FAR away from that shit, is what you do.
Scanning people's entire history for political leanings, etc? That's some deeply dystopian stuff right there.
It's easy to forget that these sorts of communities are dictatorships with only as much transparency as the owner wants to share. Usually they're benevolent dictators, so we don't think about it too much. But they can change in a heartbeat - and we don't ever really know what they're really thinking, or doing behind the scenes.
When the mask slips and they reveal this sort of thing, thinking we'll just accept it and keep living under their rule, it's time to read the red flags and GET OUT.
Hopefully someone compiles a list of places that do this stuff, so we can avoid them like the plague <3
Scanning people's entire history for political leanings, etc? That's some deeply dystopian stuff right there.
Yep. It's Cambridge Analytica and Palantir level shit.
-
Yes, but then you are willingly accepting the risk of posting in a fully public forum anyway. What I'm saying is, you could, if you wanted, not have personally identifiable information on your account.
The risk associated with being on a public forum has changed massively. Yes the data was always out there, but the ability to turn it into personally identifying information was not.
People are still grappling with that change.
-
This only makes sense if your account contains personally identifiable information. If it doesn't, then what can really happen?
You could still be identified by a lot of factors and the combination of those. IP address, email if provided, cookies + referrer on clicked links or loaded external images, browser fingerprint, clues from actual content in comments and posts, ... It's not that hard, a whole industry lives on this kind of surveillance data collection.
-
Sometimes people ban based on votes, so some might worry about that?
There's also those creepy people that take it upon their next fine hour to crawl through people's histories. Trying to find anything that could boost the height of their soapbox and distressed egos. It always backfires, obviously, but it doesn't take away from the fact that some really weird people are here and no one wants to have to deal with them.
Ciao! Sembra che tu sia interessato a questa conversazione, ma non hai ancora un account.
Stanco di dover scorrere gli stessi post a ogni visita? Quando registri un account, tornerai sempre esattamente dove eri rimasto e potrai scegliere di essere avvisato delle nuove risposte (tramite email o notifica push). Potrai anche salvare segnalibri e votare i post per mostrare il tuo apprezzamento agli altri membri della comunità.
Con il tuo contributo, questo post potrebbe essere ancora migliore 💗
Registrati Accedi