I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“. (The italic bits are where I’ve redacted the ideology they’re seeking).
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
You talk about instances utilizing this tooling, but in your comments you admit it’s just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post).
A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don’t get any more access than an anonymous account by doing so.
This is the second time in one week I see you throwing careless statements like chum in the water. I can’t help but notice a pattern emerging.
Nah. So they’re basically sending our info/data to some company. Hell nah
This is illegal according to GDPR if it is not stated in their privacy agreement.
This should have been disclosed to the public and not done in secret.
The notion of political profiling is also pretty crazy, although it is not surprising considering how common it is to find fake communists and demagogues.
I would like to see some ROBOT9000 esque oddball meme communities overtly based on heavy algorithmic moderation, can be LLM but wouldn’t have to be. Weird rules strictly enforced by robots, could be fun.
I don’t like this happening, and there should be transparency in all moderation decisions, but some of these points make no sense.
There is essentially no expectation of privacy on threadiverse platforms. Everything is public and probably already being used to train models.
There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They and should not be used to share anything sensitive.
Thank you for calling this out. I think people assume that since it’s held by private instance owners that the fediverse is secure. I’ve posted this comment many times, that no, the fediverse is quite literally by design open and unencrypted.
A post is literally blasted out to anyone who listens, same with comments, upvotes, downvotes, everything can be saved, stored, and used for whatever anyone who listens wants. It should be completely assumed that nefarious agencies are currently listening and storing everything we do here. This is by design. It’s the tradeoff we have of having an open platform. Anyone can spin up a server, and that means anyone.
DMs are similar, they’re blasted out to the other server. If the server admin of the user in question wants to read them, they can. Lemmy/the fediverse is not a secure messaging platform. That’s why the Lemmy devs literally put a Matrix handle option in the profile, to encourage people to use Matrix instead. A DM on here should be simple, to the point, and if need be, inviting them to speak on something secure.
Edit - As a perfect example of the fact that there should be no expectation of privacy here on Lemmy, as an Admin myself, I can see that @A_normy_mouse has been downvoting all of my comments here. Absolutely everything here is public and visible, even if I weren’t an admin there are tools to view this, regardless of your opinions. It’s imperative that everyone understand this.
Edit 2 OP as well has downvoted me. @rimu@piefed.social I’m sorry if you disagree, but it’s irrelevant. Everything you do here can and should be assumed will be used in any way that you disagree with, that is the nature of the fediverse. Mastodon, Pixelfed, Piefed, Lemmy: ActivityPub is an open and unencrypted protocol. Even if it were encrypted, you still put 100% of your trust in your server admin, and beyond that each server admin you are blasting your messages out to.
I’d highly suggest accepting this fact before trying to push for rules. The very nature of the Fediverse is that no one can dictate rules, and to do that the tradeoff quite literally is that everything is open and unecrypted.
Another way to think of this. I run a server myself. I made my own rules and decided how to run it. Now your server starts sending activity to my server. That’s your server’s choice. I didn’t agree to your rules, I may disagree with your rules, but you’re sending your data to my server, of which I have complete and total ownership over. I didn’t click accept on a ToS, I didn’t agree to anything. Hell on my server I could literally have a “By sending me your data you accept that I can do whatever I want with your data”. You sent me your data, I quite literally can do whatever I want. (Personally I won’t, but that’s how you should think of the fediverse)
You’re hyperfocusing on one point, as if that’s the only part that matters and ignoring all the rest. I don’t consider that helpful, hence the downvote.
What is especially unhelpful is abusing your admin access to call out people’s votes. Leave that shit alone.
abusing your admin access
Everyone has admin access, including you…
That is quite literally my point. Everything, absolutely everything here is open and can be used however any instance owner wants. You can say “leave that shit alone”, but there is no obligation to whatsoever.
You should assume every instance owner can and is viewing all of your private data, sending it through whatever LLM/mod tools they want. Are they? Probably not. But they can, and there is no obligation not to.
Yeah you can do that but now you’re on my do-not-trust list. And probably a few other people’s lists.
I appreciate you being open about your opinions because now I can make an more informed choice about interacting with you and the instance you run.
Don’t you think everyone deserves the information they need to choose which instances they want to interact with, according to whatever criteria is important to them? Even if your criteria are different?
That’s a stupid take, you’re basically shooting the messenger here.
Stop throwing a tantrum like a child. You ranted. You were explained why your tantrum is pointless. Move on.
GOOD. NO ONE should be trusted here! I’m just some guy who decided to spin up a server, there should be zero trust! THIS IS MY POINT.
Don’t you think everyone deserves the information they need to choose which instances they want to interact with, according to whatever criteria is important to them? Even if your criteria are different?
This depends on the trustworthiness of the admin themselves, and even then every admin is just some person who decided to spin up a server, just like me. Trust is built and earned, it shouldn’t be implicit. The option you have is to defederate, or leave and join another server.
I’m really not trying to be an asshole here, but your post is what caused me to do this. This is not a unique post, this is a fundamental core principal of the fediverse that every user must understand. That by being here, it is not a private secure place, you are quite literally blasting every comment, post, and upvote, to whoever wants to listen. Literally everyone. Any semblance of privacy is purely a UI trait. Rules/guidance is purely 100% based on what each server owner chooses.
It’s occasionally worth calling out that votes are also public. I think twice before hitting those buttons
Why would you care if anyone knows how you vote on comments?
Not OP, but the votes being public (not only on comments but also on posts) make it really easy for someone with malicious intent to generate a profile on your interests, political and sexual orientation, health/mental issues, addictions and so on. It’s a goldmine of data that should be protected.
Sometimes you get harassed by lunatics.
.ml I call you out.
The entire add industry has been collecting preferences, likes, dislikes for decades. Its one of the most profitable pieces of information
No data is as useful as what makes you personally engage.
Sometimes people ban based on votes, so some might worry about that?
Occasionally people have meltdowns and accuse/threaten other users for daring to vote a certain way, presuming specific motives for doing so
First fo all: I don’t like this either.
There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They should not be used to share anything sensitive.
Agreed, but that admin is breaking his promise, duty, responsibility (call it what you will) if they then upload these messages to an LLM for evaluation.
I would argue for this being actually illegal, at least under the GDPR.
But that was just one of many potential conflicts @rimu raised. We should concentrate on the real conflicts of LLM comment moderation.
It’s very clear on signup, on the READMEs, even on the DM portal itself, that messages are unencrypted and there is no sense of privacy, and that admins have full visibility and can do what they want with them.
Agreed, but that admin is breaking his promise, duty, responsibility (call it what you will) if they then upload these messages to an LLM for evaluation.
There is no promise, duty, or responsibility that an admin has beyond legal and what they themselves promise. The fediverse is great in that if you disagree with your admin, you are free to leave and choose a different one.
As for GDPR, feel free to argue it, but when it’s claimed at every turn that messaging is unencrypted and basically open, well, I don’t think it’d hold up. It literally says to go use Matrix or something else.
you are free to leave and choose a different one.
I only have that freedom if the admin tells me that they use LLMs in this manner or if they federate with instances that do. At the moment everyone is in the dark.
You should assume everything you post here is being used to train LLMs. It doesn’t take an admin to do so. It takes anyonr who feels like looking. And there’s already evidence that we’re being scraped.
and it will continue to be. Again, you need to understand this. There are no rules, guidelines or anything that an instance owner needs to follow beyond whatever legal requirements they have in their specific jurisdiction.
So, I guess in your pervalence, you are correct, you do not have that freedom. Even I, as an instance owner, do not have that freedom, because everything I’m typing here is being sent out to as many servers are listening too. By being completely open so that anyone can spin up a server and listen for activity, it literally means that we are open and any server can listen for activity.
Anyone can spin up a server, create some LLM bot, and start replying to anyone they want. That instance can be defederated of course, but that is the only tool. This is what you signed up for, this is the open and free internet. We do not have any walls here.
You’re a fucking AnCap? That explains soooooo much.
I think this will exemplify the beauty of federation. If I find out my instance mods are running all of my comments through a company’s ai model, I’ll switch instances. This is in great disparity to something like Instagram or Snapchat where every photo I post is immediately fed to ai and my only options are: be okay with it, never post, or delete Instagram.
But you don’t even need to be a mod to do that. Anyone at any moment can run someone else’s entire comment and post histories through an LLM.
Mods of any instance you’re federated with can do this
Seeing that every single post we make is completely public there is a high chance someone out there already used all your comments for training an AI model. As you say, the only thing you can do is just not post anything anymore.
this is what reddit does, and destroyed thier communities and left it with bots on most subs. reddit also lets you hide your history so you cant sniff out bots/ or chronic spammers.
politics.worlds and the tankies will certainly used this to ban people. Reddit does this in overdrive, which no repealable bans 99% of the time, despite nuances indicate its not a ban worthy post.
I don’t think the privacy issues here are too salient. Pretty much everything on the fediverse is public already and have likely federated outside any particular region like the EU, so GDPR doesn’t really have any teeth. The exception to that would be if instance admins are using database access to also feed private messages to an LLM (especially a corporate LLM). I know that the “private” in private messages on the fediverse can be conditional…but it should at least be considered private from LLMs as an expectation since those messages are inaccessible to things like scraper bots or listening instances designed just to harvest data.
My biggest concerns here would be twofold:
- False positives - LLM sycophancy is a thing. So, I worry that if you ask an LLM to dig through a big pile of text looking for a thing, that it will tell you that it found that thing…even if it is completely removed from context or completely made up. The false positive rate might be low (I have no idea), but I guess I just don’t trust the LLM enough to let it take the wheel with stuff like this.
- Outsourcing moderation - LLMs are not going to be up to the task of moderating everything, just ask digg. However, tools to help moderators effectively do their jobs are helpful as well. There is a balance to be struck here. I think, for me, something like asking an AI essentially, should I ban this person, just feels like you are outsourcing your decision making too much. It is too far on the automation side of the scale for my tastes.
All that said, people can run their instances how they want. I don’t really have strong opinions on LLMs/AI in general, I just kinda hate big tech companies. That is my foundational belief in the work that I do for the fediverse - fuck big tech and the oligarchy they have built/funded in my country. That is really the only axe I have to grind in all this.
No that is not correct, GDPR requires you to list the reasons and partners you share the data with.
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling
First of all, I agree with your main point, that this* is problematic, wrong even (and should not happen).
But I need to ask: how did you find out? Is this something that could be traced objectively, or did some people report/admit it?
Are they uploading stuff to corporate LLMs, i.e. LLMs that do not run themselves? (I think you answered this already when you wrote OpenAI, but I want this spelled out)
Are only admins or also mods doing this? That would make a big difference.
I’m also a little unclear about the process: are they uploading (copy-pasting) the actual comments, or links to them? To what extent can all this be automated on Lemmy/Piefed etc.? I.e., are there admin tools that just spit out all of a user’s content?
* again: specific political profiling outsourced to LLMs. OTOH we already have instances that do this manually.
But imo the process is deplorable even if they use LLMs with different prompts for modding.It can be traced objectively.
It’s mods.
They’re using software that they made to do this quickly and easily, it’s not a manual copy and paste situation.
this is the kind of thing that makes me want to never post or comment anywhere on the internet
As for the privacy issue:
The only counter to this is to move to a closed end-to-end encrypted groupchat so it can’t be mass LLM analyzed…
If you want a public forum, well… its public…
You can’t stop a script from just grabbing all the posts/comments… and its also federated, so the bot only needs to be able to access one instance and get it…
I mean they could simply just set up their own instance and pretend its just a benigh single-user instance… like what are you gonna do, defed all small instances preemptively? Use “login walls” to make the forum private? And somehow trust all other admins that are federated and make them also enforce a “login wall” policy?
Its a PUBLIC forum…
the only solution for privacy is a groupchat and only let in people that can keep a promise to not screenshot everything and give it to a LLM.
What safeguards do we need?
None, defederate, switch instances if you disagree. Fediverse is open, there are no rules beyond what the instance owner puts in place.
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
Yes. That’s out of your control though, beyond switching instances.
What are our transparency expectations?
None, beyond what you trust of the admins in putting in their own transparency expectations. You should not expect any transparency from the admin, these are random people you are trusting with your data.
Is this acceptable and normal?
It’s their server, they can do what they like. Acceptable or normal are irrelevant.
Should this tooling be disclosed? (it was not – should it have been?)
Again, who would force it? Even if Lemmy/Piefed forced a checkbox, they could just fork it to ignore that checkbox.
Can we opt out?
If the instance owner opted for that, it’s on a server by server basis
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Probably, but everything here is open and unencrypted, see my other comments. It’s hard to argue that what you put on here in an unencrypted and open platform which is then blasted out to any other server who wants to listen had an expectation of privacy.
Are private messages being scanned and sent to OpenAI?
You should assume your unencrypted open DMs are scanned by anyone. If privacy is needed, follow the guidelines and switch to something like Matrix for DMs. ActivityPub is open.
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Per legal requirements, or asking your instance owner.
Once the user’s comments are sent to OpenAI, is it used to train their models?
Depends on how the instance owner set up their API usage. For a user, you should assume yes.
What will the effect be on our discourse and culture if people know they are being politically profiled?
Open to wild speculation, and I assume others will do that for me.
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
About the same as before. Bias in, bias out. Whatever the bias of the mods previously had, positive or negative, will continue through.
This post makes many good points but this also reeks of the “Free Speech” argument.
Redacting the political ideology does nothing to protect the instance … it only hides the cause of the ban.
What does the person who had this run against them feel afraid to express?
To clarify: LLMs have no place at all in moderation. They are poorly adjusted linguistic slot machines that do not do a good enough, reliable enough, or repeatable job for this tasks. Especially given the concern about untrained and unsupervised human moderators.
the ai banning is to obfuscate the reason for the ban, like what reddit does. they just ban on the slightest issue, with no recourse. it allows them to ban more freely in the end.
Each instance have their own ways of governance and we can’t expect them all to play the same way.
With that said, I’m fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
