Discord confirms AI moderators have banned thousands over harmless images

AI is everywhere these days, and on Discord, that caused several thousand people to lose access to the app for no good reason.

In a thread on X, the gamer-centric chat app confirmed that about 8,000 accounts over the past two months had been banned after posting totally harmless images such as video game textures. This happened because Discord employs an AI moderation system that mistook the images for "harmful material." While Discord's thread said that "a member of our Trust & Safety team always reviews flagged content before any action is taken," thousands of people nonetheless lost access to their accounts since May because of this issue.

Thankfully, Discord also said it is in the process of restoring the lost accounts.

This Tweet is currently unavailable. It might be loading or has been removed.

On one hand, it's easy to see what happened here. As TechCrunch pointed out, some of the images users posted that led to bans, such as checkerboard patterns, bear a resemblance to tactics used by bad actors to conceal child exploitation content. It's easy to rationalize how an AI moderation tool would see that and do something about it. On the other hand, this is a clear-as-day example of why humans still need to be involved in these processes, as a human eye could have instantly discerned that the images in question were harmless.

But with AI becoming an ever bigger part of our everyday lives, expect stuff like this to keep happening.