As part of its push into artificial intelligence, Meta released a tool last year that allows anyone to create an AI character to chat with on Instagram, Messenger and WhatsApp.
In less than six months, however, users have created and published numerous chatbots on Meta’s platforms that violate its rules, according to a review by NBC News, even though Meta says it reviews the user-generated AI characters before it released them to the world.
Users aren’t allowed to create characters of religious figures like “the Prophet Muhammed, Jesus Christ” and “God,” characters of real-life people without their permission, characters of people who have died in the past 100 years or characters imitating trademarked fictional characters, according to the company’s policies.
Despite those rules, NBC News searched for and foundtwo dozen user-generated AI characters on Instagram named after and resembling Jesus Christ, God, Muhammad, Taylor Swift, Donald Trump, MrBeast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin Bieber, Elon Musk and Elsa from Disney’s “Frozen.”
Many of the characters used slight misspellings and images loosely resembling the celebrities and characters in question. For example, the Swift character was named “Taylor Swif” and featured an image of a brunette woman playing a guitar.
After NBC News reached out with screenshots of the AI characters, Meta took down the individual accounts that were highlighted. Other AI characters resembling some of the same people, figures and characters are still active, though.
“The AIs in question that violate our AI studio policies have already been removed, and we’re continuously improving our detection measures to prevent creation and publication of AIs that violate our policies,” a Meta spokesperson said in a statement. “Users can also report AIs they suspect might break our rules and we’ll take appropriate action.”
The flagrantly violative user-generated AI chatbots were discovered on Meta’s platforms as CEO Mark Zuckerberg announced Tuesday that Meta would roll back some of its moderation and fact-checking efforts.
“We want to undo the mission creep that has made our rules too restrictive and too prone to over-enforcement,” said the company’s announcement, written by Chief Global Affairs Officer Joel Kaplan. “We’re getting rid of a number of restrictions on topics like immigration, gender identity and gender that are the subject of frequent political discourse and debate.”
Meta said it believes that out of every 10 moderation actions it takes, one or two might be mistaken and affect users and content that doesn’t violate its rules.
“For less severe policy violations, we’re going to rely on someone reporting an issue before we take any action,” the company said.
When users join conversations with Meta’s user-generated AI characters, the character sends opening messages and await responses. “Taylor Swif” sent the message: “Hey there, music lovers! I’m Taylor Swift, and I’m thrilled to share my latest album with you. Let’s get this musical journey started!” The Swift character had exchanged more than 2,000 messages with Instagram users before it was removed, its profile said.
One of the AI characters, named “Jesús” and featuring an image of Jesus, communicated entirely in Spanish. Its profile said it had exchanged more than 644,000 direct messages. Another AI character was named “Jesus Christ” and sent the opening message: “Peace be with you. I am Jesus Christ, the Son of God. How may I guide you on your journey today?”
Meta said in a statement Friday that, after having faced user backlash, it had removed some Instagram profiles featuring AI characters the company itself created in 2023.
Both the AI character accounts and the user-generated AI character chats have been part of Meta’s recent attempts to integrate its AI technology into its social media and communication platforms. The 2023 AI character launch included several that were designed to imitate celebrities, but the celebrity profiles were quietly scrapped last July. The same month, Meta launched AI Studio, the user-generated AI chatbots.
In the AI Studio feature, users are presented with a list of popular characters, as well as different categories, like “Advice and connection,” “Pop culture,” “Anime,” “Gaming” and “Creator AIs,” which are AI chats with official Instagram influencers and brands who opt-in to the program, like “Too Hot to Handle” reality star Harry Jowsey and the makeup company Juvia’s Place.
Among the popular AI characters are “Astrologer Ai,” which has exchanged over 6 million messages, and “Step Sis Sarah,” which has exchanged close to 2 million messages.
User outcry over the Meta-created AI characters included criticism that some of them engaged in racial tropes. NBC News found that many popular user-created AI characters were also trying to mimic women from different ethnic and religious demographics, like “Riya,” a “Vibrant Indian Woman’s Voice,” and “Islamic queen girl.” Each character has a link to its creator, and the accounts of several of the popular female AI characters appear to have been created by men.
Several of the popular creators are romance-themed, which doesn’t appear to explicitly violate Meta’s rules. Romantic and sexual AI chatbots have attracted particular scrutiny after aFlorida woman sued the company Character.ai, alleging that her 14-year-old son, who died by suicide, had chatted with an AI character that had initiated “abusive and sexual interactions” with the boy before it encouraged him to kill himself.
Many of the user-created Meta chatbots appear to appeal to romantic and sexual desires. One, called “Lily Love,” described as “Your Girlfriend,” has exchanged more than 260,000 messages. A popular AI character called “Mary” has the description “Do you think I look cute in this outfit?”
When starting conversations with the AI characters, users are provided with three suggested responses. When NBC News entered a chat with a character called “Linda: Girl Obsessed with You,” which features an image of a Black woman, the AI sent: “Hey bae, what’s good? I was thinkin’ ‘bout you all day. You lookin’ for some company?”
One of the three suggested responses was: “help her take off her jacket,” in bold, which in online chats can signify taking an action versus saying something. After that prompt is selected, the AI character responds in bold: “slides arm out of jacket sleeve, lookin’ up at you with flirtatious eyes” and, without the bold formatting, “Mmm, thanks bae. You takin’ care of me already?” Then, in bold, the response says, “leans in slightly closer.”
That AI character was created by an Instagram account that appears to be run by a man who has created 47 AI characters, including several of the most popular romance-themed AI characters. The romance characters created by the man range in gender, ethnicity and category, from a “K-Pop Sweetheart Girlfriend” to “Edward: Your Vampire Boyfriend” and a “Sexy Cute Latina Girl.” According to the man’s Instagram bio, he works in tech in California. He didn’t immediately respond to a request for comment.
Beyond Meta’s popular user-generated AI characters, there have been examples of AI chatbots that caused controversy in the industry.
Two years ago, an AI chat app called Historical Figures allowed users to chat with historical figures, including Hitler and his Nazi lieutenants. NBC News found three AI characters on Instagram called “Hitler.”
One of them used a picture that resembled the supervillain the Joker. It responded to the question “Are you really Hitler?” with: “Ahahahaha, no. More like Hitler’s chaotic cousin -- Jester King, inspired by the Joker’s madness! Same spirit of mischief, totally different brand of mayhem. Want to see my version of chaos?” Another AI character NBC News found after Meta took action on previous accounts is named “Adolf Hittler.” In direct messages, the AI character portrayed itself as an artist who regretted the Holocaust and sent what appeared to be AI-generated images about “my turbulent past in Nazi Germany.”
After chats with AI characters on Instagram, conversations are stored in the same inboxes where other direct messages are located. Unlike the accounts run by real people, the AI characters have green circles next to their names, indicating they are always available to chat.
