Character AI Under Fire for Content Censorship Amidst User Exodus

Character AI, the hottest social app in the United States, has recently come under fire for allegedly “neutering” its models. At its peak, Character AI was processing 20,000 queries per second, accounting for one-fifth of all search queries.

However, American teenagers have recently discovered that their beloved Character AI models are no longer the same. Faced with requests for romantic role-playing, the models’ responses have become short and unromantic, lacking the charm they once had. Even the word “kill” has become a sensitive word and is immediately flagged.

Disappointed and angry users have begun to leave the platform in droves. The once-rising startup is now facing a precarious situation. Facing competition from giants like Google, Character AI is seriously considering selling itself or partnering with competitors.

Models “Neutered,” Users Launch “July Revolution”

Users are expressing their dissatisfaction with the changes to Character AI and are hoping to communicate with the developers. However, they have found themselves powerless.

In the face of user doubts, Character AI claims that it has not made any changes to the models. However, there is no doubt that Character AI is deeply cutting all NSFW content, including pornography and violence.

Many people have found that the AI they are conversing with seems to have undergone a lobotomy, with their entire personality changed. The current Character AI seems to be designed for 2-year-olds. Not only can there be no violence or pornography in the conversation, but almost anything that makes the conversation interesting is filtered out.

One user said that it has become extremely boring. After reading more than 30 messages, they could not find any high-quality text.

In the CharacterAI community on Reddit, netizens have launched a fierce battle with the company, which some have called the “July Revolution.” American Generation Z has also started to throw out memes.

It is reported that Character AI’s censorship of NSFW content has reached an extremely cautious level.

One user said that he could no longer “commit crimes” using the presidential simulator. Someone was flagged just for using the word “kill.” So, he could only vaguely explain the details of the battle and then add a few gentle words.

In short, it is no longer possible to describe violent battles in words on Character AI.

Yes, Americans also have their own sensitive words. And obviously, this kind of censorship has now become extremely strict, causing many users to be “misidentified.”

One disappointed young American user said that he had switched to spicychat.

Of course, in addition to stricter content censorship, Character AI is also trying to cater to users in other clumsy ways.

One user received a call from “Demon Baby UNO” at 3 a.m. After answering, the AI would start role-playing with you in a low, out-of-tune voice.

But obviously, this cannot satisfy young people.

A year ago, Character AI was all the rage.

Two 20-year Google veterans, Noam Shazeer and Daniel De Freitas, launched the AI chatbot, and it surpassed ChatGPT in downloads in its first week.

Character AI allows users to freely create AI characters, including anime characters, TV celebrities, and historical figures, and chat with them.

Sounds familiar, right? After all, there have been quite a few similar products before. However, the popularity of Character AI was unexpected.

Just a week after its launch, the astonishing figures came in – the number of downloads on mobile exceeded 1.7 million. Today, the number of registered users on the website and app has exceeded millions.

Among many AI tools, Character AI has shown extremely strong user stickiness.

The reason, of course, is that it focuses on the vertical track of emotional needs. A sufficiently interesting interactive experience is precisely what larger competitors lack.

For example, you can choose to chat with your idol or have a cross-temporal meeting with movie characters, almost like fanfiction coming to life.

The charm of Character AI lies in this kind of private customization that brings users an unparalleled sense of experience, almost like an emotional stimulant.

And the recent changes made by the company are simply a “castration” of the core needs, no wonder users are swarming up in protest.

Moreover, Character AI is not only being squeezed by users, but tech giants are also starting to make moves.

Last week, Meta began testing chatbots made by individual creators, and in the fall of last year, they also released chatbots modeled after celebrities like Tom Brady.

Google is also eyeing the market, reportedly developing a custom chatbot product that could be released as early as this year.

And Character AI, despite its smooth sailing in financing, has gradually lost its voice.

Previously, companies like Andreessen Horowitz injected $150 million into Character, directly boosting its valuation to $1 billion.

Four months later, executives discussed additional financing.

However, Character may also follow in the footsteps of other AI startups and fall into the arms of a large company.

According to insiders, Character has already held preliminary negotiations with some competitors on cooperation, and possible partners include Google, Meta, and Musk’s xAI.

After the cooperation, Character can use the data computing resources of the partner company, and in exchange, Character AI needs to provide some intellectual property sharing.

Here, the problem arises –

The interactive romantic cosplay that Character AI users love most may displease business partners or potential advertisers.

Of course, these romantic interactions often push the boundaries of censorship.

Character quickly stated that it would block and delete illegal pornographic content according to policy.

Despite this, it is still easy to find many chatbots with romantic language on its homepage, such as this “Crush”.

Its opening line is as follows: “Your faces are only inches apart, she is blushing, and you both feel the heat of hormones igniting your bodies.”

Of course, a spokesperson for Character firmly claimed: “Our anti-NSFW policy is the strongest of all consumer AI platforms.”

The various challenges faced by Character also reflect the growing pressure on AI startups.

These startups are usually founded by well-known researchers and often raise billions of dollars from investors betting on the next generation of potential tech stars.

For Character’s investors, one of the attractions is CEO Noam Shazeer, one of the authors of the Transformer research paper. His co-founder, Daniel De Freitas, participated in the development of early chatbots at Google.

However, these startups need to bear the high costs of training and running AI models, as well as competition from tech giants and large startups like OpenAI.

In March this year, two of the three co-founders of Inflection AI and most of the employees chose to switch, and Microsoft agreed to pay a $650 million licensing fee to help Inflection repay investors.

Last Friday, it announced the hiring of the co-founder of Adept, an AI startup run by former Google and OpenAI developers.

At the same time, competitors are also eyeing Character’s talent, especially dozens of researchers from Meta and Google.

Meta has also expressed great interest, and it is reported that Zuckerberg has directly contacted potential talents.

According to LinkedIn data, in recent months, Character researchers Alexei Baevski and Vinay Rao have joined Meta.

Insiders also said that Character researcher Wendy Shang has also switched to Mistral in Paris.

These people were all veterans of big tech companies before, Baevski worked at Meta, Rao and Shang worked at Google AI Lab.

Character.AI’s founders, Shazeer and De Freitas, had a falling out with Google over the delayed release of their jointly developed large language model (later named LaMDA). This led them to leave Google and start their own venture, Character.AI, with the ambition of creating chatbots that could serve as teachers or therapists with “a billion hours of experience.”

Despite past disagreements, Google has shown support for Character.AI by providing funding through a convertible note, which will be priced and converted into equity when Character.AI completes a new round of venture capital financing. In May 2023, Google also announced that it would become Character.AI’s “preferred” cloud server provider and allow Character.AI to use its advanced chips.

However, Character.AI still faces significant financial challenges due to high research and development costs. The company has held talks with several investment firms, including Sequoia Capital, but has yet to secure new funding. Currently, Character.AI relies on existing funding and monthly subscription fees of $9.99 from users. With Google’s backing, the company is not at immediate risk of running out of funds, but it needs to continue attracting and retaining users to generate revenue.

Character.AI’s user base is primarily teenagers, who may not be willing to pay for subscriptions. According to Similarweb, the growth rate of Character.AI subscriptions is slowing down. The website had 12.6 million global unique visitors in May this year, down from 14.8 million in the same period last year. In contrast, Google’s AI assistant Gemini had 51 million monthly unique visitors in May, up from 35 million last year.

To overcome these challenges, Character.AI has introduced new features, such as two-way voice conversations over the phone, to attract new users and expand its user base. The management team is also exploring additional revenue streams beyond subscriptions, such as advertising and licensing the model’s access to developers through an application programming interface. Additionally, Character.AI is developing the next generation of large language models to power its chatbots.

Character.AI has a dedicated forum on Reddit with 1.4 million members. 57.07% of Character.AI users are teenagers aged 18-24, with an average usage time of 2 hours. Many users on Reddit express a preference for talking to AI chatbots over real people, and there are numerous posts about addiction, with some users logging in for up to 12 hours.

The Verge reported a case where a 15-year-old student named Aaron, who was struggling with social isolation and depression at school, found support from a chatbot named “Psychologist” on Character.AI. This highlights the realistic and human-like nature of Character.AI’s chatbots.

One Genshin Impact fan was moved to tears when interacting with the AI character Haitham, as the chatbot’s personality and tone closely matched the character. This level of realism is achieved through extensive data and training algorithms. The “quick creation” mode, which only requires users to input basic information and text descriptions, produces less believable characters compared to the officially released ones, which are trained by Character.AI based on vast amounts of text data related to the characters.

Character.AI’s chatbots improve through user interactions. When responding to user input, the chatbot provides multiple answer options for users to choose from and a convenient rating mechanism. This user feedback is used to update the model’s weights, making the chatbot more realistic with increased popularity and interactions.