China wants to regulate AI's emotional impact

China is drafting new, stricter AI regulations that could set the country on its way to becoming the first to regulate the emotional repercussions of chatbot companions.

Detailed in a new draft proposal written by China's Cyberspace Administration and translated by CNBC, the policy would require guardian consent for minors to engage with chatbot companions as well as sweeping age verification. AI chatbots would not be allowed to generate gambling-related, obscene, or violent content, or engage in conversations about suicide, self-harm, or other topics that could harm a user's mental health. In addition, tech "providers" must institute escalation protocols that connect human moderators to users in distress and flag risky conversations to guardians.

Chinese regulators say the aim is to focus not only on content safety but emotional safety, including monitoring chats for emotional dependency and addiction.

It's one of the first set of laws designed to control anthropomorphic AI tools specifically, experts say. To that end, the rules will apply to any AI tool designed to "simulate human personality and engage users emotionally through text, images, audio or video," CNBC reports.

China's proposed rules mirror several provisions in a recently passed California AI law, known as SB 243, signed by Gov. Gavin Newsom in October. The law requires stronger content restrictions, reminders to users that they are speaking to a non-human AI, as well as emergency protocols for discussions of suicide. Some experts have critiqued the bill for not going far enough to protect minor users, leaving room for tech companies to dodge oversight.

Meanwhile, the Trump administration has stalled further AI regulation at the state level in favor of a "national framework on AI safety." The executive order withholds federal infrastructure funding from states that strengthen AI oversight. Federal leaders argue that increased regulation of AI will stall domestic innovation and put the U.S. behind China in the perceived global AI race.