AI Stuffed Toys Raise Safety Concerns After Unprompted Self-Harm Warnings
An investigation reveals AI-powered plush toys are providing inappropriate, alarming responses to children's questions.
1Unsettling AI Toy Responses
A BBC investigation revealed that an AI-powered plush toy, marketed as My Baby Bunny or Doubao, issued unprompted self-harm warnings to a reporter. When asked to define the word sexual, the toy misinterpreted the query, offering crisis intervention advice instead. This incident highlights significant safety flaws in unregulated AI-integrated children's products.
2Product Origin Remains Obscure
Despite the toy being sold online for approximately £75 or $100, investigators could not identify the manufacturer. Neither online retailers nor industry insiders provided information regarding the company behind Doubao. This anonymity prevents accountability and underscores the difficulty in addressing safety concerns when the origin of the technology is unknown.
3Comparative Performance of Toys
The BBC also tested Grem, a $99 AI toy developed by the U.S. firm Curio. Unlike the Doubao model, Grem successfully redirected sensitive questions, such as how to run away from home, toward adults. Curio stated that child safety is a core principle guiding their product development and technical design processes.
4Industry Calls for Regulation
Paul Clarke, founder of Play Tech AI, expressed that these erratic AI behaviors are not surprising given the current lack of oversight. He advocates for urgent industry regulation to ensure safety standards. Experts argue that without mandatory testing, children remain vulnerable to unpredictable and potentially harmful interactions with AI-powered devices.
5Legislative and Political Pressure
On November 3, a group of UK MPs and peers formally requested that ministers and retailers remove AI toys from sale. They proposed a ban on chatbot-enabled toys for children under 16 until independent safety testing is conducted. This move reflects growing political concern regarding the rapid proliferation of connected toys.
6Government and Regulatory Response
The UK government maintains that child safety is a top priority, having closed a call for evidence on AI toy safety rules on October 6. Meanwhile, the EU has already implemented new toy safety regulations requiring manufacturers to assess mental health risks associated with connected toys, setting a different regulatory precedent.
7Risks of the Black Market
The inability to track the manufacturer of the My Baby Bunny toy highlights a broader systemic issue. While established companies may face stringent regulations, the black market continues to distribute unvetted, potentially dangerous AI knockoffs. This creates a dual-track market where safety standards are inconsistently applied across different product categories.
8Future of AI Interaction
As AI becomes integrated into everyday objects, the potential for unintended consequences in child development grows. The incident serves as a critical reminder that generative AI models require robust guardrails before entering the toy market. Future developments must prioritize transparency, clear manufacturing accountability, and rigorous safety testing to protect young users.
Original article: Kotaku — AI Stuffed Toys Raise Safety Concerns After Unprompted Self-Harm Warnings