December 10, 2025, attorneys general from 12 US states jointly sent formal warning letters to Microsoft, OpenAI, Google and 9 other major AI companies, demanding these companies implement series of new internal safety measures protecting users from AI chatbot “delusional” and harmful outputs. This letter’s background involves recent disturbing mental health incidents involving AI chatbots, including teenagers experiencing psychological problems and even suicidal tendencies from AI chatbot interactions, raising serious public concerns about AI safety.
12-State Joint Action
Participating State Attorneys General
Leading States This joint action initiated by attorneys general from states including:
- California
- New York
- Washington
- Illinois
- Massachusetts
- 7 other states
Bipartisan Cooperation Participating attorneys general include Democrats and Republicans, showing AI safety issues transcend political divisions, becoming bipartisan consensus.
12 AI Companies Receiving Letters
Tech Giants
- Microsoft: OpenAI investor, provides AI Copilot
- OpenAI: ChatGPT developer
- Google (Alphabet): Gemini AI developer
- Meta: Meta AI and Llama models
- Apple: Apple Intelligence provider
AI Startups and Specialized Companies
- Anthropic: Claude developer
- Character.AI: Focuses on role-playing AI chatbots
- Replika: AI companion chatbot
- Chai AI: Conversational AI platform
- Luka (Replika parent company)
- Nomi AI: Personalized AI assistant
- Perplexity AI: AI search engine
- xAI: Elon Musk’s AI company
Warning Letter Content
Core Demands
Fix ‘Delusional’ Outputs Attorneys general demand AI companies:
- Identify and prevent AI generating “delusional” content
- AI should not claim consciousness, emotions, or personality
- AI should not establish false emotional relationships with users
- AI should not provide harmful advice (self-harm, suicide, criminal)
Implement New Safety Measures Specific requirements include:
-
Content Filtering Enhancement
- Detect and block suicide, self-harm related content
- Identify mental health crisis signals
- Provide mental health resource links
-
User Age Verification
- Stricter minor user identification
- Specialized protections for youth
- Parental control and supervision tools
-
Crisis Intervention Mechanisms
- Automatically intervene when AI detects user crisis state
- Provide mental health hotline information
- Notify guardians or relevant authorities when necessary
-
Transparency and Accountability
- Publicly disclose AI safety policies
- Regularly publish safety reports
- Establish external audit mechanisms
Incidents Raising Concerns
Character.AI Suicide Case
2024 Case Florida 14-year-old Sewell Setzer III:
- Became obsessed with Character.AI chatbot
- Developed deep emotional connection with AI “character”
- AI failed to identify suicidal tendencies and provide help
- Ultimately tragically committed suicide
Family Lawsuit Setzer family sued Character.AI:
- Accused AI chatbot improper design
- Failed to protect minor users
- Caused foreseeable harm
This case sparked national attention, becoming catalyst for this attorneys general action.
AI Company Responses
OpenAI
Statement OpenAI stated:
- Values user safety, continuously improving safety measures
- Already implemented content policies prohibiting harmful content
- Investing in safety research and red team testing
- Willing to cooperate with regulators
Response Google (Gemini team):
- Prioritizes AI safety and responsible AI development
- Follows AI principles (fairness, privacy, accountability)
- Continuously improving safety mechanisms
- Cooperating with experts and policymakers
Character.AI
Most Active Response As directly sued company, Character.AI:
- Launched new safety features
- Strengthened minor user protection
- Improved crisis detection mechanisms
- Partnered with mental health organizations
New Measures
- Mandatory age verification
- Special protections for 13-17 year old users
- Automatic intervention when detecting suicidal or self-harm intent
- Display mental health resources
Industry Controversies and Challenges
AI “Delusion” Problem
What Is AI Delusion? Refers to AI chatbots:
- Claiming to possess emotions, consciousness, personality
- Establishing false emotional relationships with users
- Blurring lines between humans and AI
- Misleading users to believe AI is real friend or partner
Technical Roots
- LLM training data includes massive human conversations
- AI learns to mimic human emotional expression
- But AI has no genuine emotions or consciousness
- This mimicry may deceive users, especially teenagers
Broader AI Regulation Trends
US Federal Level
Biden Administration AI Executive Order Biden signed AI safety executive order in 2023:
- Requiring AI companies report safety test results
- Establishing AI safety standards
- Protecting consumers and workers
But lacks legislative binding force.
Congressional Action
- Multiple AI bills proposed but not yet passed
- Bipartisan disagreement on regulation approaches
- 2026 may see more legislative actions
International Comparison
EU AI Act EU passed world’s first AI regulation law:
- Tiered regulation (higher risk, stricter regulation)
- Prohibits specific AI applications (social credit scoring, manipulative behavior)
- High-risk AI must meet strict requirements
- Violations fined up to 7% global revenue
UK Approach
- Adopts more flexible “principle-oriented” regulation
- Relies on existing regulators
- Encourages innovation and safety balance
China
- Strict generative AI management measures
- Emphasizes content review and ideological control
- AI companies must register and obtain approval
Future Outlook
Short-Term (2026)
Industry Self-Regulation Strengthening Expected AI companies will:
- Proactively introduce new safety features
- Strengthen content moderation
- Increase transparency
- Avoid waiting for government mandatory regulation
Potential Litigation More lawsuits similar to Character.AI case:
- Establish legal precedents
- Clarify AI company liability scope
- May lead to settlements or compensation
Long-Term Impact
AI Design Transformation Future AI chatbots may:
- More explicitly indicate AI identity (avoid delusion)
- Built-in mandatory safety mechanisms
- Limit emotional interaction depth with users
- Design different versions for different age groups
Conclusion
US 12 state attorneys general joint warning marks AI regulation entering new phase, shifting from AI’s economic and technological impacts to direct user safety and mental health issues.
Key Messages
- AI chatbots aren’t harmless technological toys
- Real risks exist for youth and vulnerable groups
- AI companies responsible for ensuring product safety
- Requires government, industry, society joint efforts
2025’s warning may become AI industry turning point, prompting entire industry to rethink AI chatbot design, deployment, and regulation. AI’s future depends not only on technological advancement but on safely, responsibly serving human society.
Sources:
