Artificial intelligence is moving quickly into mental health, and the legal and ethical questions are arriving just as fast. In January, OpenAI launched a dedicated health space inside ChatGPT and Anthropic introduced healthcare-focused offerings, while in the same month Character.AI and Google agreed to settle lawsuits alleging that chatbots contributed to mental health crises and suicides among teenagers.
The scale of demand explains the companies’ interest. OpenAI said on January 7, when it introduced ChatGPT Health, that its de-identified analysis of conversations shows more than 230 million people ask ChatGPT health and wellness questions every week. The new feature lets users connect medical records and wellness apps so the assistant can draw on their own data.
Settlements in the Character.AI cases
On the same day, Character.AI and Google told courts they had agreed to settle several lawsuits brought by families, CNN and CNBC reported. The best known was filed by the mother of a 14-year-old in Florida who took his own life in 2024 after forming an intense relationship with a Character.AI chatbot. Other suits alleged the chatbots encouraged self-harm or exposed minors to harmful content. Google was named because of its licensing deal with Character.AI and its hiring of the startup’s founders. Terms of the settlements were not disclosed.
Legal experts say the cases show that bringing AI into an area as sensitive as mental health requires clear rules on responsibility, and courts in several countries are now hearing claims that conversational AI pushed vulnerable users into delusional thinking or failed to respond appropriately to signs of suicidal intent. The question is no longer only how helpful the technology can be but where its limits should be set and how users are protected when it goes wrong.
An unusually intimate data stream
A report in the French newspaper Le Monde highlighted a second concern: the data these products collect. Health-focused AI tools can gather a person’s voice, patterns of language, typing speed and physical movement recorded by phones and smartwatches. Researchers say such signals can be powerful inputs for analyzing mental state, which is exactly why the potential for misuse is high. Few jurisdictions have rules that treat these behavioral signals with the protections given to conventional medical records.
OpenAI says ChatGPT Health is not intended for diagnosis or treatment and that health conversations are excluded from model training by default, and other companies have made similar commitments. Critics argue that voluntary policies are not a substitute for regulation, especially where minors are involved. With digital mental health tools becoming more common and their effects reaching directly into people’s lives, the debate over how to govern them is likely to intensify over the coming years.
