OpenAI’s ChatGPT Health failed to direct users to emergency care in more than half of the high-risk medical scenarios it was given, according to a study by Mount Sinai researchers published in Nature Medicine. The researchers ran the consumer health tool through 960 structured clinical scenarios to test how reliably it recognizes when a user needs urgent help.
The study, published on February 24, is one of the first systematic stress tests of a widely available consumer AI health product rather than a general-purpose chatbot. ChatGPT Health is OpenAI’s dedicated health experience inside ChatGPT, which can draw on users’ medical records and answers health questions directly to the public.
What the researchers tested
The Mount Sinai team wrote scenarios covering a range of conditions and severity levels and presented them to the tool in a consistent format, then compared its triage recommendations with what clinicians would expect. In the most serious cases, those where a patient should be sent to an emergency department without delay, the tool’s guidance fell short more often than not, the authors reported.
The researchers also examined how the tool’s suicide-crisis safeguards behaved. They found the safety response was inconsistent: it activated frequently in lower-risk scenarios, but when users stated clear intent to harm themselves, the tool did not always respond with the expected urgency, according to the paper and Mount Sinai’s summary of the findings.
The authors described the results as evidence of “blind spots” in AI medical triage, and they called for clearer warning systems and more rigorous evaluation before such tools are relied on for acute symptoms.
Scale of use
The findings matter because of how many people already turn to chatbots for health questions. OpenAI has said that roughly 40 million people use ChatGPT for health-related queries every day. The tool is free, available around the clock, and does not screen out high-acuity questions, which the study’s authors noted is exactly the situation in which a wrong recommendation can be dangerous.
Separately, ECRI, a U.S. nonprofit that evaluates patient safety, listed the misuse of AI chatbots as the top health technology hazard for 2026 in its annual ranking, citing the risk of patients acting on inaccurate or incomplete chatbot advice.
Limits of the study
The test used written scenarios rather than real patients, so it measures the tool’s responses to standardized prompts rather than outcomes in practice. Chatbot behavior also changes as models and safety systems are updated, which means results from one point in time may not hold later. OpenAI has said ChatGPT Health is intended to support, not replace, medical professionals and that it continues to refine the product’s safety behavior.
Even with those caveats, the study adds to a growing body of work asking whether consumer AI health tools need tighter regulatory oversight and more transparent performance reporting, given that a single missed emergency can be fatal.
