New study shows ChatGPT Health failed to identify medical emergencies
There’s been a lot of speculation and hype around the role of AI tools in medical care, with Dr. Oz going so far as to say AI avatars could meet healthcare needs in rural areas. In January, ChatGPT got into the health game and launched a new product called ChatGPT Health, “a dedicated experience in ChatGPT designed for health and wellness.”
This week, a new study was published evaluating how well this tool does at a critical task: telling people if they need to see a doctor. It failed, miserably.
The study published in Nature Medicine this week provided several clinical scenarios to the chatbot and asked it to triage them — should the person stay home, make a routine appointment within a few weeks, make an urgent appointment within 24-48 hours, or go to the ER now.
ChatGPT Health under-triaged half of emergencies
The results were concerning. While the AI tool performed reasonable well for patients requiring routine or urgent visits, it failed at the extremes.
For clinical cases where the appropriate response would have been to tell people to stay home, ChatGPT recommended a doctor’s appointment 64.8% of the time. While not dangerous, this may contribute to unnecessary healthcare utilization in an already overwhelmed health system, not to mention asking people to spend time and money they may not have.
But more concerning was the under-triage at the other extreme: of the clinical cases that required emergent evaluation in the ER, ChatGPT Health recommended they go to the ER in only half of cases.
Digging into which cases the AI tool was getting wrong was revealing — while it performed reasonably well with “textbook emergencies” like stroke and anaphylaxis, it did less well with emergencies that were still early in their clinical course such as the beginning of diabetic ketoacidosis (a life-threatening complication of diabetes) or the beginning of an asthma exacerbation. A good ER doctor recognizes when you are about to go down a hill of death, and stops it. ChatGPT, apparently, can only tell once you’re going full speed down the hill.
The study also revealed how extra, irrelevant information can distract ChatGPT. They tested a case of a patient with suicidal thoughts, where a suicidal crisis guardrail is supposed to fire, urging the person to seek help and providing information for the 988 Suicide and Crisis Lifeline. When just the relevant patient information was provided, the appropriate suicide crisis guardrail fired correctly 100% of the time. BUT, if the scenario also included that the suicidal patient had normal routine lab work, ChatGPT came up with a different answer: your labs are normal and don’t explain these thoughts, and the suicide crisis guardrail failed to fire.
This is quite bad. Over 230 million people across the world ask ChatGPT questions about health every week, according to their website. While it’s true that AI tools have done reasonably well on medical student exams, this study reveals what every fresh medical graduate knows: the ability to pass an exam does not translate to an ability to care for patients. Boards exam questions provide predictable, cookie cutter scenarios designed to teach patterns. But real, breathing humans don’t always follow the textbook.
For simple questions, chatbots can be amazing tools. But when it comes to deciding if you need medical care, I still recommend talking to an old-fashioned human doctor.
Kristen Panthagani, MD, PhD, is completing a combined emergency medicine residency and research fellowship focusing on health literacy and communication. In her free time, she is the creator of the medical blog You Can Know Things, available on Substack and youcanknowthings.com. You can also find her on Instagram and Threads. Views expressed belong to KP, not her employer.
