How reliable is ChatGPT Health for emergencies?
Study raises safety concerns about a health-trained chatbot
A recent clinical evaluation found that a health-specific version of a large language model frequently underestimated the severity of medical emergencies, under-triaging a substantial share of cases that required urgent care. The findings suggest the tool may not reliably distinguish between routine issues and conditions that need immediate medical attention.
Key limitations identified
- Under-triage: The model sometimes assessed serious presentations as low-risk, which could delay needed care.
- Missing red flags: It struggled to consistently recognize cues that clinicians use to escalate care, particularly in scenarios with subtle or atypical symptoms.
- Overreliance risk: People using the tool as a substitute for professional triage may be falsely reassured and postpone calling emergency services.
Practical takeaways for users and clinicians
- Seek real-time clinical evaluation when symptoms are severe, sudden, or worsening—calling emergency services or going to an emergency department is still the safest option.
- Use AI tools for information and general advice, not for definitive triage of potentially life‑threatening complaints.
- Clinicians and health systems should treat these models as decision-support adjuncts, not replacements for trained triage nurses or emergency physicians.
What regulators and developers should focus on
- Independent testing against established triage standards and transparent reporting of failure rates.
- Clear user warnings and design defaults that escalate ambiguous or concerning inputs toward higher urgency recommendations.
- Ongoing post-deployment monitoring to capture real-world performance and unintended harms.
The technology shows promise for expanding access to health information, but current evidence indicates it should not be relied on to judge emergency needs. Users must err on the side of caution when symptoms could signal a serious condition.