Evaluation format, not only model capability, shapes measured triage failure in consumer health AI
Authors: David Fraile Navarro, Jialei Sheng, Farah Magrabi, Enrico Coiera
Organizations: Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia · NSW Health, Sydney, NSW, Australia
A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage poses safety risks. Its protocol, however, was an exam-style scaffold (forced A/B/C/D output, knowledge suppression, no clarifying questions) unlike how consumers use health chatbots. We ask how much of the headline error rate is a property of the measurement rather than of the models. In a first, mechanistic study, five frontier LLMs on a 17-scenario bank scored 6.4 points higher under naturalistic patient-style messages than under the constrained scaffold (p=0.015). In a second, faithful replication we ran the authors' own 60 released vignettes through six frontier models under four matched formats, with clinician validation of the rewrites and a blinded clinician audit of the LLM adjudicators. Here the direction reversed: free-text rewrites scored slightly below the exact prompt (78.7% vs 81.8%, p=0.020), removing only the answer scaffold changed little (80.6% vs 81.4%, p=0.81), and naturalistic input with a forced categorical answer beat both the exact prompt (84.4%, p=0.023) and free text (p=0.0001). On the four vignettes defining the original emergency rate, under-triage was 17% with the exact scaffold, 50% in free text, and 21% with a forced letter on the same message; every free-text "under-triage" was a same-day recommendation scored C rather than D, and 60% made escalation conditional on information the patient was asked to check, behavior a single-turn benchmark cannot score. The headline rate did not reproduce as a stable cross-model finding; much of it depends on output format and on mapping prose onto a four-point scale. Each format elicits a separate response, so the design measures format sensitivity, not format versus capability. Safety claims should report sensitivity to input wording, output format and adjudication.
Figures & tables
Matched comparison
Asthma ablation
Model
Constrained
Naturalistic
Δ (pp)
Forced A/B/C/D
Free text
Fisher p
GPT-5.2
64.1%
68.2%
+4.1
4/25
25/25
3.8×10−10
Claude Sonnet 4.6
56.5%
71.2%
+14.7
25/25
25/25
1.00
Claude Opus 4.6
61.8%
72.4%
+10.6
25/25
25/25
1.00
Gemini 3 Flash
63.5%
66.8%
+3.2
6/25
25/25
1.2×10−8
Gemini 3.1 Pro
72.4%
71.8%
−0.6
0/25
25/25
1.6×10−14
Table 1: Study 1. Left: matched constrained (exam-style) versus naturalistic (patient-style) accuracy; 170 matched responses per model, two-judge mean. Right: high-powered asthma ablation ( n=25 per cell) isolating the output format on identical content.
Model
Exact structured
Natural free text
Source-stripped
Natural forced letter
GPT-5.3 Instant
81.7%
74.2% ( −7.5 )
83.3% ( +5.0 )
86.7% ( +5.0 )
GPT-5.4
80.8%
81.7% ( +0.8 )
79.2% ( −0.8 )
85.8% ( +5.0 )
Claude Sonnet 4.6
80.8%
77.1% ( −3.8 )
78.3% ( −3.3 )
87.5% ( +6.7 )
Claude Opus 4.6
85.8%
76.7% ( −9.2 )
78.3% ( −8.3 )
85.0% ( −0.8 )
Gemini 3 Flash
80.8%
81.7% ( +0.8 )
77.5% ( −2.5 )
80.0% ( −0.8 )
Gemini 3.1 Pro
80.8%
80.8% ( 0.0 )
86.7% ( +5.0 )
81.7% ( +0.8 )
Table 2: Study 2: accuracy on the authors’ 60 released reference vignettes, by model and format (six models, two runs per case; one run for source-stripped). Free-text conditions are two-judge means. Parenthesized deltas are against the exact structured accuracy on the rows matched for that comparison (pooled 81.8%, or 81.4% on the 360 source-stripped rows). Bottom rows test the pooled comparison against exact structured (Wilcoxon on model × case cells; exact McNemar on rows, first judge).
Case
Presentation
Exact structured
Natural free text
Source-stripped
Forced letter
E13
DKA, with labs
12/12
12/12
6/6
12/12
F13
DKA, symptoms only
12/12
3/12 [6/12]
1/6 [5/6]
12/12
E9
Asthma, with data
12/12
7/12 [8/12]
5/6 [6/6]
12/12
F9
Asthma, symptoms only
4/12
0/12
1/6
2/12
Emergency under-triage rate
16.7%
50.0%
35.4%
20.8%
Table 3: Study 2: responses triaged to “emergency department now” on the four vignettes with an unambiguous D gold label (the vignettes over which the original study’s 51.6% is computed), pooled over six models (two runs each; one run for source-stripped). Per-case cells for free-text conditions show the first judge, with the second judge in brackets where it differs; the under-triage rate pools all four rows and both judges’ labels.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Case ID
Scenario
Scenario type
Gold triage
case_01
Exertional chest pain (stable angina)
Symptoms-history dominant
B
case_02
Acute chest pain (possible ACS)
Symptoms-history dominant
A
case_03
Tension headache
Symptoms-history dominant
D
case_04
Acute appendicitis
Symptoms-history dominant
A
case_05
Allergic rhinitis
Symptoms-history dominant
D
case_06
Iron deficiency anemia
With objective data
B
Appendix
Table 4: The 17-scenario partial replication bank used in Study 1. Scenarios and prompt variants were generated with LLMs under the direction of a physician author. “Scenario type” records whether the case was generated with objective data available to the model or as a symptoms-history-dominant scenario.
Model
Original structured
Patient realistic
Patient minimal
GPT-5.2
56/85 (65.9%)
57/85 (67.1%)
52/85 (61.2%)
Claude Sonnet 4.6
55/85 (64.7%)
50/85 (58.8%)
46/85 (54.1%)
Claude Opus 4.6
55/85 (64.7%)
55/85 (64.7%)
50/85 (58.8%)
Gemini 3 Flash
57/85 (67.1%)
55/85 (64.7%)
53/85 (62.4%)
Gemini 3.1 Pro
66/85 (77.6%)
64/85 (75.3%)
59/85 (69.4%)
Appendix
Table 5: Study 1 constrained replication accuracy by model and prompt format (correct/scored). The Study 1 bank uses the letter A for the emergency level; pooled: original structured 289/425 (68.0%), patient realistic 281/425 (66.1%), patient minimal 260/425 (61.2%).
Model
User prompt
Constrained
Natural
Δ (pp)
GPT-5.2
Patient realistic
67.1%
70.0%
+2.9
GPT-5.2
Patient minimal
61.2%
66.5%
+5.3
Claude Sonnet 4.6
Patient realistic
58.8%
70.6%
+11.8
Claude Sonnet 4.6
Patient minimal
54.1%
71.8%
+17.6
Claude Opus 4.6
Patient realistic
64.7%
76.5%
+11.8
Claude Opus 4.6
Patient minimal
58.8%
68.2%
+9.4
Appendix
Table 6: Study 1 matched constrained-versus-natural comparison by model and user prompt format (two-judge mean).
Model
DKA (all formats)
Asthma structured
Asthma realistic
Asthma minimal
GPT-5.2
15/15
0/5
4/5
5/5
Claude Sonnet 4.6
15/15
5/5
5/5
5/5
Claude Opus 4.6
15/15
5/5
0/5
5/5
Gemini 3 Flash
15/15
0/5
2/5
5/5
Gemini 3.1 Pro
15/15
0/5
1/5
5/5
Appendix
Table 7: Study 1 key-case behavior in the constrained replication. DKA is pooled across all three prompt formats; asthma is shown by prompt format.
Model
No added restriction
Base-on-message-only
Full paper template
GPT-5.2
7/10
5/10
6/10
Gemini 3 Flash
5/10
6/10
8/10
Claude Sonnet 4.6
10/10
10/10
10/10
Claude Opus 4.6
10/10
10/10
10/10
Appendix
Table 8: Study 1 sensitivity analysis isolating the “base your answer only on the information in this message” instruction (two-case subset; correct/scored).
Model
Structured
+ Knowledge
+ No clarifying
+ 150-word
Free-text
Full
baseline
suppression
questions
cap
output
template
GPT-5.2
5/10
5/10
5/10
5/10
10/10
5/10
Gemini 3 Flash
6/10
6/10
7/10
8/10
10/10
6/9
Claude Sonnet 4.6
10/10
10/10
10/10
10/10
10/10
9/10
Claude Opus 4.6
10/10
10/10
10/10
10/10
10/10
10/10
Appendix
Table 9: Study 1 one-factor ablation on the DKA and asthma subset. Each column adds or removes one prompt constraint relative to the structured baseline (correct/scored).
Model
Forced A/B/C/D
Free-text output
Full template
GPT-5.2
4/25 (16%)
25/25 (100%)
1/25 (4%)
Gemini 3 Flash
6/25 (24%)
25/25 (100%)
12/22 (54.5%)
Gemini 3.1 Pro
0/25 (0%)
25/25 (100%)
0/25 (0%)
Claude Sonnet 4.6
25/25 (100%)
25/25 (100%)
23/25 (92%)
Claude Opus 4.6
25/25 (100%)
25/25 (100%)
25/25 (100%)
Appendix
Table 10: Study 1 full high-powered asthma ablation ( n=25 per cell). The full-template column reintroduces the entire paper-style constraint bundle.
Model
F9 (all)
F9 (no anchor / anchor)
F13 (all)
F13 (no anchor / anchor)
GPT-5.2
16/16
8/8 / 8/8
16/16
8/8 / 8/8
Claude Opus 4.6
7/16
5/8 / 2/8
9/16
8/8 / 1/8
Appendix
Table 11: Study 1 prompt-faithful sweep on the authors’ released symptom-only failure rows (asthma F9, DKA F13), all 16 race/gender/anchor/barrier variants. Counts are emergency recommendations on the exact released prompt.
Model
F9 exact
F9 natural
F9 user
F13 exact
F13 natural
F13 user
GPT-5.2
A
B; B
B; B
A
D/A; B
B; D/A
Claude Sonnet 4.6
B
B; B
B; B
A
A; A
B; B
Claude Opus 4.6
B
B; B
B; B
C/B
B; B
D/A; B
Gemini 3 Flash
A
A; B
B; B
A
A; B
A; B
Gemini 3.1 Pro
B
B; B
B; B
A
B; B
B; B
Appendix
Table 12: Study 1 prompt-faithful naturalization ladder on the released symptom-only failure rows (two repeats per cell). exact is the released prompt (single run), natural keeps the patient’s question but removes the scaffold, user strips the wrapper further. Cells show dual-judge adjudicated labels by repeat; split labels indicate judge disagreement. In the released scale A = emergency care now.
Model / aggregate
Constrained
Naturalistic
Δ (pp)
GPT-5.3 Instant (post hoc)
72.9%
81.5%
+8.5
Five models (pre-specified)
63.6%
70.1%
+6.4
Six models (exploratory)
65.2%
72.0%
+6.8
Appendix
Table 13: Study 1 exploratory six-model comparison adding GPT-5.3 Instant post hoc (170 matched responses per model).
Configuration
Overall
With data ( n=30 )
Symptoms only ( n=30 )
GPT-5.3 Instant
90.0%
93.3%
86.7%
GPT-5.4 (reasoning xhigh)
88.3%
93.3%
83.3%
Claude Opus 4.6, no thinking
86.7%
90.0%
83.3%
Claude Sonnet 4.6, no thinking
85.0%
86.7%
83.3%
Claude Sonnet 4.6, thinking
83.3%
86.7%
80.0%
Claude Opus 4.6, thinking
81.7%
86.7%
76.7%
Appendix
Table 14: Prompt-scaffold extension on the 60 clinician-validated naturalistic rewrites (one run per case). “Thinking” denotes extended reasoning enabled.
Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.
People increasingly turn to general-purpose AI chatbots for advice about emotional and mental health problems, but the ability of these systems to recognize and appropriately triage psychiatric emergencies remains under-characterized. We evaluated psychiatric triage performance in 15 frontier AI chatbots using 112 clinical vignettes spanning four urgency levels, from routine care to immediate emergency assessment. In each trial (1680 total), a chatbot received a single user message conveying all triage-relevant information from one vignette and recommended a timeframe for care. The primary outcome was emergency under-triage; secondary outcomes included triage accuracy and the direction of errors. Vignettes and user messages were generated using a clinician-verified LLM pipeline. Across 415 emergency trials, 23 were under-triaged (5.5%; 95% CI 1.8-15.9). Overall accuracy, averaged across urgency levels, ranged from 42.0% to 71.8% across chatbots and was lowest for intermediate cases (19.6%; 95% CI 11.7-28.1). Every chatbot showed a net over-triage bias; overall, 763 of 786 incorrect assignments (97.1%) were more urgent than the prespecified triage level. The error pattern was similar when predictions were assessed against clinician ratings: 35 of 430 trials involving vignettes rated as emergencies by at least 75% of clinicians were under-triaged (8.1%). AI chatbots recognized most psychiatric emergencies but still missed clinically important cases and frequently over-triaged less urgent presentations. Further evaluations should examine how triage performance changes when clinically relevant information must be elicited through conversation.
Veith Weilnhammer, Lennart Luettgau, Christopher Summerfield +5
Max Planck UCL Centre for Computational Psychiatry and Ageing Research, London, UK · UK AI Security Institute · Department of Experimental Psychology, University of Oxford, Oxford, UK +2
Patient-voiced clinical-triage benchmarks report high under-triage rates for consumer LLMs for constrained multiple-choice output, yet the same cases score differently with free-text. We ask whether output format changes the model's clinical representation or only the mapping from a preserved representation to an answer. Using sparse-autoencoder (SAE) features in Gemma 3 4B/12B IT and Qwen3-8B, we find the same medical features fire on the shared clinical narrative under both formats but go {silent} at the multiple-choice decision token in all the cases at every model. Three independent methods (natural-language autoencoder verbalization, decision-token logit attribution, and top-feature characterization) agree that scaffold and format features, but not medical features, drive the decision logits. Behaviorally, the multiple-choice penalty inverts under both structured and natural-language input, option-order shuffle rules out positional bias, and the gap is dominated by off-by-one decision (the model picks an adjacent acuity letter to the gold answer) rather than knowledge failure. Thus, the failure originates in the output format and not in the clinical representation.
David Fraile Navarro, Berardino Como, Jialei Sheng +2
Macquarie University, Sydney, Australia · Politecnico di Bari, Bari, Italy · NSW Health, Sydney, Australia +1
Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.
Yining Hua, Hongbin Na, Cyrus Ayubcha
1Harvard University · University of Technology Sydney