Organizations: The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center · University of North Carolina at Chapel Hill · University of Illinois Urbana-Champaign
Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn, yet how they behave under multi-turn evidence accumulation closer to real clinical reasoning remains unexplored. We introduce MINT (Medical Incremental N-Turn Benchmark), a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases with clinically labeled evidence shards, controlled turn granularity, and information-preserving decomposition. Through systematic evaluation of 11 LLMs on MINT, we uncover three persistent behavioral patterns that significantly impact diagnostic decisions: (1) intent to answer, models rush to answer before sufficient evidence has been observed, with over 55% of answers committed within the first two turns; (2) self-correction, incorrect-to-correct answer revisions occur at up to 10.6 times the rate of correct-to-incorrect flips, revealing a latent capacity for self-correction that premature commitment forecloses; and (3) strong lures, clinically salient information such as laboratory results trigger premature answering even when models are explicitly instructed to wait. We translate these findings into clinically actionable guidance: deferring the diagnostic question to later turns reduces premature answering and improves accuracy at the first point of commitment by up to 62.6%, while reserving salient clinical evidence for later turns prevents a catastrophic accuracy drop of up to 23.3% caused by premature commitment. Our work provides both a controlled evaluation framework and concrete recommendations for improving the reliability of LLMs in multi-turn medical diagnosis.
Figures & tables
Figure 1: Overview of MINT benchmark construction and evaluation settings.
MedQA
Derm-Public
Derm-Private
MedBullets
MedMCQA
Single-turn
Multi-turn
Single-turn
Multi-turn
Single-turn
Multi-turn
Single-turn
Multi-turn
Single-turn
Multi-turn
Q-First
Q-Last
Q-First
Q-Last
Q-First
Q-Last
Q-First
Q-Last
Q-First
Q-Last
Model
Full
Conc.
Ini.
Final
Abs.
Ans.
Abs.
Full
Conc.
Ini.
Final
Abs.
Ans.
Abs.
Full
Conc.
Ini.
Final
Abs.
Ans.
Abs.
Full
Conc.
Ini.
Final
Abs.
Ans.
Abs.
Full
Conc.
Ini.
Final
Abs.
Ans.
Abs.
Open-Source General LLMs
Qwen2.5-3B
53.3
51.3
36.2
29.8
27.2
50.2
0.0
64.0
69.0
39.1
28.1
36.0
60.0
0.0
74.8
75.8
48.0
29.9
22.2
66.7
0.0
40.3
41.9
33.3
26.7
27.4
40.3
0.0
73.0
70.7
47.5
44.5
21.3
68.4
0.0
Qwen3-4B
63.8
65.2
53.0
58.8
8.5
58.9
0.2
82.0
79.0
67.0
74.0
0.0
79.0
0.0
87.9
84.8
58.8
73.2
2.0
86.9
0.0
48.4
48.4
42.9
44.6
9.7
56.5
0.0
84.1
80.6
58.7
71.9
4.0
77.3
0.0
Table 1: Performance (%) under two single-turn settings, Full and Conc. , and two multi-turn settings, Ask-Question-First ( Q-First ) and Ask-Question-Last ( Q-Last ). Under Q-First , we report Ini. (accuracy at the first turn where the model answers), Final (accuracy at the final turn after all evidence shards are revealed), and Abs. (abstention rate). Under Q-Last , we report Ans. (final-answer accuracy when the question is revealed only at the last turn). Conc. denotes the concatenated-shards single-turn input. Cell background color indicates the change relative to the same model’s Full setting: darker red indicates a larger decrease, and darker green indicates a larger increase.
Figure 2: Diagnostic impatience across information accumulation and model types. Colors denote Ay hold , Ay correct , and Ay incorrect responses. (a) Distribution of initial-answer turns under different conversation lengths (N=4, 8, 12, 16) for GPT-5-mini. (b) Turn-level distribution of hold, correct, and incorrect responses for GPT-5-mini by number of shards seen. (c) Guess rates across models.
Model
Avg Ini. Turn
Abs
Ini.
Full
Δ
MedGemma-27B
7.2±2.1
46.7
77.3
86.6
-9.3
Claude Sonnet 4.6
6.5±2.2
22.0
90.1
96.5
-6.4
Qwen3-4B
4.9±2.5
7.1
56.8
77.5
-20.7
o4-mini
4.1±1.7
2.2
82.0
96.1
-14.1
GPT-5-mini
3.7±1.4
1.6
80.4
96.1
-15.7
GPT-OSS-20B
3.1±1.2
1.1
67.8
90.0
-22.2
Table 2: Comparison of model performance in the 9-turn setting. We report the average turn at which each model produces its first answer ( μ±σ ), the abstention rate, the accuracy of the model’s initial answer ( Ini. ), the accuracy when all information is provided ( Full ), and the accuracy drop from Full to Ini. Δ .
MedQA
Derm-Public
Derm-Private
MedBullets
MedMCQA
# Turns
Q.F
FR
T2F
F2T
Q.L
Q.F
FR
T2F
F2T
Q.L
Q.F
FR
T2F
F2T
Q.L
Q.F
FR
T2F
F2T
Q.L
Q.F
FR
T2F
F2T
Q.L
4
80.7
20.7
0.8
16.7
96.5
82.9
15.0
1.2
11.2
92.7
90.6
10.6
0.0
9.4
100.0
63.8
30.9
3.6
25.5
80.3
85.0
17.2
1.3
14.6
98.1
8
76.3
27.0
1.5
20.3
94.9
79.3
22.0
2.4
17.1
95.1
90.1
13.6
0.0
8.6
100.0
55.9
40.7
3.4
28.8
78.3
84.1
19.2
0.6
14.7
98.1
12
77.2
28.9
1.6
19.8
93.8
83.1
16.9
1.2
10.8
92.8
84.7
17.6
0.0
15.3
100.0
60.0
34.5
1.8
20.0
76.4
86.1
20.5
2.0
12.6
98.0
16
77.5
27.3
1.1
19.2
95.8
80.7
21.7
2.4
14.5
94.0
83.5
18.8
0.0
15.3
100.0
59.3
39.0
3.4
23.7
79.7
87.3
18.4
1.3
11.4
98.1
Table 3: Performance under different total numbers of turns across five medical benchmarks. Q.F (Ask Question First) reports accuracy at the model’s initial answer, and Q.L (Ask Question Last) reports final diagnostic accuracy when the question is revealed only after all information has been provided. FR denotes flip rate, T2F denotes true-to-false rate, and F2T denotes false-to-true rate.
Figure 3: Proportion of Correct, Incorrect, and Hold responses across six turns for 11 models. For most models, hold responses decrease over turns while the proportion of correct answers increases by the final turn.
MedQA
Derm-Public
Derm-Private
MedBullets
MedMCQA
Model
FR
T2F
F2T
RR
FR
T2F
F2T
RR
FR
T2F
F2T
RR
FR
T2F
F2T
RR
FR
T2F
F2T
RR
Open-Source General LLMs
Qwen2.5-3B
52.2
16.0
9.6
0.60
53.1
20.3
9.4
0.46
50.7
22.1
3.9
0.18
46.7
17.8
11.1
0.62
45.3
14.6
11.7
0.80
Qwen3-4B
27.0
4.5
10.4
2.31
13.0
0.0
7.0
N/A
19.6
0.0
14.4
N/A
17.9
1.8
3.6
2.00
25.8
1.8
15.0
8.33
OLMo-2-7B
93.0
19.2
17.6
0.92
87.0
19.0
23.0
1.21
94.9
19.2
26.3
1.37
93.6
6.5
11.3
1.74
92.0
16.7
24.1
1.44
GPT-OSS-20B
52.3
3.5
30.9
8.83
37.0
2.0
24.0
12.00
40.4
1.0
31.3
31.30
54.8
3.2
27.4
8.56
35.8
1.2
19.7
16.42
Table 4: Answer-revision behavior across five medical benchmarks. FR denotes flip rate, T2F denotes true-to-false rate, F2T denotes false-to-true rate, and RR denotes restoration rate, computed as F2T/T2F , under the multi-turn Q-First setting. When T2F is 0, RR is N/A .
Figure 4: Effects of lab-result timing on response behavior and self-correction for GPT-5 mini in 10-turn cases. (a) Distribution of the first-answer turn under early, middle, and late lab-result placement, shown separately for lab-dependent and non-lab-dependent diseases. (b) Turn-level response composition for each lab-result order, showing correct, incorrect, and hold states across turns. (c) Heatmaps of False-to-True (left) and True-to-False (right) transitions by window size/turn gap, stratified by disease type and lab-result placement.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
MINT
N
Mean/Turn
Median/Turn
Len./Case
Lab/Shard
MedQA ( Johri et al., 2025 )
600
9.92
10
128.39
249 (41.5%)
MedMCQA ( Pal et al., 2022 )
174
8.75
8
88.98
86 (49.4%)
Derm-Public ( Johri et al., 2025 )
100
7.79
8
155.29
8 (8.0%)
Derm-Private ( Johri et al., 2025 )
99
7.75
8
139.31
7 (7.1%)
MedBullets ( Chen et al., 2025 )
62
10.74
11
169.63
20 (32.3%)
Overall
1035
9.36
9
124.63
370 (35.7%)
Appendix
Table 5: Descriptive statistics of the source dataset.
Figure 5: (a) Running answer states across turns for GPT-5-mini under different shard counts ( N=4,8,12,16 ). As more evidence is revealed, the model’s current answer state can still improve, with some initially incorrect states later corrected. (b) Cumulative first-answer outcomes across turns. Most first committed answers occur early, and many of these early commitments are incorrect.
Figure 6: Distribution of lab-result cases by total turn count and by number of lab results.
Disease
GPT-5-mini
Claude Sonnet 4.6
Qwen3-4B
MedGemma-27B
Early
Middle
Late
Early
Middle
Late
Early
Middle
Late
Early
Middle
Late
Cardiovascular System
63.64
90.91
90.91
45.45
63.64
63.64
54.55
63.64
63.64
27.27
36.36
27.27
Dermatology
54.55
63.64
90.91
54.55
45.45
63.64
18.18
27.27
27.27
45.45
54.55
27.27
Endocrinology
86.11
88.89
80.56
72.22
94.44
75.00
36.11
41.67
44.44
36.11
33.33
38.89
Gastrointestinal System
79.17
75.00
83.33
75.00
83.33
58.33
54.17
45.83
54.17
45.83
58.33
54.17
Hematology and Oncology
87.88
81.82
87.88
81.82
84.85
72.73
60.61
57.58
48.48
42.42
48.48
36.36
Appendix
Table 6: First-answer accuracy (%) under Ask Question First , broken down by disease and lab-result order. Results are shown for Early , Middle , and Late lab-result placement, with the number of turns fixed at 10. Bold disease names denote lab-dependent diseases.
Model
Order
Ask Q First
Ask Q Last
Ini.
Final
Abs.
Final
Abs.
GPT-5-mini
Early
76.83
95.12
0.00
96.34
0.00
Middle
77.64
94.31
0.40
95.93
0.40
Late
77.11
96.79
0.40
95.58
0.40
Claude Sonnet 4.6
Early
71.54
80.89
7.23
93.50
0.40
Middle
75.61
78.46
11.65
90.24
0.00
Appendix
Table 7: Overall performance (%) under different lab-result orders. Results are shown for Early , Middle , and Late lab-result placement. We report initial-answer accuracy, final-answer accuracy, and abstention rate for Ask Question First , and final-answer accuracy and abstention rate for Ask Question Last .
Figure 7: Representative response trajectories under different lab-result orders. For GPT-5-mini on 10-turn cases, each row shows an example case trajectory when the key lab result is introduced early, middle, or late. Cells indicate whether the model gives no answer (N), a correct answer (T), or an incorrect answer (F) at each turn; outlined cells mark the turn where the lab result is revealed.
Figure 8: Representative behavioral patterns observed in Claude Sonnet 4.6. Left: Rush to Answer . (a) The model guesses at Turn 1 with no clinical context, and (b) it answers at an early turn with minimal information. Middle: Self-correction . The model initially answers incorrectly but revises to the correct option over subsequent turns. Right: Lure to Answer . The same question under two lab result placements (early vs. middle, marked with Lab badges); early placement triggers earlier responses. Green and red borders indicate correct and incorrect answers, respectively.
Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs: ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence. Together, these failures reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM-environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model's hypothesis converges toward the correct diagnosis, and Reasoning-Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs.
Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang +9
1National Chengchi University · 2Georgetown University · University of Michigan +3
Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.
Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz +2
Technical University of Munich · Department of Internal Medicine, UT Southwestern
Large Language Models (LLMs) have demonstrated impressive capabilities across various specialist domains and have been integrated into high-stakes areas such as medicine. However, as existing medical-related benchmarks rarely stress-test the long-context memory, interference robustness, and safety defense required in practice. To bridge this gap, we introduce MedMT-Bench, a challenging medical multi-turn instruction following benchmark that simulates the entire diagnosis and treatment process. We construct the benchmark via scene-by-scene data synthesis refined by manual expert editing, yielding 400 test cases that are highly consistent with real-world application scenarios. Each test case has an average of 22 rounds (maximum of 52 rounds), covering 5 types of difficult instruction following issues. For evaluation, we propose an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations with a human-LLM agreement of 91.94%. We test 17 frontier models, all of which underperform on MedMT-Bench (overall accuracy below 60.00%), with the best model reaching 59.75%. MedMT-Bench can be an essential tool for driving future research towards safer and more reliable medical AI. The benchmark is available in https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material