Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

Jinrui Fang; Runhan Chen; Xu Yang; Jian Yu; Jiawei Xu; Ashwin Vinod; Wenqi Shi; Tianlong Chen

cs.CLApr 6, 2026

Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

Authors: Jinrui Fang, Runhan Chen, Xu Yang, Jian Yu, Jiawei Xu, Ashwin Vinod, Wenqi Shi, Tianlong Chen,

Organizations: The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center · University of North Carolina at Chapel Hill ·

Abstract

Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn, yet how they behave under multi-turn evidence accumulation closer to real clinical reasoning remains unexplored. We introduce MINT (Medical Incremental N-Turn Benchmark), a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases with clinically labeled evidence shards, controlled turn granularity, and information-preserving decomposition. Through systematic evaluation of 11 LLMs on MINT, we uncover three persistent behavioral patterns that significantly impact diagnostic decisions: (1) intent to answer, models rush to answer before sufficient evidence has been observed, with over 55% of answers committed within the first two turns; (2) self-correction, incorrect-to-correct answer revisions occur at up to 10.6 times the rate of correct-to-incorrect flips, revealing a latent capacity for self-correction that premature commitment forecloses; and (3) strong lures, clinically salient information such as laboratory results trigger premature answering even when models are explicitly instructed to wait. We translate these findings into clinically actionable guidance: deferring the diagnostic question to later turns reduces premature answering and improves accuracy at the first point of commitment by up to 62.6%, while reserving salient clinical evidence for later turns prevents a catastrophic accuracy drop of up to 23.3% caused by premature commitment. Our work provides both a controlled evaluation framework and concrete recommendations for improving the reliability of LLMs in multi-turn medical diagnosis.

Figures & tables

Figure 1: Overview of MINT benchmark construction and evaluation settings.

	MedQA							Derm-Public							Derm-Private							MedBullets							MedMCQA
	Single-turn		Multi-turn					Single-turn		Multi-turn					Single-turn		Multi-turn					Single-turn		Multi-turn					Single-turn		Multi-turn
			Q-First			Q-Last				Q-First			Q-Last				Q-First			Q-Last				Q-First			Q-Last				Q-First			Q-Last
Model	Full	Conc.	Ini.	Final	Abs.	Ans.	Abs.	Full	Conc.	Ini.	Final	Abs.	Ans.	Abs.	Full	Conc.	Ini.	Final	Abs.	Ans.	Abs.	Full	Conc.	Ini.	Final	Abs.	Ans.	Abs.	Full	Conc.	Ini.	Final	Abs.	Ans.	Abs.
Open-Source General LLMs
Qwen2.5-3B	53.3	51.3	36.2	29.8	27.2	50.2	0.0	64.0	69.0	39.1	28.1	36.0	60.0	0.0	74.8	75.8	48.0	29.9	22.2	66.7	0.0	40.3	41.9	33.3	26.7	27.4	40.3	0.0	73.0	70.7	47.5	44.5	21.3	68.4	0.0
Qwen3-4B	63.8	65.2	53.0	58.8	8.5	58.9	0.2	82.0	79.0	67.0	74.0	0.0	79.0	0.0	87.9	84.8	58.8	73.2	2.0	86.9	0.0	48.4	48.4	42.9	44.6	9.7	56.5	0.0	84.1	80.6	58.7	71.9	4.0	77.3	0.0

Table 1: Performance (%) under two single-turn settings, Full and Conc. , and two multi-turn settings, Ask-Question-First ( Q-First ) and Ask-Question-Last ( Q-Last ). Under Q-First , we report Ini. (accuracy at the first turn where the model answers), Final (accuracy at the final turn after all evidence shards are revealed), and Abs. (abstention rate). Under Q-Last , we report Ans. (final-answer accuracy when the question is revealed only at the last turn). Conc. denotes the concatenated-shards single-turn input. Cell background color indicates the change relative to the same model’s Full setting: darker red indicates a larger decrease, and darker green indicates a larger increase.

Figure 2: Diagnostic impatience across information accumulation and model types. Colors denote Ay hold , Ay correct , and Ay incorrect responses. (a) Distribution of initial-answer turns under different conversation lengths (N=4, 8, 12, 16) for GPT-5-mini. (b) Turn-level distribution of hold, correct, and incorrect responses for GPT-5-mini by number of shards seen. (c) Guess rates across models.

Model	Avg Ini. Turn	Abs	Ini.	Full	$\Delta$
MedGemma-27B	$7.2\!\pm\!2.1$	46.7	77.3	86.6	-9.3
Claude Sonnet 4.6	$6.5\!\pm\!2.2$	22.0	90.1	96.5	-6.4
Qwen3-4B	$4.9\!\pm\!2.5$	7.1	56.8	77.5	-20.7
o4-mini	$4.1\!\pm\!1.7$	2.2	82.0	96.1	-14.1
GPT-5-mini	$3.7\!\pm\!1.4$	1.6	80.4	96.1	-15.7
GPT-OSS-20B	$3.1\!\pm\!1.2$	1.1	67.8	90.0	-22.2

Table 2: Comparison of model performance in the 9-turn setting. We report the average turn at which each model produces its first answer (

\mu\pm\sigma

), the abstention rate, the accuracy of the model’s initial answer ( Ini. ), the accuracy when all information is provided ( Full ), and the accuracy drop from Full to Ini.

\Delta

	MedQA					Derm-Public					Derm-Private					MedBullets					MedMCQA
# Turns	Q.F	FR	T2F	F2T	Q.L	Q.F	FR	T2F	F2T	Q.L	Q.F	FR	T2F	F2T	Q.L	Q.F	FR	T2F	F2T	Q.L	Q.F	FR	T2F	F2T	Q.L
4	80.7	20.7	0.8	16.7	96.5	82.9	15.0	1.2	11.2	92.7	90.6	10.6	0.0	9.4	100.0	63.8	30.9	3.6	25.5	80.3	85.0	17.2	1.3	14.6	98.1
8	76.3	27.0	1.5	20.3	94.9	79.3	22.0	2.4	17.1	95.1	90.1	13.6	0.0	8.6	100.0	55.9	40.7	3.4	28.8	78.3	84.1	19.2	0.6	14.7	98.1
12	77.2	28.9	1.6	19.8	93.8	83.1	16.9	1.2	10.8	92.8	84.7	17.6	0.0	15.3	100.0	60.0	34.5	1.8	20.0	76.4	86.1	20.5	2.0	12.6	98.0
16	77.5	27.3	1.1	19.2	95.8	80.7	21.7	2.4	14.5	94.0	83.5	18.8	0.0	15.3	100.0	59.3	39.0	3.4	23.7	79.7	87.3	18.4	1.3	11.4	98.1

Table 3: Performance under different total numbers of turns across five medical benchmarks. Q.F (Ask Question First) reports accuracy at the model’s initial answer, and Q.L (Ask Question Last) reports final diagnostic accuracy when the question is revealed only after all information has been provided. FR denotes flip rate, T2F denotes true-to-false rate, and F2T denotes false-to-true rate.

Figure 3: Proportion of Correct, Incorrect, and Hold responses across six turns for 11 models. For most models, hold responses decrease over turns while the proportion of correct answers increases by the final turn.

	MedQA				Derm-Public				Derm-Private				MedBullets				MedMCQA
Model	FR	T2F	F2T	RR	FR	T2F	F2T	RR	FR	T2F	F2T	RR	FR	T2F	F2T	RR	FR	T2F	F2T	RR
Open-Source General LLMs
Qwen2.5-3B	52.2	16.0	9.6	0.60	53.1	20.3	9.4	0.46	50.7	22.1	3.9	0.18	46.7	17.8	11.1	0.62	45.3	14.6	11.7	0.80
Qwen3-4B	27.0	4.5	10.4	2.31	13.0	0.0	7.0	N/A	19.6	0.0	14.4	N/A	17.9	1.8	3.6	2.00	25.8	1.8	15.0	8.33
OLMo-2-7B	93.0	19.2	17.6	0.92	87.0	19.0	23.0	1.21	94.9	19.2	26.3	1.37	93.6	6.5	11.3	1.74	92.0	16.7	24.1	1.44
GPT-OSS-20B	52.3	3.5	30.9	8.83	37.0	2.0	24.0	12.00	40.4	1.0	31.3	31.30	54.8	3.2	27.4	8.56	35.8	1.2	19.7	16.42

Table 4: Answer-revision behavior across five medical benchmarks. FR denotes flip rate, T2F denotes true-to-false rate, F2T denotes false-to-true rate, and RR denotes restoration rate, computed as

\text{F2T}/\text{T2F}

, under the multi-turn Q-First setting. When T2F is 0, RR is N/A .

Figure 4: Effects of lab-result timing on response behavior and self-correction for GPT-5 mini in 10-turn cases. (a) Distribution of the first-answer turn under early, middle, and late lab-result placement, shown separately for lab-dependent and non-lab-dependent diseases. (b) Turn-level response composition for each lab-result order, showing correct, incorrect, and hold states across turns. (c) Heatmaps of False-to-True (left) and True-to-False (right) transitions by window size/turn gap, stratified by disease type and lab-result placement.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

MINT	N	Mean/Turn	Median/Turn	Len./Case	Lab/Shard
MedQA ( Johri et al., 2025 )	600	9.92	10	128.39	249 (41.5%)
MedMCQA ( Pal et al., 2022 )	174	8.75	8	88.98	86 (49.4%)
Derm-Public ( Johri et al., 2025 )	100	7.79	8	155.29	8 (8.0%)
Derm-Private ( Johri et al., 2025 )	99	7.75	8	139.31	7 (7.1%)
MedBullets ( Chen et al., 2025 )	62	10.74	11	169.63	20 (32.3%)
Overall	1035	9.36	9	124.63	370 (35.7%)

Appendix

Table 5: Descriptive statistics of the source dataset.

Figure 5: (a) Running answer states across turns for GPT-5-mini under different shard counts (

N=4,8,12,16

). As more evidence is revealed, the model’s current answer state can still improve, with some initially incorrect states later corrected. (b) Cumulative first-answer outcomes across turns. Most first committed answers occur early, and many of these early commitments are incorrect.

Figure 6: Distribution of lab-result cases by total turn count and by number of lab results.

Disease	GPT-5-mini			Claude Sonnet 4.6			Qwen3-4B			MedGemma-27B
	Early	Middle	Late	Early	Middle	Late	Early	Middle	Late	Early	Middle	Late
Cardiovascular System	63.64	90.91	90.91	45.45	63.64	63.64	54.55	63.64	63.64	27.27	36.36	27.27
Dermatology	54.55	63.64	90.91	54.55	45.45	63.64	18.18	27.27	27.27	45.45	54.55	27.27
Endocrinology	86.11	88.89	80.56	72.22	94.44	75.00	36.11	41.67	44.44	36.11	33.33	38.89
Gastrointestinal System	79.17	75.00	83.33	75.00	83.33	58.33	54.17	45.83	54.17	45.83	58.33	54.17
Hematology and Oncology	87.88	81.82	87.88	81.82	84.85	72.73	60.61	57.58	48.48	42.42	48.48	36.36

Appendix

Table 6: First-answer accuracy (%) under Ask Question First , broken down by disease and lab-result order. Results are shown for Early , Middle , and Late lab-result placement, with the number of turns fixed at 10. Bold disease names denote lab-dependent diseases.

Model	Order	Ask Q First			Ask Q Last
		Ini.	Final	Abs.	Final	Abs.
GPT-5-mini	Early	76.83	95.12	0.00	96.34	0.00
	Middle	77.64	94.31	0.40	95.93	0.40
	Late	77.11	96.79	0.40	95.58	0.40
Claude Sonnet 4.6	Early	71.54	80.89	7.23	93.50	0.40
	Middle	75.61	78.46	11.65	90.24	0.00

Appendix

Table 7: Overall performance (%) under different lab-result orders. Results are shown for Early , Middle , and Late lab-result placement. We report initial-answer accuracy, final-answer accuracy, and abstention rate for Ask Question First , and final-answer accuracy and abstention rate for Ask Question Last .

Figure 7: Representative response trajectories under different lab-result orders. For GPT-5-mini on 10-turn cases, each row shows an example case trajectory when the key lab result is introduced early, middle, or late. Cells indicate whether the model gives no answer (N), a correct answer (T), or an incorrect answer (F) at each turn; outlined cells mark the turn where the lab result is revealed.

Figure 8: Representative behavioral patterns observed in Claude Sonnet 4.6. Left: Rush to Answer . (a) The model guesses at Turn 1 with no clinical context, and (b) it answers at an early turn with minimal information. Middle: Self-correction . The model initially answers incorrectly but revises to the correct option over subsequent turns. Right: Lure to Answer . The same question under two lab result placements (early vs. middle, marked with Lab badges); early placement triggers earlier responses. Green and red borders indicate correct and incorrect answers, respectively.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 395 papers

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
May 8, 2026Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang +9Healthcare Medical Diagnosis
1National Chengchi University · 2Georgetown University · University of Michigan +3
MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
Aug 25, 2026Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz +2Clinical Language Model Evaluation Differential Diagnosis
Technical University of Munich · Department of Internal Medicine, UT Southwestern
MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?
Mar 6, 2026Lin Yang, Yuancheng Yang, Xu Wang +2Long-Context Language Model Evaluation Clinical Language Model Evaluation
ByteDance

May 8, 2026J/K move · Enter open · S save

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang, Nai-Chia Chen+8

1National Chengchi University · 2Georgetown University · University of Michigan+3

Healthcare Medical Diagnosis Clinical Reasoning LLM Agent Training Differential Diagnosis Medical LLMs

Open paper AskPDF arXiv

Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs: ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence. Together, these failures reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM-environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model's hypothesis converges toward the correct diagnosis, and Reasoning-Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs.

Abstract

Clinical Language Model Evaluation

Explore similar work

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?