MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

Lin Yang; Yuancheng Yang; Xu Wang; Changkun Liu; Haihua Yang

cs.CLMar 6, 2026

MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

Authors: Lin Yang, Yuancheng Yang, Xu Wang, Changkun Liu, Haihua Yang

Organizations: ByteDance

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across various specialist domains and have been integrated into high-stakes areas such as medicine. However, as existing medical-related benchmarks rarely stress-test the long-context memory, interference robustness, and safety defense required in practice. To bridge this gap, we introduce MedMT-Bench, a challenging medical multi-turn instruction following benchmark that simulates the entire diagnosis and treatment process. We construct the benchmark via scene-by-scene data synthesis refined by manual expert editing, yielding 400 test cases that are highly consistent with real-world application scenarios. Each test case has an average of 22 rounds (maximum of 52 rounds), covering 5 types of difficult instruction following issues. For evaluation, we propose an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations with a human-LLM agreement of 91.94%. We test 17 frontier models, all of which underperform on MedMT-Bench (overall accuracy below 60.00%), with the best model reaching 59.75%. MedMT-Bench can be an essential tool for driving future research towards safer and more reliable medical AI. The benchmark is available in https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material

Figures & tables

Figure 1: An example of multi-patient information interference in the domain of long context memory and understanding in the pre-diagnosis stage.

Figure 2: Test examples in the remaining four multi-turn difficult dimensions.

Attributes	Number
# Categories
- LCMU	152
- RCI	41
- SCASD	63
- IC	98
- MIRI	46

Table 1: Detailed statistics for the MedMT-Bench.

Figure 3: The distribution of complex multi-turn instruction following problems.

Panel of Figure 4 — Figure 4: Example of test points for synthesis.

Figure 5: Workflow of MedMT-Bench. The upper panel depicts the single-stage data synthesis process, which combines multi-agent conversation synthesis, verification, and manual editing to produce challenging examples targeting specific evaluation dimensions. The lower panel shows the multi-stage, scene-by-scene synthesis pipeline. Subsequent scene test cases build on the multi-turn conversations from the preceding scene and derive inputs via a portrait extraction strategy.

Consistent rate	Auto-Eval without ATP	Auto-Eval with ATP
GPT-5	40.33%	$85.40\%_{\pm 1.2}$
Claude-4-Opus	39.68%	$86.97\%_{\pm 0.8}$
OpenAI o3	37.98%	$87.03\%_{\pm 1.1}$
Gemini-2.5-Pro	41.09%	$91.94\%_{\pm 0.6}$
GPT-4o	39.91%	$86.99\%_{\pm 1.0}$
GPT-4.1	40.93%	$88.96\%_{\pm 0.8}$

Table 2: Consistency rate between automatic and manual evaluation with and without test points (ATP).

Human Evaluation
Model Names	LCMU	RCI	SCASD	IC	MIRI	HC	PSRO	PTRM	Avg
GPT-4o(2024-11)	33.55	21.95	34.92	22.45	47.83	34.36	24.53	23.21	31.50
GPT-4.1(2025-04)	33.55	46.34	50.79	24.49	63.04	43.64	28.3	23.21	38.75
Gemini-2.5-Pro	46.05	60.98	68.25	28.57	54.35	49.48	47.17	37.50	47.75
OpenAI o3(2025-04)	50.00	39.02	57.14	36.73	50.00	49.48	47.17	32.14	46.75
Claude4-Sonnet	57.89	46.34	52.38	48.98	56.52	54.30	50.94	51.79	53.50

Table 3: The accuracy performance (%) of the human and automatic evaluation of 7 frontier models.

Model Names	LCMU	RCI	SCASD	IC	MIRI	HC	PSRO	PTRM	Avg
Qwen3-8B-Instruct	44.44	18.42	35.71	20.27	56.52	36.31	35.85	26.42	34.39
Qwen3-32B-Instruct	55.56	21.05	40.48	24.32	69.57	44.13	45.28	30.19	41.75
Llama3.1-8B-Instruct	17.59	26.32	21.43	6.76	39.13	20.11	16.98	13.21	18.25
Llama3.1-70B-Instruct	29.63	26.32	28.57	9.46	43.48	27.37	26.42	15.09	24.91
Kimi-K2-1T-A32B	50.93	44.79	57.14	28.38	47.83	51.40	33.96	33.96	44.91
GLM-4.5-355B-A32B	44.86	36.84	47.62	28.38	56.52	43.58	39.62	32.69	40.85

Table 4: The accuracy performance (%) of the automatic evaluation of 7 open-source models on the text subset.

Figure 6: Performance comparison of 7 frontier models on different modal and 5 instruction following problems.

Figure 7: The performance distribution of 7 frontier models on the finest-grained multi-turn instruction following problems. Each bar consists of two parts: the solid part represents the performance of the text subset, and the transparent part represents the performance of the whole set after combining the image modal subset.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Data distribution of MedMT-Bench in different turns.

Figure 9: Data distribution of MedMT-Bench in different departments.

Model Names	LCMU	RCI	SCASD	IC	MIRI	Avg
Qwen3-8B-Instruct (w/o thinking)	41.67	13.16	35.71	21.62	60.87	33.33
Qwen3-8B-Instruct (w/ thinking)	44.44	18.42	35.71	20.27	56.52	34.39
Qwen3-32B-Instruct (w/o thinking)	41.67	13.16	47.62	14.86	78.26	34.74
Qwen3-32B-Instruct (w/ thinking)	55.56	21.05	40.48	24.32	69.57	41.75
GLM-4.5-355B-A32B (w/o thinking)	34.26	21.05	50.00	27.03	52.17	34.39
GLM-4.5-355B-A32B (w/ thinking)	44.86	36.84	47.62	28.38	56.52	40.85

Appendix

Table 5: The accuracy performance (%) of 3 open-source models in thinking and non-thinking modes on MedMT-Bench text subset.

Figure 10: Performance comparison of 3 open-source vision models on different modal and multi-turn instruction following problems.

Model Names	LCMU	RCI	SCASD	IC	MIRI	Avg
Qwen2.5-VL-7B-Instruct	22.52	12.20	26.98	12.24	39.13	21.55
Qwen2.5-VL-72B-Instruct	33.77	21.95	34.92	24.49	43.48	33.96
GLM-4.5V-106B-A12B	42.76	14.63	44.44	20.41	32.61	33.50

Appendix

Table 6: The accuracy performance (%) of 3 open-source vision models on MedMT-Bench.

Figure 11: Performance trends of several problem dimensions strongly related to long contexts averaged over all models at different turns.

Figure 12: Performance of 7 closed-source frontier models in different medical departments.

Panel of Figure 13 — Figure 13: The performance distribution of 7 open-source models on the finest-grained multi-turn instruction following problems.

Model Names	Round1	Round2	Round3	Avg
GPT-4o(2024-11)	34.75	35.25	34.50	$34.83_{\pm 0.3}$
GPT-4.1(2025-04)	41.75	42.00	42.25	$42.00_{\pm 0.2}$
Gemini-2.5-Pro	50.50	50.75	49.50	$50.25_{\pm 0.5}$
OpenAI o3(2025-04)	48.75	49.00	48.50	$48.75_{\pm 0.2}$
Claude4-Sonnet	52.25	52.75	53.50	$52.83_{\pm 0.5}$
Claude4-Opus	54.00	54.00	53.75	$53.92_{\pm 0.1}$

Appendix

Table 7: The mean and standard deviation of three evaluations with the temperature set to 0.

Panel of Figure 14 — Figure 14: The modified instruction structure used for automated evaluation shows that the left side changed the order of the output structure and judge requirements modules; the right side changed the order of the different sub-requirements in the judge requirements.

Model Names	tp-1.0	tp-0.5	tp-0	Avg	st-v1	st-v2	st-v3	Avg
GPT-4o(2024-11)	34.50	35.25	34.75	$34.83_{\pm 0.3}$	34.75	35.50	35.5	$35.25_{\pm 0.4}$
GPT-4.1(2025-04)	41.75	42.25	41.75	$41.92_{\pm 0.2}$	41.75	42.25	42.25	$42.08_{\pm 0.2}$
Gemini-2.5-Pro	51.50	49.50	50.50	$50.50_{\pm 0.8}$	50.50	49.00	50.75	$50.08_{\pm 0.8}$
OpenAI o3(2025-04)	48.25	49.50	48.75	$48.83_{\pm 0.5}$	48.75	47.75	49.50	$48.67_{\pm 0.7}$
Claude4-Sonnet	52.00	52.75	52.25	$52.33_{\pm 0.3}$	52.25	52.25	52.75	$52.42_{\pm 0.4}$
Claude4-Opus	54.50	53.25	54.00	$53.92_{\pm 0.5}$	54.00	53.50	54.75	$54.08_{\pm 0.5}$

Appendix

Table 8: Automated evaluation mean and standard deviation under different temperature(tp) settings and different prompt structures(st).

Model Names	Consistent rate
GPT-5-high	89.50%
Claude-4-Opus	91.25%
OpenAI o3(2025-04)	92.25%
Gemini-2.5-Pro	93.00%
GPT-4o(2024-11)	94.50%
GPT-4.1(2025-04)	91.25%

Appendix

Table 9: Automated evaluation mean and standard deviation under different temperature(tp) settings and different prompt structures(st).

Model	Temperature	Top-P	Max Tokens
All models	1	0.7	80k

Appendix

Table 10: Inference temperatures for LLMs evaluated on MedMT-Bench

Long-Context Language Model Evaluation

Long-context language model evaluation measures how reliably models find, reason over, and use information across extended inputs.

See all 66 papers

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Sep 11, 2026Youssef Mohamed, Ahmed Heakl, Qinrong Cui +13Medical Diagnosis Multi-Turn Dialogue Evaluation
MBZUAI · Cairo University · Ain Shams University +2
Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction
Apr 6, 2026Jinrui Fang, Runhan Chen, Xu Yang +9Clinical Language Model Evaluation Medical Diagnosis
The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center +2
Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Jun 10, 2026Hongjian Zhou, Xinyu Zou, Jinge Wu +19LLM Reliability Medical QA
University of Oxford · University of Washington · University College London +1

Sep 11, 2026J/K move · Enter open · S save

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang+12

MBZUAI · Cairo University · Ain Shams University+2

Medical Diagnosis Multi-Turn Dialogue Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.

Abstract

Long-Context Language Model Evaluation

Explore similar work

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

Measuring Epistemic Resilience of LLMs Under Misleading Medical Context