MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Boqi Chen; Xudong Liu; Jiachuan Peng; Marianne Frey-Marti; Bang Zheng; Kyle Lam; Lin Li; Jianing Qiu

cs.CLFeb 25, 2026

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Authors: Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Bang Zheng, Kyle Lam, Lin Li, Jianing Qiu

Organizations: ETH Zurich · MBZUAI · Amazon · University of Oxford · University of Bern · Peking University · Imperial College London

Abstract

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE ( $\it{e.g.}$ , medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Clinicians curate a broad differential diagnosis (DDx) list before determining a final diagnosis (FDx) via evidence synthesis. (b) Models exhibit a substantial gap between DDx coverage rate and FDx accuracy, far exceeding that observed in human experts.

Benchmark	# Images	Multi- lingual	Avg. # Images per Case	Avg. # Evidence Types per Case
VQA-Rad	$204$	✗	$1$	$1$
VQA-Med	$500$	✗	$1$	$1$
Slake	$96$	✓	$1$	$1$
PMC-VQA	$29$ k	✗	$1$	$1$
OmniMedVQA	$118$ k	✗	$1$	$1$
GMAI-MMBench	$21$ k	✗	$1$	$1$

Table 1: Comparison of MEDSYN with existing multimodal medical benchmarks.

Figure 2: Example final diagnosis selection tasks in English (top) and Chinese (bottom) . Colors mark different visual clinical evidence (CE) types referenced in the question, with corresponding expert-derived diagnostic findings; gray denotes textual CE. In our experiment, each CE type is input as either raw images or text findings, not both.

Panel of Figure 3 — Figure 3: (a) Distribution of visual clinical evidence (CE) types. (b) Number of visual CE types per case.

Type	Model	English		Chinese		Overall
Type	Model	DDx Coverage Rate (%)	FDx Selection Acc. (%)	DDx Coverage Rate (%)	FDx Selection Acc. (%)	DDx Coverage Rate (%)	FDx Selection Acc. (%)
Expert	Senior Physician	77.13	72.11	-	-	-	-
Proprietary	Claude Opus 4.5	84.92	64.57	61.11	55.56	82.08	63.49
	GPT-O3	86.15	64.07	64.81	48.15	83.60	62.17
	Gemini 2.5 pro	80.60	62.81	64.81	53.70	78.72	61.72
	GPT-5.2	88.55	60.45	66.04	55.56	85.86	59.87

Table 2: Comparison of differential diagnosis (DDx) coverage rate (%) and final diagnosis (FDx) selection accuracy (%) across models on the English and Chinese subsets, and overall. Best and second-best scores are in bold and underlined , computed separately for proprietary and open-source models. Open-source models are grouped by parameter scale into Large (

\approx

70B), Medium (

\approx

30B), Small (

\approx

7B), and Tiny (

\approx

3B) categories. Clinical evidence from diagnostic investigations is provided as raw images.

Model	Raw Images	Expert-Derived Diagnostic Findings
GPT-O3	64.07	75.38 (+11.31)
GPT-5.2	60.45	75.13 (+14.68)
Qwen2.5-VL (72B)	48.24	65.83 (+17.59)
Qwen3-VL (32B)	49.87	65.58 (+15.71)
InternVL3 (78B)	44.72	63.82 (+19.10)
Qwen3-VL (8B)	40.20	59.05 (+18.85)

Table 3: Final diagnosis accuracy (%) across models on English subset. We compare performance when clinical evidence from diagnostic investigations is provided as raw images versus expert-derived text findings.

Figure 4: Mean final diagnosis accuracy (%) for proprietary models, domain-specific models, and open-source general-purpose models across clinical specialties.

Setting	HuatuoGPT-Vision	Lingshu	Qwen2.5-VL (7B)
Baseline	38.69	36.93	33.92
Remove-Text	42.21 (+3.52 pp)	43.97 (+7.04 pp)	34.42 (+0.50 pp)
Random-Text	39.70 (+1.01 pp)	41.71 (+4.78 pp)	34.67 (+0.75 pp)

Table 4: Final diagnosis accuracy (%) before and after Remove-Text and Random-Text interventions.

Figure 5: Layer-wise Relative Attention per Token (RAPT) for Lingshu on text (excluding question stem) versus image tokens, before and after the Random-Text intervention.

Panel of Figure 6 — Figure 6: Cross-modal sensitivity on CT and microscopy. Green and brown points indicate cases where $S^{(m)}{\text{image}}>S^{(m)}{\text{text}}$ and $S^{(m)}{\text{image}}<S^{(m)}{\text{text}}$ , respectively. The right-hand bar chart shows the proportion of cases in each category.

Model	Average NMSE $\downarrow$	Acc. $\uparrow$
Qwen2.5-VL (7B)	0.98	33.92
Lingshu (7B)	0.95	36.93
HuatuoGPT-Vision (7B)	0.93	38.69

Table 5: Comparison of average normalized mean squared error (NMSE) and final diagnosis accuracy (%) across models. Best results are highlighted in bold .

Model	Baseline Acc.	Prompt Refinement Acc.
Qwen2.5-VL (7B)	33.92	34.67 (+0.75 pp)
Lingshu	36.93	38.44 (+1.51 pp)
HuatuoGPT-Vision	38.69	38.94 (+0.25 pp)

Table 6: Comparison of final diagnosis accuracy (%) before and after prompt refinement across models.

Setting	Acc.
Baseline	33.92
General SFT	34.67 (+0.75)
Targeted SFT	35.43 ( +1.51 )

Table 7: Final diagnosis accuracy (%) for Qwen2.5VL before and after supervised fine-tuning (SFT), comparing general with targeted SFT. Best results are in bold .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Examples of raw case reports. (a) A case record from Massachusetts General Hospital (Boston, MA); (b) A case report published by National Medical Journal of China .

Figure 8: Example differential diagnosis generation cases in English (top) and Chinese (bottom) . Colors mark different visual clinical evidence (CE) types referenced in the question, with corresponding expert-derived diagnostic findings; gray denotes textual CE. In our experiment, each CE type is input as either raw images or text findings, not both.

Panel of Figure 9 — Figure 9: Distributions of clinical specialties and specialty groups.

Model	Average Score $\uparrow$
	English	Chinese	Overall
Claude Opus 4.5	4.36	3.65	4.28
GPT-O3	4.46	3.85	4.38
Gemini 2.5 pro	4.15	3.70	4.10
GPT-5.2	4.52	3.78	4.43
Gemini 2.5 Flash	3.99	3.35	3.91

Appendix

Table 8: Comparison of average GPT-5 scores on differential diagnosis generation across models on the English subset. Clinical evidence from diagnostic investigations is provided as raw images.

Panel of Figure 10 — Figure 10: Distributions of GPT-5 scores on differential diagnosis across models on the English and Chinese subsets. Clinical evidence from diagnostic investigations is provided as raw images.

Figure 11: Distributions of GPT-5 scores on open-ended final diagnosis generation across models on the English subset. Clinical evidence from diagnostic investigations is provided as raw images.

Model	Average Score $\uparrow$
Claude Opus 4.5	3.48
GPT-O3	3.59
Gemini 2.5 pro	3.45
GPT-5.2	3.47
Gemini 2.5 Flash	3.26
Qwen3-VL (32B)	2.92

Appendix

Table 9: Comparison of average GPT-5 scores on open-ended final diagnosis generation across models on the English subset. Clinical evidence from diagnostic investigations is provided as raw images.

Model	Cardio.	Clin. Gen.	Emerg. Med.	Endo. & Diab.	Gastro.	Gen. Med.	Gen. Psych.	Haem.	Infect. Dis.	Med. Oncol.	Neuro.	Obs. & Gyn.	Paed.	Renal Med.	Resp. Med.	Rheum.	Trop. Med.
Claude Opus 4.5	71.43	44.44	100.00	68.18	69.23	65.15	71.43	64.86	64.71	52.17	84.21	0.00	33.33	50.00	83.33	57.14	66.67
GPT-O3	42.86	55.56	50.00	68.18	69.23	65.15	71.43	70.27	65.88	56.52	63.16	33.33	33.33	25.00	66.67	61.90	100.00
Gemini 2.5 Pro	42.86	55.56	75.00	77.27	69.23	57.55	42.86	67.57	67.06	65.22	63.16	66.67	66.67	50.00	83.33	52.38	100.00
GPT-5.2	42.86	66.67	75.00	63.64	61.54	64.39	57.14	59.46	58.33	47.83	73.68	100.00	66.67	0.00	66.67	42.86	100.00
Gemini 2.5 Flash	42.86	22.22	75.00	77.27	61.54	56.06	14.29	45.95	62.35	65.22	78.95	33.33	33.33	50.00	66.67	47.62	100.00
Qwen3-VL (32B)	42.86	55.56	75.00	50.00	61.54	49.24	42.86	51.35	44.05	65.22	68.42	33.33	0.00	0.00	50.00	47.62	66.67

Appendix

Table 10: Comparison of final diagnosis selection accuracy (%) across models on the English subset stratified by clinical specialties. Clinical evidence from diagnostic investigations is provided as raw images.

Model	Gen. & Emergency	Infectious & Trop.	Haemato- Oncol.	Endo, Rheum & Genetics	Neuro. & Psych.	Cardio. & Resp.	Renal & Gastro.	Obst., Gyn. & Paed.
Claude Opus 4.5	66.18	64.78	60.00	59.61	80.77	76.92	64.71	16.66
GPT-O3	64.70	67.04	65.00	63.46	65.39	53.85	58.82	33.33
Gemini 2.5 Pro	58.06	68.18	66.67	63.46	57.69	61.54	64.71	66.67
GPT-5.2	64.70	59.75	55.00	55.77	69.23	53.85	47.06	83.33
Gemini 2.5 Flash	56.62	63.63	53.34	55.77	61.54	53.85	58.82	33.33
Qwen3-VL (32B)	50.00	44.82	56.67	50.00	61.54	46.16	47.06	16.66

Appendix

Table 11: Comparison of final diagnosis selection accuracy (%) across models on the English subset stratified by clinical specialty groups. Clinical evidence from diagnostic investigations is provided as raw images.

Panel of Figure 12 — Figure 12: Layer-wise Relative Attention per Token (RAPT) for Qwen2.5-VL (7B) and HuatuoGPT-Vision on text (excluding question stem) versus image tokens, before and after the Random-Text intervention.

Modality	Qwen2.5-VL (7B)	Lingshu	HuatuoGPT-Vision
Laboratory	1.15	1.08	1.12
CT	0.90	0.88	0.91
Microscopy	0.73	0.63	0.74
MRI	1.00	0.99	0.99
Clinical Photography	0.85	0.91	0.90
X-ray	1.23	1.25	0.94

Appendix

Table 12: Comparison of normalized mean squared error for different clinical evidence types across models.

Panel of Figure 13 — Figure 13: Cross-modal sensitivity across different clinical evidence types. Green and brown points indicate cases where $S^{(m)}{\text{image}}>S^{(m)}{\text{text}}$ and $S^{(m)}{\text{image}}<S^{(m)}{\text{text}}$ , respectively.

Panel of Figure 14 — Figure 14: Distributions of cases where $S^{(m)}_{\text{image}}>S^{(m)}_{\text{text}}$ (green) and $S^{(m)}_{\text{image}}<S^{(m)}_{\text{text}}$ (brown) .

Modality	Qwen2.5-VL (7B)	Lingshu	HuatuoGPT-Vision
Baseline	0.73	0.63	0.74
Prompt Refinement	0.67 (-0.06)	0.61 (-0.02)	0.70 (-0.04)

Appendix

Table 13: Comparison of normalized mean squared error (lower is better) for microscopy before and after prompt refinement across models.

Figure 15: Distributions of cases where $S^{(m)}_{\text{image}}>S^{(m)}_{\text{text}}$ (green) and $S^{(m)}_{\text{image}}<S^{(m)}_{\text{text}}$ (brown) for microscopy before and after prompt refinement.

Figure 16: Example of human evaluation interface.

Setting	Acc.
Baseline	36.93
Targeted SFT	40.70 (+3.77)

Appendix

Table 14: Final diagnosis accuracy (%) for Lingshu before and after targeted supervised fine-tuning (SFT).

Explore similar work

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
Aug 23, 2026Lai Wei, Yuchao Chen, Zhenbiao Cao +5Multimodal Model Evaluation Multimodal Large Language Models
Huazhong University of Science and Technology · Fudan University · The First Affiliated Hospital with Nanjing Medical University and Jiangsu Province Hospital
CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
Feb 28, 2026Yannian Gu, Zhongzhen Huang, Linjie Mu +3Clinical Language Model Evaluation Multimodal Model Evaluation
Shanghai Jiao Tong University, Shanghai, China · SenseTime Research, China · Shanghai Innovation Institute, Shanghai, China
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Healthcare Medical Image Analysis
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Aug 23, 2026J/K move · Enter open · S save

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

Lai Wei, Yuchao Chen, Zhenbiao Cao, Xiaojin Zhang+4

Huazhong University of Science and Technology · Fudan University · The First Affiliated Hospital with Nanjing Medical University and Jiangsu Province Hospital

Multimodal Model Evaluation Multimodal Large Language Models Multimodal Reasoning Medical Image Benchmarks

Open paper AskPDF arXiv

The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus on textual reasoning or isolated visual question-answering (VQA) tasks, lacking holistic integration of clinical narratives and medical imaging, and thus failing to assess the multimodal diagnostic synthesis capability central to expert clinical judgment. To bridge this gap, we introduce MedReaMM, a benchmark specifically designed to evaluate models' ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm. Constructed from case reports sourced from top-tier medical journals and curated clinical case databases, MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes. These cases predominantly represent rare, atypical, or multi-system presentations that demand expert-level evidence integration beyond routine pattern recognition. We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability. Further analysis reveals that medical knowledge proficiency, medical image understanding, and evidence integration are all highly correlated with diagnostic performance.

Abstract

Explore similar work

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis