MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

Lai Wei; Yuchao Chen; Zhenbiao Cao; Xiaojin Zhang; Zhongyu Wei; Bangting Wang; Wei Chen; Xiang Bai

cs.CVAug 23, 2026

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

Authors: Lai Wei, Yuchao Chen, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Bangting Wang, Wei Chen, Xiang Bai

Organizations: Huazhong University of Science and Technology · Fudan University · The First Affiliated Hospital with Nanjing Medical University and Jiangsu Province Hospital

Abstract

The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus on textual reasoning or isolated visual question-answering (VQA) tasks, lacking holistic integration of clinical narratives and medical imaging, and thus failing to assess the multimodal diagnostic synthesis capability central to expert clinical judgment. To bridge this gap, we introduce MedReaMM, a benchmark specifically designed to evaluate models' ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm. Constructed from case reports sourced from top-tier medical journals and curated clinical case databases, MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes. These cases predominantly represent rare, atypical, or multi-system presentations that demand expert-level evidence integration beyond routine pattern recognition. We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability. Further analysis reveals that medical knowledge proficiency, medical image understanding, and evidence integration are all highly correlated with diagnostic performance.

Figures & tables

Figure 1: Comparison between MedReaMM and prior benchmarks. Upper: Existing datasets often feature factoid-based QA tasks or are limited to text-only input. Lower: MedReaMM is a diagnostic benchmark featuring multimodal inputs that combine clinical narratives and medical images under a complete-information paradigm, with ICD-11-coded diagnoses enabling standardized evaluation. Beyond core evaluation data, it also includes supplementary data to support in-depth analysis of diagnostic synthesis performance factors.

Benchmark	Size	Image Coverage		Text Coverage		Task Format	Data Source	Clinical Scenarios	ICD Standardized
Benchmark	Size	Average Images	Image Types	Average Length	Clinical Info	Task Format	Data Source	Clinical Scenarios	ICD Standardized
MedQA-USMLE 19	1,273	-	-	215.46	Partial	MCQ	Exams	✗	✗
MedMCQA-Dev 32	4,183	-	-	53.84	None	MCQ	Exams	✗	✗
PubMedQA 20	1,000	-	-	328.41	Partial	Closed	PubMed	✗	✗
PATH-VQA 16	6,719	0.13	1	15.38	None	Mixed	Textbooks	✗	✗
VQA-RAD 22	451	0.45	3	14.61	None	Mixed	MedPix	✗	✗

Table 1: Comparison with existing multimodal medical benchmarks. MedReaMM demonstrates high complexity through its exceptional multimodal diagnostic design, featuring the highest average image count (2.79 per case), the largest number of imaging modalities (13 types), and comprehensive clinical information (833 words per case). Benchmarks are categorized by their primary focus: traditional text-only QA , medical image understanding , diagnostic reasoning , and comprehensive multimodal clinical diagnosis .

Figure 2: Overview of the MedReaMM construction pipeline. (a) Case Collection : Diagnostic case reports are collected from NEJM, The Lancet, Eurorad, and other clinical sources. (b) Multimodal Extraction : Clinical text, medical images, and a diverse range of additional information are parsed from case reports. (c) Label Leakage Detection : A screening process identifies potential label leakage in both textual and visual modalities. (d) Diagnosis Encoding : All diagnoses are first decomposed into individual disease terms, then standardized via the ICD-11 API. (e) Expert Review : Final samples are validated by licensed clinicians to ensure quality and clinical plausibility.

Models	Top-5				Top-10				Top-1		MRR		NDCG@10
	Primary		Complete		Primary		Complete		Top-1		MRR		NDCG@10
	ICD	Sem.	ICD	Sem.	ICD	Sem.	ICD	Sem.	ICD	Sem.	ICD	Sem.	ICD	Sem.
Qwen2.5-VL-7B	22.4	17.9	13.3	9.1	23.2	18.4	14.1	9.3	17.1	13.1	19.7	15.3	20.9	16.5
Lingshu-7B	27.5	21.4	18.2	12.3	30.6	23.4	21.4	13.4	18.6	8.3	22.6	18.2	24.9	19.7
MedGemma-4B	28.3	20.8	17.6	11.2	31.2	22.9	19.7	12.6	15.8	13.3	19.6	15.9	21.3	18.3
Qwen3-VL-8B-Instruct	35.2	27.0	21.9	15.8	39.2	29.4	24.8	17.0	25.4	20.2	30.2	23.4	32.0	25.7

Table 2: Performance of 23 state-of-the-art LMMs on MedReaMM . Evaluation metrics include Recall (Top-5/10) and rank-sensitive indices (Top-1, MRR, NDCG@10). Note that Top-1 and MRR exclusively use the Primary diagnosis as the ground truth. All values are scaled by 100 for readability.

Figure 3: Linear regression analysis between primary diagnosis recall under the top-5 setting and 3 factors: medical image understanding ability, disease knowledge proficiency and evidence integration capability. Correlations are observed in all cases.

Models	Hint	Top-5 Primary	Top-10 Primary
Claude-Sonnet-4		52.2	56.4
Claude-Sonnet-4	✓	58.9 ${}_{\textbf{(+6.7)}}$	61.4 ${}_{\textbf{(+5.0)}}$
Gemini-2.5-Flash		55.8	56.4
Gemini-2.5-Flash	✓	58.6 ${}_{\textbf{(+2.8)}}$	60.3 ${}_{\textbf{(+3.9)}}$
GPT-4.1		53.9	55.3
GPT-4.1	✓	55.0 ${}_{\textbf{(+1.1)}}$	56.7 ${}_{\textbf{(+1.4)}}$

Table 3: Performance comparison on the rare disease subset of 180 cases.

Models	CoT	Top-5 Primary	Top-10 Primary
GPT-4o		48.9	51.9
GPT-4o	✓	50.3 ${}_{\textbf{(+1.4)}}$	53.5 ${}_{\textbf{(+1.6)}}$
GPT-4.1		49.3	52.2
GPT-4.1	✓	51.1 ${}_{\textbf{(+1.8)}}$	55.1 ${}_{\textbf{(+2.9)}}$
Llama-4-Maverick		38.5	43.8
Llama-4-Maverick	✓	49.0 ${}_{\textbf{(+10.5)}}$	49.5 ${}_{\textbf{(+5.8)}}$

Table 4: Impact of Chain-of-Thought prompting on standard models across benchmark cases.

Panel of Figure 4 — Figure 4: Ablation results comparing the effect of different input combinations on Primary Diagnosis Recall .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Distributions of image modalities (a) and diagnostic categories (b) in MedReaMM , reflecting the benchmark’s clinical realism and coverage.

Panel of Figure 6 — Figure 6: Word clouds of textual inputs (a) and diagnosis terms (b) in MedReaMM .

# Images per Case	# Cases	# Diagnoses per Case	# Cases
1	131	1	334
2	165	2	197
3	151	3	67
4	100	4	21
5-10	78	5	6

Appendix

Table 5: Distributions of the number of images (left) and diagnoses (right) per case in MedReaMM . Many cases contain multiple images and diagnoses, reflecting the complexity of real-world clinical data.

Figure 7: Bootstrap-estimated 95% confidence intervals for Primary Diagnosis Recall .

Method	Accuracy (%)	DSC
Human	10.87	0.2201
Gemini-2.5-Flash	43.48	0.6203
GPT-4.1	47.83	0.6453

Appendix

Table 6: Comparison of diagnostic accuracy between human gastroenterologists and LMMs on gastrointestinal disorders.

Model	Prompt Type	Top-5
Model	Prompt Type	Primary	Complete
GPT-4o	prompt1	48.9	25.0
	prompt2	49.2	26.7
	prompt3	49.8	28.7
Claude-Sonnet-4	prompt1	51.4	34.4
	prompt2	49.0	34.0

Appendix

Table 7: Performance Comparison by Model and Prompt Type (Top-5) on the 60-case subset.

Figure 8: Model performance across disease categories under different evaluation settings.

Figure 9: Model performance by publication year of case reports. No significant drop is observed for 2025 cases, suggesting minimal data leakage from pretraining corpora.

Panel of Figure 10 — Figure 10: The UI interface used in the clinical expert validation phase.

Stage	Remaining Cases
Case Report Collection	2867
Multimodal Data Extraction	1527
Label Leakage Detection	1193
Diagnosis Encoding	1193
Clinical Expert Review	625

Appendix

Table 8: Case selection process across five stages.

Evaluation Dimension	Sample Size	GPT-H	H-H
Causes	40	0.93	0.88
Symptoms		0.93	0.90
Diagnostic Tests		0.88	0.85
Treatments		0.90	0.85
Key Feature Detection	70	0.94	0.91
Medical Interpretation	70	0.91	0.89

Appendix

Table 9: Agreement between GPT-5 and human annotators, and inter-annotator agreement among three medical students. GPT-H means GPT-Human agreement; H-H means Human-Human agreement.

Panel of Figure 11 — Figure 11: A case study of MedReaMM. While strong models like Gemini-2.5-Pro correctly identified the primary disease, none recognized the actinomycosis co-infection. This reveals a common limitation: models often prioritize obvious cues while missing critical but subtle clinical findings.

Panel of Figure 12 — Figure 12: Content extraction prompt.

Panel of Figure 13 — Figure 13: Remove findings from detailed caption prompt.

Panel of Figure 14 — Figure 14: Main evaluation prompt.

Panel of Figure 15 — Figure 15: Image understanding test prompt.

Panel of Figure 16 — Figure 16: Image understanding evaluation prompt.

Panel of Figure 17 — Figure 17: Disease knowledge test prompt.

Panel of Figure 18 — Figure 18: Disease knowledge evaluation prompt.

Panel of Figure 19 — Figure 19: Ground truth reasoning path extraction prompt.

Panel of Figure 21 — Figure 21: Reasoning path evaluation prompt.

Panel of Figure 22 — Figure 22: Prompt 2 used in prompt robustness analysis.

Panel of Figure 23 — Figure 23: Prompt 3 used in prompt robustness analysis.

Explore similar work

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
Feb 25, 2026Boqi Chen, Xudong Liu, Jiachuan Peng +5Medical Diagnosis Medical VLMs
ETH Zurich · MBZUAI · Amazon +4
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
Jul 28, 2026Rui Yang, Weihao Xuan, Yi Lin +23Medical VLMs Multi-Turn Dialogue Evaluation
Center for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore · Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore · Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA +24
MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
Apr 12, 2026Junzhi Ning, Jiashi Lin, Yingying Fang +9Medical Image Benchmarks Multimodal Model Evaluation
Shanghai AI Laboratory · Imperial College London · Shanghai Jiao Tong University +1

Feb 25, 2026J/K move · Enter open · S save

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti+4

ETH Zurich · MBZUAI · Amazon+4

Medical Diagnosis Medical VLMs Multimodal Model Evaluation Multimodal Reasoning

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE (

\it{e.g.}

, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.

Abstract

Explore similar work

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark