Medical thinking with multiple images

Zonghai Yao; Benlu Wang; Yifan Zhang; Junda Wang; Iris Xia; Zhipeng Tang; Shuo Han; Feiyun Ouyang

cs.CVApr 14, 2026

Medical thinking with multiple images

Authors: Zonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang, Iris Xia, Zhipeng Tang, Shuo Han, Feiyun Ouyang, +3 more

Organizations: Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University · Miner School of Computer and Information Sciences, UMass Lowell · Department of Electrical and Computer Engineering, UMass Lowell

Abstract

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Figures & tables

Figure 1: Medical Thinking with Images (TwI): task and diagnosing-style evaluation. Left: a sample case with a clinical scenario, multi-view images (e.g., radiograph + CT), and a five-option single-best-answer diagnosis. Middle: TwI’s three supervised steps: (1) Per-Image Findings (detect and name key radiological signs for each image, expert-annotated, brief statements); (2) Case-level Integrated Imaging Summary (synthesize cross-view evidence into a single case summary); (3) Differential-Diagnosis (DDx) reasoning (align the summary with candidate diagnoses, rule out distractors with image-grounded arguments, and pick the most consistent answer). Right: Beyond-accuracy evaluation. Steps 1–2 use automatic metrics (ROUGE / RadCliQ), while Step 3 and the Medical Education Case Discussion are assessed with structured human- and LLM-judge rubrics that check clinical correctness and educational value, localizing failures in image reading, cross-view fusion, and teaching quality (see Section 3 ).

Benchmark	# Case	Expert Annotation	Clinical Scenarios	# Img per Case	Multi-Mod Imaging	Longitud Studies	Think-with-Images Intermediate Signals	Beyond-ACC Evaluation
VQA-Rad Lau et al.	451	✗	✗	0.45	✗	✗	✗	✗
VQA-Med Ben Abacha et al.	500	✗	✗	1.00	✗	✗	✗	✗
Path-VQA He et al.	6,719	✗	✗	0.13	✗	✗	✗	✗
SLAKE-En Liu et al.	1,061	✗	✗	0.09	✗	✗	✗	✗
PMC-VQA Zhang et al.	33,430	✗	✗	0.87	✗	✗	✗	✗
OmniMedVQA Hu et al.	127,995	✗	✗	0.92	✗	✗	✗	✗

Table 1: Comparisons with multimodal medical QA benchmarks. MedThinkVQA is expert-annotated, averages 6.62 images per case (prior maxima

\leq

1.43;

\geq

4.5

\times

more), and is the largest corpus at the expert-annotation level; a checkmark in the Expert Annotation column indicates that items are curated and labeled by clinicians rather than automatically generated. Clinical Scenarios. Prior work lacks broad, fine-grained coverage of real diagnostic scenarios; only MedThinkVQA and MedXpertQA-MM include scenario labels. Multi-Modal Imaging / Longitudinal Studies. We mark Multi-Modal Imaging when at least some questions require joint interpretation of images from multiple distinct medical imaging modalities for the same case (e.g., radiograph+CT), and Longitudinal Studies when questions are built from follow-up imaging of the same patient at different time points (e.g., baseline vs follow-up studies). Think-with-Images Intermediate Signals. This merged column indicates whether a benchmark provides intermediate supervision for think-with-images reasoning, including per-image findings , a case-level imaging summary , and a case discussion (teaching note). Beyond-ACC Evaluation. Leveraging these signals, only MedThinkVQA supports fine-grained, end-to-end assessment of think-with-images reasoning and teaching discussions: stepwise checks, error-type tags, education-value scoring, and automatic intermediate metrics, rather than accuracy alone.

Figure 2: Distribution of radiology imaging main categories

Overall
Samples	720
Images	5845
Per-sample
Imgs/sample	8.12
Cap. length	1338.6
Find. length	875.8

Table 2: Test stats ( Cap/Find/Disc. = caption, findings, discussion; Pos. correct = avg. position of correct option; Mean mod. cnt = mean # of imaging modalities; see Appendix N for modality statistics and Appendix O for longitudinal-study statistics.

Figure 3: MedThinkVQA benchmark results on the test set , highlighting the core challenge of thinking with multiple images: performance is limited less by raw reasoning length alone, and more by whether a model can reliably ground, align, and compose evidence across views. Top: Overall test accuracy (%). When available, paired bars report the same model in thinking and matched non-thinking mode (gray); single bars report the thinking result when such a mode exists, otherwise the standard configuration. Detailed results are provided in Appendix D . Bottom: (left) Masking a fraction of images shows stable gains as more visual evidence is provided, indicating that the benchmark contains real cross-image information gain. However, larger reasoning models benefit more, suggesting that success depends on how well additional images are actually used, rather than simply being available. The gains of thinking over non-thinking also peak at +8.1/+6.5/+3.7 points for GPT-5/mini/nano as more visual evidence becomes available. (middle) Accuracy generally increases with greater reasoning effort, as measured by average output tokens, indicating a clear scaling trend. (right) Both Dense and MoE LLMs within each family broadly follow model-size scaling, but their ability to convert extra reasoning budget into better answers differs. Within the medium-to-large ( $>2$ B) Qwen3.5 family, reasoning gains increase as models scale, whereas Qwen3-VL and small-scale Qwen3.5 models show limited and unstable returns, suggesting that extra inference-time computation helps mainly when early visual evidence extraction is already reliable and cannot reliably compensate for weak multi-image grounding.

Error type	All error steps (N=1509)	Critical error steps (N=182)
Image Understanding Err	959 (63.55%)	126 (69.23%)
Reasoning Err	583 (38.63%)	71 (39.01%)
Medical Knowledge Err	362 (23.99%)	60 (32.97%)
Clinical Scenario Err	191 (12.66%)	22 (12.09%)

Table 3: LLM-judge error-type coverage. Note: categories are multi-label; percentages are step-level coverage over error steps and may sum to

>

100%. Full per-split (answer-correct vs. wrong) breakdowns are in the Appendix.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Case group	#Cases	#Images	#Supportive imgs	Supportive ratio (%)
All images supportive	65	463	463	100.00
Mixed supportive / redundant	31	315	222	70.48
Overall (96 cases)	96	778	685	88.05

Appendix

Table 4: Round 2 expert audit: image-level redundancy vs. support. Most cases use all images as supportive evidence; even in mixed cases, the majority of views remain supportive rather than redundant.

Figure 4: Performance of open-source multimodal models on MedThinkVQA under varying visible image ratios (0%–100%). Models are grouped by parameter scale into small ( $<27$ B, n=14) and large ( $>27$ B, n=11) categories. See Appendix D for full list of models

Model	Reasoning/Thinking	acc	#correct
OpenAI (GPT-5 family)
GPT-5.2	none	0.4958	357
GPT-5.2	low	0.5292	381
GPT-5.2	medium	0.5444	392
GPT-5.2	high	0.5333	384
GPT-5.2	x-high	0.5486	395

Appendix

Table 5: Full-test results on the full test set (

N=720

) with full visual evidence (

r=100\%

Model	Reasoning/Thinking	$0\%$	$20\%$	$40\%$	$60\%$	$80\%$
OpenAI (GPT-5 family; API)
GPT-5.1	low	0.3389 (244)	–	–	–	–
GPT-5.2	low	0.3653 (263)	–	–	–	–
GPT-5	medium	0.3264 (235)	0.3736 (269)	0.4319 (311)	0.4583 (330)	0.4736 (341)
GPT-5	minimal	0.2806 (202)	0.3278 (236)	0.3847 (277)	0.3806 (274)	0.3931 (283)
GPT-5 mini	medium	0.2847 (205)	0.3528 (254)	0.3542 (255)	0.3667 (264)	0.3861 (278)

Appendix

Table 6: Image-ratio ablation results (

r\in\{0,20,40,60,80\}\%

Figure 5: Accuracy on 200 randomly sampled cases from the MedThinkVQA when augmenting images with text. We compare Image Hint (caption-like) and Integrated Imaging Summary (diagnosis-oriented findings), each provided either by an expert or generated by the model itself (self). Both combines the two.

	ROUGE-L ( $\uparrow$ )		RadCliQ-v1 ( $\uparrow$ )
Model	Caption	Findings	Caption	Findings
gpt-5-nano	0.1435	0.1585	0.8080	0.6781
gpt-5-mini	0.1510	0.1636	0.8317	0.6931
GPT-5	0.1534	0.1627	0.8341	0.6818
medgemma-27b-it	0.1336	0.1621	0.7810	0.7192

Appendix

Table 7: Scores of VLMs for Image

\rightarrow

Caption and Image

\rightarrow

Findings across two metrics (ROUGE-L and RadCliQ).

Model	Background	Clinical	Imaging	Outcome	Take-Home	Overall
	(%)	(%)	(%)	(%)	(%)	(%)
gpt-5	100.0	100.0	97.81	98.70	100.0	99.08
gpt-5-mini	98.59	98.65	99.10	100.0	100.0	99.22
gpt-5-nano	97.87	98.99	97.39	95.89	98.46	97.76
medgemma-27b-it	89.0	97.89	94.93	85.71	93.65	92.81

Appendix

Table 8: Sentence-level factual correctness evaluation across discussion subsections

Model	Total	Disease	Clinical	Imaging	Reasoning	Transfer
		Overview	Pathophys.		Different.	Learning
gpt-5	9.9	2.0	1.9	2.0	2.0	2.0
gpt-5-mini	9.4	1.95	1.6	2.0	1.85	2.0
gpt-5-nano	8.4	1.7	1.25	2.0	1.45	2.0
medgemma-27b-it	7.05	1.4	1.15	1.85	1.1	1.55

Appendix

Table 9: Rubric evaluation scores across different models

Figure 6: Endoscopy image for this case ( img_id = l9iMrGt3 ).

Panel of Figure 7 — Figure 7: CT images ( img_id = 3uIJtKe-, pfx97TC8 ) stored under images/cases/modality/ .

Panel of Figure 8 — Figure 8: MRI images ( img_id = O5kEGVZq, IAV1h4UN, FjWYFzXB, F0KIjEeq, U14cWn_5 ) with uniform image size and aligned top edges.

Figure 9: Histology / pathology image for this case ( img_id = 3xrCMRPY ).

Panel of Figure 10 — Figure 10: Baseline chest radiographs and CT at admission, all displayed with a uniform relative size.

Panel of Figure 11 — Figure 11: Serial chest radiographs during ICU stay, showing pneumothoraces and multiple chest drains.

Panel of Figure 12 — Figure 12: CT during peak disease severity, with confluent cystic lesions and diffuse parenchymal involvement.

Panel of Figure 13 — Figure 13: CT with larger pneumothoraces and pneumomediastinum, on a background of cystic pulmonary tuberculosis.

Panel of Figure 14 — Figure 14: Pre-discharge CT: persistent cystic lesions but improved clinical tolerance and removal of chest drains.

Panel of Figure 15 — Figure 15: Late follow-up CT eight months after discharge, demonstrating almost complete recovery with limited sequelae.

Panel of Figure 16 — Figure 16: Imaging from a paraspinal chest-wall mass later diagnosed as hibernoma: chest radiograph, contrast-enhanced CT, and multiparametric MRI (T1, T2 with fat suppression, and post-contrast). All images are stored in the dataset as individual files associated with this case.

Panel of Figure 17 — Figure 17: Nine CT slices from a single case demonstrating three concurrent vascular compression syndromes: Wilkie (SMA) syndrome, Nutcracker syndrome with pelvic venous congestion, and May–Thurner syndrome. All images are stored as separate files but belong to one MedThinkVQA item.

Model	Correct	Total	Accuracy
Qwen3-Next-80B-A3B-Instruct	1,657	2,859	0.580 (57.96%)
gpt-oss-120b	1,747	2,859	0.611 (61.11%)
medgemma-27b-text-it	1,644	2,859	0.575 (57.50%)
Llama-3.3-70B-Instruct	1,613	2,859	0.564 (56.42%)
Four-model joint-correct intersection	1,074	2,859	0.376 (37.57%)

Appendix

Table 10: Per-model accuracy and four-model joint-correct intersection on the 2,859-item raw test set.

Modality	Train images	Train (%)	Test images	Test (%)
CT	17,280	36.32	1,929	33.00
MRI	15,026	31.59	2,179	37.28
X-ray	6,337	13.32	540	9.24
Ultrasound	3,989	8.39	609	10.42
Non-modality / Workflow / Post-processing	2,434	5.12	230	3.93
Pathology	1,135	2.39	151	2.58

Appendix

Table 11: Per-image distribution of coarse (Level-1) modalities in the training and test splits. Counts are absolute image counts, and percentages are relative to the total number of images in each split.

Modality	Train cases	Train (%)	Test cases	Test (%)
CT	4,552	61.96	455	63.19
MRI	3,144	42.79	393	54.58
X-ray	2,749	37.42	248	34.44
Ultrasound	1,648	22.43	220	30.56
Non-modality / Workflow / Post-processing	1,411	19.21	151	20.97
Pathology	571	7.77	78	10.83

Appendix

Table 12: Per-case coverage of coarse (Level-1) modalities. A case is counted for a modality if it contains at least one image annotated with that modality. Percentages are relative to the total number of cases in each split.

Modality	Sub-modality	Train images	Train (%)	Test images	Test (%)
CT	Contrast-enhanced CT	8,822	18.54	1,041	17.81
CT	Non-contrast CT	4,861	10.22	503	8.61
CT	CT Angiography	1,538	3.23	132	2.26
CT	HRCT / Thin-slice CT	1,399	2.94	202	3.46
CT	Other_CT	632	1.33	43	0.74
CT	CT Perfusion	28	0.06	8	0.14

Appendix

Table 13: Per-image distribution of fine-grained (Level-2) sub-modalities, grouped by their corresponding coarse (Level-1) modality.

# Modalities	Train cases	Train (%)	Test cases	Test (%)
1	2,476	33.70	165	22.92
2	2,859	38.91	292	40.56
3	1,463	19.91	166	23.06
4	443	6.03	74	10.28
5	87	1.18	22	3.06
6	13	0.18	1	0.14

Appendix

Table 14: Distribution of the number of distinct coarse (Level-1) modalities per case ( modalities_count ) in the training and test splits.

# Sub-modalities	Train cases	Train (%)	Test cases	Test (%)
1	1,593	21.68	94	13.06
2	2,152	29.29	183	25.42
3	1,819	24.76	196	27.22
4	1,051	14.31	132	18.33
5	441	6.00	66	9.17
6	183	2.49	30	4.17

Appendix

Table 15: Distribution of the number of distinct fine-grained (Level-2) sub-modalities per case ( sub_modalities_count ) in the training and test splits.

Split	# Cases	# Longitudinal	Share (%)
Train	7,347	2,154	29.3
Test	720	219	30.4
Overall	8,067	2,373	29.4

Appendix

Table 16: Prevalence of longitudinal studies in MedThinkVQA.

timepoint_count	Train	Test	Overall
2	1,631 (75.7%)	163 (74.4%)	1,794 (75.6%)
3	379 (17.6%)	41 (18.7%)	420 (17.7%)
4	115 (5.3%)	9 (4.1%)	124 (5.2%)
5+	29 (1.3%)	6 (2.7%)	35 (1.5%)

Appendix

Table 17: Distribution of the number of imaging time points ( timepoint_count ) among longitudinal cases. Each cell shows # cases and the percentage relative to the longitudinal subset of the corresponding split.

Interval category	Train	Test	Overall
Unknown / Unspecified	555 (25.8%)	56 (25.6%)	611 (25.7%)
Hours	52 (2.4%)	12 (5.5%)	64 (2.7%)
Days	327 (15.2%)	29 (13.2%)	356 (15.0%)
Weeks	210 (9.7%)	16 (7.3%)	226 (9.5%)
Months	577 (26.8%)	69 (31.5%)	646 (27.2%)
Years	363 (16.9%)	30 (13.7%)	393 (16.6%)

Appendix

Table 18: Coarse distribution of follow-up interval descriptions ( interval_text ) among longitudinal cases. Categories are assigned by keyword matching to temporal units and explicit calendar spans; percentages are relative to the longitudinal subset of each split.

Max follow-up	Train	Test	Overall
$\leq$ 1 week	230 (10.7%)	30 (13.7%)	260 (11.0%)
1 week–1 month	285 (13.2%)	20 (9.1%)	305 (12.9%)
1–6 months	431 (20.0%)	56 (25.6%)	487 (20.5%)
6–12 months	200 (9.3%)	22 (10.0%)	222 (9.4%)
1–3 years	191 (8.9%)	14 (6.4%)	205 (8.6%)
3–5 years	66 (3.1%)	8 (3.7%)	74 (3.1%)

Appendix

Table 19: Distribution of the maximum follow-up interval/span (estimated from interval_text ) among longitudinal cases. Each cell shows # cases and the percentage relative to the longitudinal subset of the corresponding split.

Pairwise comparison	Cohen’s $\kappa$
Expert 1 vs. Expert 2	0.822833
Expert 1 vs. LLM judge	0.838357
Expert 2 vs. LLM judge	0.701566

Appendix

Table 20: Inter-rater reliability on step factuality (Cohen’s

\kappa

). High agreement with Expert 1 and substantial agreement with Expert 2 support the reliability of the LLM judge.

Figure 18: MELD similarity distributions on the MedThinkVQA test set across seven representative models. All distributions remain well below the 50% high-risk threshold, with similar behavior for text-only LLMs and VLMs.

\ell\;=\;\frac{\mathrm{int}\!\big(\mathrm{round}\big(\tfrac{2.0\times M}{|q|}\times 100\big)\big)}{100},

Appendix

Algorithm 1 MELD (original reproduction)

Explore similar work

Oct 19, 2024

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

Medical multimodal large language models (MLLMs) can perform well on existing medical visual question answering (MedVQA) benchmarks, but their training data often does not match clinical diagnosis. Most supervision is organized around isolated images or short QA pairs, leaving two structures weakly specified: how evidence leads to a decision, and how views, series, modalities, and patient context from the same case are linked. We introduce MultiViewDx, a partly physician-validated multimodal instruction dataset for evidence-linked multi-view medical imaging diagnosis. MultiViewDx uses the clinical case as the supervision unit. It links imaging studies with patient context, normalizes heterogeneous reports into an evidence-linked workflow (evidence -> findings -> differential discussion -> diagnosis), and uses a unified image-text retriever to constrain instruction synthesis to source-supported evidence. It covers X-ray, CT, MRI, ultrasound, histopathology, and other clinical visual sources. We fine-tune MultiViewDx-8B-AN and evaluate it on both existing MedVQA benchmarks and real-world case-based diagnostic reasoning. Across four MedVQA benchmarks, it achieves the best average accuracy among compared systems (79.0%), outperforming HuatuoGPT-Vision-34B (66.7%) and Claude3-Opus (55.7%). Beyond MedVQA, on JAMA Clinical Challenge cases, it receives the strongest overall rating under a physician-designed rubric for key clinical points, diagnostic inference, and evidence grounding. Controlled ablations and clinician evaluation show that both case-level multi-view organization and evidence-linked reasoning targets contribute to the gain.

Junda Wang, Zonghai Yao, Yujan Ting +5

United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Healthcare Medical Image Analysis Medical VQA Medical Image Grounding Multimodal Foundation Models Multimodal Instruction Tuning+1

Ask PDF

Jun 10, 2026

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

High-stakes clinical use of large vision-language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image-question-answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason.

Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6

York University · Vector Institute · University of British Columbia +5

Medical VQA Medical VLMs VLM Reasoning Visually Grounded Reasoning Medical Image Benchmarks

Ask PDF

Sep 7, 2026

A visual large language foundational model for medical image recognition using clinician-contributed online resources

Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their application in medical settings remains limited by the scarcity of visual question answering (VQA) datasets that capture clinical reasoning and explicit image-text alignment. Here, we leverage de-identified medical images and expert commentaries shared through clinician-oriented online resources. By combining an advanced LLM with clinician-in-the-loop verification, we established a rigorous pipeline to construct ThoughtMed-1M, a long-form medical VQA dataset containing over one million VQA pairs and designed to capture structured clinical reasoning and medical image-text alignment. To demonstrate its utility, we developed a FOundational LLM Trained on ThoughtMed-1M (FOLTMed). FOLTMed achieved state-of-the-art performance across 42 medical VQA benchmark datasets, with a macro accuracy of 85.4 percent. It also generated more clinically coherent responses on the ThoughtMed-1M test set, outperforming state-of-the-art models by 3 to 5 percent across factuality and similarity metrics, highlighting a scalable paradigm for advancing research on clinically grounded multimodal LLMs.

Lingxuan Hou, Yuhua Xie, Yue Hu +14

College of Biomedical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Engineering, Computer & Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand · Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China +2

Medical VQA Medical VLMs VLM Reasoning Medical Image Benchmarks

Ask PDF