cs.CVApr 14, 2026
SaveMedical thinking with multiple images
Organizations: Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University · Miner School of Computer and Information Sciences, UMass Lowell · Department of Electrical and Computer Engineering, UMass Lowell
Abstract
Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.
Figures & tables
Figure 1: Medical Thinking with Images (TwI): task and diagnosing-style evaluation. Left: a sample case with a clinical scenario, multi-view images (e.g., radiograph + CT), and a five-option single-best-answer diagnosis. Middle: TwI’s three supervised steps: (1) Per-Image Findings (detect and name key radiological signs for each image, expert-annotated, brief statements); (2) Case-level Integrated Imaging Summary (synthesize cross-view evidence into a single case summary); (3) Differential-Diagnosis (DDx) reasoning (align the summary with candidate diagnoses, rule out distractors with image-grounded arguments, and pick the most consistent answer). Right: Beyond-accuracy evaluation. Steps 1–2 use automatic metrics (ROUGE / RadCliQ), while Step 3 and the Medical Education Case Discussion are assessed with structured human- and LLM-judge rubrics that check clinical correctness and educational value, localizing failures in image reading, cross-view fusion, and teaching quality (see Section 3 ).
| Benchmark | # Case | Expert Annotation | Clinical Scenarios | # Img per Case | Multi-Mod Imaging | Longitud Studies | Think-with-Images Intermediate Signals | Beyond-ACC Evaluation |
| VQA-Rad Lau et al. | 451 | ✗ | ✗ | 0.45 | ✗ | ✗ | ✗ | ✗ |
| VQA-Med Ben Abacha et al. | 500 | ✗ | ✗ | 1.00 | ✗ | ✗ | ✗ | ✗ |
| Path-VQA He et al. | 6,719 | ✗ | ✗ | 0.13 | ✗ | ✗ | ✗ | ✗ |
| SLAKE-En Liu et al. | 1,061 | ✗ | ✗ | 0.09 | ✗ | ✗ | ✗ | ✗ |
| PMC-VQA Zhang et al. | 33,430 | ✗ | ✗ | 0.87 | ✗ | ✗ | ✗ | ✗ |
| OmniMedVQA Hu et al. | 127,995 | ✗ | ✗ | 0.92 | ✗ | ✗ | ✗ | ✗ |
Table 1: Comparisons with multimodal medical QA benchmarks. MedThinkVQA is expert-annotated, averages 6.62 images per case (prior maxima 1.43; 4.5 more), and is the largest corpus at the expert-annotation level; a checkmark in the Expert Annotation column indicates that items are curated and labeled by clinicians rather than automatically generated. Clinical Scenarios. Prior work lacks broad, fine-grained coverage of real diagnostic scenarios; only MedThinkVQA and MedXpertQA-MM include scenario labels. Multi-Modal Imaging / Longitudinal Studies. We mark Multi-Modal Imaging when at least some questions require joint interpretation of images from multiple distinct medical imaging modalities for the same case (e.g., radiograph+CT), and Longitudinal Studies when questions are built from follow-up imaging of the same patient at different time points (e.g., baseline vs follow-up studies). Think-with-Images Intermediate Signals. This merged column indicates whether a benchmark provides intermediate supervision for think-with-images reasoning, including per-image findings , a case-level imaging summary , and a case discussion (teaching note). Beyond-ACC Evaluation. Leveraging these signals, only MedThinkVQA supports fine-grained, end-to-end assessment of think-with-images reasoning and teaching discussions: stepwise checks, error-type tags, education-value scoring, and automatic intermediate metrics, rather than accuracy alone.
Figure 2: Distribution of radiology imaging main categories
| Overall | |
|---|---|
| Samples | 720 |
| Images | 5845 |
| Per-sample | |
| Imgs/sample | 8.12 |
| Cap. length | 1338.6 |
| Find. length | 875.8 |
Table 2: Test stats ( Cap/Find/Disc. = caption, findings, discussion; Pos. correct = avg. position of correct option; Mean mod. cnt = mean # of imaging modalities; see Appendix N for modality statistics and Appendix O for longitudinal-study statistics.
Figure 3: MedThinkVQA benchmark results on the test set , highlighting the core challenge of thinking with multiple images: performance is limited less by raw reasoning length alone, and more by whether a model can reliably ground, align, and compose evidence across views. Top: Overall test accuracy (%). When available, paired bars report the same model in thinking and matched non-thinking mode (gray); single bars report the thinking result when such a mode exists, otherwise the standard configuration. Detailed results are provided in Appendix D . Bottom: (left) Masking a fraction of images shows stable gains as more visual evidence is provided, indicating that the benchmark contains real cross-image information gain. However, larger reasoning models benefit more, suggesting that success depends on how well additional images are actually used, rather than simply being available. The gains of thinking over non-thinking also peak at +8.1/+6.5/+3.7 points for GPT-5/mini/nano as more visual evidence becomes available. (middle) Accuracy generally increases with greater reasoning effort, as measured by average output tokens, indicating a clear scaling trend. (right) Both Dense and MoE LLMs within each family broadly follow model-size scaling, but their ability to convert extra reasoning budget into better answers differs. Within the medium-to-large ( B) Qwen3.5 family, reasoning gains increase as models scale, whereas Qwen3-VL and small-scale Qwen3.5 models show limited and unstable returns, suggesting that extra inference-time computation helps mainly when early visual evidence extraction is already reliable and cannot reliably compensate for weak multi-image grounding.
| Error type | All error steps (N=1509) | Critical error steps (N=182) |
|---|---|---|
| Image Understanding Err | 959 (63.55%) | 126 (69.23%) |
| Reasoning Err | 583 (38.63%) | 71 (39.01%) |
| Medical Knowledge Err | 362 (23.99%) | 60 (32.97%) |
| Clinical Scenario Err | 191 (12.66%) | 22 (12.09%) |
Table 3: LLM-judge error-type coverage. Note: categories are multi-label; percentages are step-level coverage over error steps and may sum to 100%. Full per-split (answer-correct vs. wrong) breakdowns are in the Appendix.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Case group | #Cases | #Images | #Supportive imgs | Supportive ratio (%) |
|---|---|---|---|---|
| All images supportive | 65 | 463 | 463 | 100.00 |
| Mixed supportive / redundant | 31 | 315 | 222 | 70.48 |
| Overall (96 cases) | 96 | 778 | 685 | 88.05 |
Table 4: Round 2 expert audit: image-level redundancy vs. support. Most cases use all images as supportive evidence; even in mixed cases, the majority of views remain supportive rather than redundant.
Figure 4: Performance of open-source multimodal models on MedThinkVQA under varying visible image ratios (0%–100%). Models are grouped by parameter scale into small ( B, n=14) and large ( B, n=11) categories. See Appendix D for full list of models
| Model | Reasoning/Thinking | acc | #correct |
|---|---|---|---|
| OpenAI (GPT-5 family) | |||
| GPT-5.2 | none | 0.4958 | 357 |
| GPT-5.2 | low | 0.5292 | 381 |
| GPT-5.2 | medium | 0.5444 | 392 |
| GPT-5.2 | high | 0.5333 | 384 |
| GPT-5.2 | x-high | 0.5486 | 395 |
Table 5: Full-test results on the full test set ( ) with full visual evidence ( ).
| Model | Reasoning/Thinking | |||||
|---|---|---|---|---|---|---|
| OpenAI (GPT-5 family; API) | ||||||
| GPT-5.1 | low | 0.3389 (244) | – | – | – | – |
| GPT-5.2 | low | 0.3653 (263) | – | – | – | – |
| GPT-5 | medium | 0.3264 (235) | 0.3736 (269) | 0.4319 (311) | 0.4583 (330) | 0.4736 (341) |
| GPT-5 | minimal | 0.2806 (202) | 0.3278 (236) | 0.3847 (277) | 0.3806 (274) | 0.3931 (283) |
| GPT-5 mini | medium | 0.2847 (205) | 0.3528 (254) | 0.3542 (255) | 0.3667 (264) | 0.3861 (278) |
Table 6: Image-ratio ablation results ( ).
Figure 5: Accuracy on 200 randomly sampled cases from the MedThinkVQA when augmenting images with text. We compare Image Hint (caption-like) and Integrated Imaging Summary (diagnosis-oriented findings), each provided either by an expert or generated by the model itself (self). Both combines the two.
| ROUGE-L ( ) | RadCliQ-v1 ( ) | |||
|---|---|---|---|---|
| Model | Caption | Findings | Caption | Findings |
| gpt-5-nano | 0.1435 | 0.1585 | 0.8080 | 0.6781 |
| gpt-5-mini | 0.1510 | 0.1636 | 0.8317 | 0.6931 |
| GPT-5 | 0.1534 | 0.1627 | 0.8341 | 0.6818 |
| medgemma-27b-it | 0.1336 | 0.1621 | 0.7810 | 0.7192 |
Table 7: Scores of VLMs for Image Caption and Image Findings across two metrics (ROUGE-L and RadCliQ).
| Model | Background | Clinical | Imaging | Outcome | Take-Home | Overall |
|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (%) | |
| gpt-5 | 100.0 | 100.0 | 97.81 | 98.70 | 100.0 | 99.08 |
| gpt-5-mini | 98.59 | 98.65 | 99.10 | 100.0 | 100.0 | 99.22 |
| gpt-5-nano | 97.87 | 98.99 | 97.39 | 95.89 | 98.46 | 97.76 |
| medgemma-27b-it | 89.0 | 97.89 | 94.93 | 85.71 | 93.65 | 92.81 |
Table 8: Sentence-level factual correctness evaluation across discussion subsections
| Model | Total | Disease | Clinical | Imaging | Reasoning | Transfer |
|---|---|---|---|---|---|---|
| Overview | Pathophys. | Different. | Learning | |||
| gpt-5 | 9.9 | 2.0 | 1.9 | 2.0 | 2.0 | 2.0 |
| gpt-5-mini | 9.4 | 1.95 | 1.6 | 2.0 | 1.85 | 2.0 |
| gpt-5-nano | 8.4 | 1.7 | 1.25 | 2.0 | 1.45 | 2.0 |
| medgemma-27b-it | 7.05 | 1.4 | 1.15 | 1.85 | 1.1 | 1.55 |
Table 9: Rubric evaluation scores across different models
Figure 6: Endoscopy image for this case ( img_id = l9iMrGt3 ).
Figure 7: CT images ( img_id = 3uIJtKe-, pfx97TC8 ) stored under images/cases/modality/ .
Figure 8: MRI images ( img_id = O5kEGVZq, IAV1h4UN, FjWYFzXB, F0KIjEeq, U14cWn_5 ) with uniform image size and aligned top edges.
Figure 9: Histology / pathology image for this case ( img_id = 3xrCMRPY ).
Figure 10: Baseline chest radiographs and CT at admission, all displayed with a uniform relative size.
Figure 11: Serial chest radiographs during ICU stay, showing pneumothoraces and multiple chest drains.
Figure 12: CT during peak disease severity, with confluent cystic lesions and diffuse parenchymal involvement.
Figure 13: CT with larger pneumothoraces and pneumomediastinum, on a background of cystic pulmonary tuberculosis.
Figure 14: Pre-discharge CT: persistent cystic lesions but improved clinical tolerance and removal of chest drains.
Figure 15: Late follow-up CT eight months after discharge, demonstrating almost complete recovery with limited sequelae.
Figure 16: Imaging from a paraspinal chest-wall mass later diagnosed as hibernoma: chest radiograph, contrast-enhanced CT, and multiparametric MRI (T1, T2 with fat suppression, and post-contrast). All images are stored in the dataset as individual files associated with this case.
Figure 17: Nine CT slices from a single case demonstrating three concurrent vascular compression syndromes: Wilkie (SMA) syndrome, Nutcracker syndrome with pelvic venous congestion, and May–Thurner syndrome. All images are stored as separate files but belong to one MedThinkVQA item.
| Model | Correct | Total | Accuracy |
|---|---|---|---|
| Qwen3-Next-80B-A3B-Instruct | 1,657 | 2,859 | 0.580 (57.96%) |
| gpt-oss-120b | 1,747 | 2,859 | 0.611 (61.11%) |
| medgemma-27b-text-it | 1,644 | 2,859 | 0.575 (57.50%) |
| Llama-3.3-70B-Instruct | 1,613 | 2,859 | 0.564 (56.42%) |
| Four-model joint-correct intersection | 1,074 | 2,859 | 0.376 (37.57%) |
Table 10: Per-model accuracy and four-model joint-correct intersection on the 2,859-item raw test set.
| Modality | Train images | Train (%) | Test images | Test (%) |
|---|---|---|---|---|
| CT | 17,280 | 36.32 | 1,929 | 33.00 |
| MRI | 15,026 | 31.59 | 2,179 | 37.28 |
| X-ray | 6,337 | 13.32 | 540 | 9.24 |
| Ultrasound | 3,989 | 8.39 | 609 | 10.42 |
| Non-modality / Workflow / Post-processing | 2,434 | 5.12 | 230 | 3.93 |
| Pathology | 1,135 | 2.39 | 151 | 2.58 |
Table 11: Per-image distribution of coarse (Level-1) modalities in the training and test splits. Counts are absolute image counts, and percentages are relative to the total number of images in each split.
| Modality | Train cases | Train (%) | Test cases | Test (%) |
|---|---|---|---|---|
| CT | 4,552 | 61.96 | 455 | 63.19 |
| MRI | 3,144 | 42.79 | 393 | 54.58 |
| X-ray | 2,749 | 37.42 | 248 | 34.44 |
| Ultrasound | 1,648 | 22.43 | 220 | 30.56 |
| Non-modality / Workflow / Post-processing | 1,411 | 19.21 | 151 | 20.97 |
| Pathology | 571 | 7.77 | 78 | 10.83 |
Table 12: Per-case coverage of coarse (Level-1) modalities. A case is counted for a modality if it contains at least one image annotated with that modality. Percentages are relative to the total number of cases in each split.
| Modality | Sub-modality | Train images | Train (%) | Test images | Test (%) |
|---|---|---|---|---|---|
| CT | Contrast-enhanced CT | 8,822 | 18.54 | 1,041 | 17.81 |
| CT | Non-contrast CT | 4,861 | 10.22 | 503 | 8.61 |
| CT | CT Angiography | 1,538 | 3.23 | 132 | 2.26 |
| CT | HRCT / Thin-slice CT | 1,399 | 2.94 | 202 | 3.46 |
| CT | Other_CT | 632 | 1.33 | 43 | 0.74 |
| CT | CT Perfusion | 28 | 0.06 | 8 | 0.14 |
Table 13: Per-image distribution of fine-grained (Level-2) sub-modalities, grouped by their corresponding coarse (Level-1) modality.
| # Modalities | Train cases | Train (%) | Test cases | Test (%) |
|---|---|---|---|---|
| 1 | 2,476 | 33.70 | 165 | 22.92 |
| 2 | 2,859 | 38.91 | 292 | 40.56 |
| 3 | 1,463 | 19.91 | 166 | 23.06 |
| 4 | 443 | 6.03 | 74 | 10.28 |
| 5 | 87 | 1.18 | 22 | 3.06 |
| 6 | 13 | 0.18 | 1 | 0.14 |
Table 14: Distribution of the number of distinct coarse (Level-1) modalities per case ( modalities_count ) in the training and test splits.
| # Sub-modalities | Train cases | Train (%) | Test cases | Test (%) |
|---|---|---|---|---|
| 1 | 1,593 | 21.68 | 94 | 13.06 |
| 2 | 2,152 | 29.29 | 183 | 25.42 |
| 3 | 1,819 | 24.76 | 196 | 27.22 |
| 4 | 1,051 | 14.31 | 132 | 18.33 |
| 5 | 441 | 6.00 | 66 | 9.17 |
| 6 | 183 | 2.49 | 30 | 4.17 |
Table 15: Distribution of the number of distinct fine-grained (Level-2) sub-modalities per case ( sub_modalities_count ) in the training and test splits.
| Split | # Cases | # Longitudinal | Share (%) |
|---|---|---|---|
| Train | 7,347 | 2,154 | 29.3 |
| Test | 720 | 219 | 30.4 |
| Overall | 8,067 | 2,373 | 29.4 |
Table 16: Prevalence of longitudinal studies in MedThinkVQA.
| timepoint_count | Train | Test | Overall |
|---|---|---|---|
| 2 | 1,631 (75.7%) | 163 (74.4%) | 1,794 (75.6%) |
| 3 | 379 (17.6%) | 41 (18.7%) | 420 (17.7%) |
| 4 | 115 (5.3%) | 9 (4.1%) | 124 (5.2%) |
| 5+ | 29 (1.3%) | 6 (2.7%) | 35 (1.5%) |
Table 17: Distribution of the number of imaging time points ( timepoint_count ) among longitudinal cases. Each cell shows # cases and the percentage relative to the longitudinal subset of the corresponding split.
| Interval category | Train | Test | Overall |
|---|---|---|---|
| Unknown / Unspecified | 555 (25.8%) | 56 (25.6%) | 611 (25.7%) |
| Hours | 52 (2.4%) | 12 (5.5%) | 64 (2.7%) |
| Days | 327 (15.2%) | 29 (13.2%) | 356 (15.0%) |
| Weeks | 210 (9.7%) | 16 (7.3%) | 226 (9.5%) |
| Months | 577 (26.8%) | 69 (31.5%) | 646 (27.2%) |
| Years | 363 (16.9%) | 30 (13.7%) | 393 (16.6%) |
Table 18: Coarse distribution of follow-up interval descriptions ( interval_text ) among longitudinal cases. Categories are assigned by keyword matching to temporal units and explicit calendar spans; percentages are relative to the longitudinal subset of each split.
| Max follow-up | Train | Test | Overall |
|---|---|---|---|
| 1 week | 230 (10.7%) | 30 (13.7%) | 260 (11.0%) |
| 1 week–1 month | 285 (13.2%) | 20 (9.1%) | 305 (12.9%) |
| 1–6 months | 431 (20.0%) | 56 (25.6%) | 487 (20.5%) |
| 6–12 months | 200 (9.3%) | 22 (10.0%) | 222 (9.4%) |
| 1–3 years | 191 (8.9%) | 14 (6.4%) | 205 (8.6%) |
| 3–5 years | 66 (3.1%) | 8 (3.7%) | 74 (3.1%) |
Table 19: Distribution of the maximum follow-up interval/span (estimated from interval_text ) among longitudinal cases. Each cell shows # cases and the percentage relative to the longitudinal subset of the corresponding split.
| Pairwise comparison | Cohen’s |
|---|---|
| Expert 1 vs. Expert 2 | 0.822833 |
| Expert 1 vs. LLM judge | 0.838357 |
| Expert 2 vs. LLM judge | 0.701566 |
Table 20: Inter-rater reliability on step factuality (Cohen’s ). High agreement with Expert 1 and substantial agreement with Expert 2 support the reliability of the LLM judge.
Figure 18: MELD similarity distributions on the MedThinkVQA test set across seven representative models. All distributions remain well below the 50% high-risk threshold, with similar behavior for text-only LLMs and VLMs.
Algorithm 1 MELD (original reproduction)