cs.CVSep 28, 2026

Medical Image Alignment Assessment as a Test of Generalist Visual Reasoning in Frontier Multimodal Models

Authors: Ross Callaghan, Niannu Gao, Hojjat Azadbakht, Hui Zhang

Organizations: Hawkes Institute, University College London, London, UK · AINOSTICS Ltd., Manchester, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Frontier multimodal large language models (MLLMs) are increasingly positioned as general purpose visual reasoners as part of the quest for artificial general intelligence. A key test of this generality is whether they can perform novel visual judgments that humans can make reliably from visual evidence and task instructions, without task-specific parameter optimisation. We investigate this question through the task of medical image alignment assessment, where the goal is to establish whether there is anatomical correspondence between two images. Human visual assessment of image alignment is still the gold standard and most common approach; however, it requires trained operators and is impractical to scale for large datasets. We evaluate recent generations of MLLMs on two exemplar medical image alignment tasks, varying both prompting strategies and image-presentation methods. We compare against a locally fine-tuned MLLM and a task-specific CNN to examine the trade-off between frontier general purpose models and smaller models that require specific task optimisation but can be used locally. We show that are reaching an inflection point, where frontier MLLMs can now perform effective visual assessment of medical image alignment. Models released only a few months ago generalise poorly and, in some settings, perform barely above chance, whereas GPT-6 achieves over 85% across almost all scenarios tested. Fine-tuned local models can match or exceed frontier-model performance on the tasks on which they are trained, but transfer substantially less effectively to unseen settings. These findings identify medical image alignment as a useful test bed for generalist visual reasoning and suggest that frontier multimodal models are beginning to acquire capabilities that could support a common quality-control mechanism across heterogeneous medical-imaging pipelines.

Figures & tables

Figure 1: Illustration of the medical image alignment assessment problem.
Figure 2: Overview of the medical image alignment assessment method. We test image alignment assessment on two modality pairs: cross-scanner T1w MRI-PET and within MRI T1w-FLAIR alignment. We highlight how different modality pairs and image-presentation techniques can affect the alignment assessment task. Yellow zoom boxes show misalignment of ventricles, something which is easy to spot for MRI-PET contours and T1w-FLAIR alpha-blend (solid yellow boxes) but harder to see in MRI-PET alpha-blend (dashed yellow box). Red zoom boxes show brain-edge misalignment which is easy to spot in MRI-PET alpha-blend (dashed red box) but harder to definitively see in MRI-PET contours (solid red box).
Model FamilyModelRelease Date
GPTGPT-5.2Dec ‘25
GPT-5.4Mar ‘26
GPT-5.6 LunaJul ‘26
GPT-5.6 TerraJul ‘26
GPT-5.6 SolJul ‘26
GPT-6 AstraSep ‘26
Table 1: Models tested in this work. 1 MedGemma tested both as zero/few-shot MLLM in Experiment 1 and locally fintuned model in Experiment 2. 2 ResNet tested as locally finetuned model in Experiment 2.
Figure 3: Example input images in an aligned case for a) contours presentation with anatomical segmentation contours from T1w MRI overlaid on the PET and b) alpha-blend presentation with the T1w image overlaid directly on the pet with 50% opacity.
Image PresentationContoursAlpha-blend
No. Shot Examples (a/m)0/01/11/201/11/2
Model FamilyModel      
GPTGPT-5.257.763.458.767.388.591.3
GPT-5.452.963.558.754.892.386.5
GPT-5.6 Luna51.048.150.057.781.779.8
GPT-5.6 Terra51.050.053.871.283.786.5
Table 2: Misalignment detection accuracy across MLLM models, zero/few shot prompting and image presentation method. a/m refers to number of aligned/misaligned examples used in the few-shot case. Values reported are accuracy as a percentage. Boldface numbers denote the best performing model per-column, underlined numbers the second-best.
Figure 4: MLLM misalignment detection performance over time across all six scenarios tested.
  No. Shot Examples (a/m)
  0/01/1
Model FamilyModelT1w-FLAIRMRI-PETT1w-FLAIRMRI-PET
GPTGPT-5.6 Luna65.0 (73/59/63)57.791.3 (88/86/97)81.7
GPT-6 Astra91.3 (81/95/97)87.596.3 (88/100/100)97.1
GeminiGemini 3.1 Pro90.0 (96/91/84)85.695.0 (92/95/97)83.7
Gemini 3.8 Flash86.3 (88/100/75)70.291.3 (88/95/91)81.7
Table 3: Misalignment detection in the T1w-FLAIR experiment, compared to equivalent performance on MRI-PET. Small numbers in brackets represent the performance on the A4/ADNI/MICCAI subsets respectively.
Figure 5: Local model performance with training set size for both the within-distribution test set (MRI-PET) and out-of-distribution test set (T1w-FLAIR). Grey line represents best frontier MLLM performance on the same task (MRI-PET with contour image presentation)
More in this topic

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

Explore similar work

CardsList
Mar 15, 2026

How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images

Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks, particularly in zero-shot settings where generalization is critical, remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle visual grounding from semantic grounding, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across eight state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images. Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance. Additional experiments are included in the Supp.

Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani +3

Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore

Medical VLMsMedical Image GroundingVLM EvaluationMedical VQA
AskPDF
Apr 14, 2026

Medical thinking with multiple images

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Zonghai Yao, Benlu Wang, Yifan Zhang +8

Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University +2

Medical VQAMultimodal ReasoningMulti-Image ReasoningMedical Image BenchmarksMedical Image Grounding
AskPDF
Jun 16, 2026

See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RL

Multimodal large language models (MLLMs) integrate strong text reasoning with visual inputs, yet their responses can be inconsistent with the underlying images, indicating ineffective utilization of visual evidence during inference. The prevailing training paradigm relies on large-scale caption-based pretraining for general alignment, followed by supervised fine-tuning and reinforcement learning to enable instruction following and complex reasoning. However, such pretraining provides only weak visual grounding: short, coarse captions bias models toward salient objects while neglecting fine-grained visual evidence. In this paper, we introduce Visual Evidence Pre-Alignment (VEPA), an intermediate stage between pretraining and post-training that explores a novel sufficiency-driven objective with Group Relative Policy Optimization (GRPO) to optimize question-conditioned visual evidence descriptions. Extensive experiments across diverse benchmarks show that our VEPA consistently enhances performance on visually demanding evaluations and complements standard supervised post-training. Further analyses show that the income stems from strengthened, transferable visual grounding, rather than from additional task-specific training.

Yilian Liu, Sicong Leng, Guoshun Nan +7

Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · China Telecom, China

Multimodal PretrainingVisually Grounded ReasoningVisual GroundingVision-Language AlignmentRL for VLMs
AskPDF