Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations

Naren Akash; Neeraja Ramanan

eess.IVAug 28, 2026

Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations

Authors: Naren Akash, Neeraja Ramanan

Organizations: Center for Visual Information Technology, IIIT Hyderabad, Hyderabad, India · Center for AI and Innovation, AIG Hospitals, Hyderabad, India

Abstract

Interpreting a CT scan means comparing structures on either side, judging how far apart organs sit, and knowing where each one belongs. Medical vision encoders are evaluated on diagnostic accuracy, or through assembled multimodal systems where a failure is hard to attribute, so it remains unclear whether their representations support any of this. We construct SPAR-Bench, eight probes over multi-organ abdominal CT that separate coordinate localization, relational reasoning, and spatial queries, and apply them to five architectural configurations and three medical foundation models, frozen and finetuned. Probes that ask for a comparison within the slice stay at chance, and neither pretraining scale, finetuning, nor architecture closes the gap. Probes that appear solved in domain fall to chance under zero-shot transfer, indicating that their accuracy reflects recall of canonical anatomy rather than computation over the image. Reading the same frozen features with a pooled head rather than the full set of tokens moves relational recovery from 0.7% to 67.8%, so pooled probing understates what a representation holds. Questions the encoders answer well are answered at chance by four open-weight MLLMs. Our results suggest these encoders carry a map of where organs usually lie, and little of the machinery for comparing structures within a particular patient. Code and data will be available at https://spar-bench.github.io.

Figures & tables

Figure 1: SPAR-Bench. (A) Organs are cropped into 16 tiles, shuffled for Level 2. (B) Level 1 maps tiles to normalized coordinates. (C) Level 3 pairs whole slices with six probe types.

Trained from scratch
	WORD (in domain)					TotalSegmentator (zero-shot)
Model	MED $\downarrow$	Perm	RelPos	Orient	Symm	MED $\downarrow$	Perm	RelPos	Orient	Symm
CNN-MLP	8.95	64.7	99.8	93.6	29.9	70.9	16.9	68.8	0 9.1	30.9
CNN-Trans	11.68	51.9	99.1	73.0	28.1	68.9	15.2	67.5	0 9.3	31.4
ViT-MLP	11.83	54.0	98.6	79.6	25.8	68.6	0 3.4	67.8	14.8	31.8
ViT-Trans	11.61	76.2	66.6	43.8	28.9	66.7	28.7	59.6	18.9	29.7

Table 1: Spatial reasoning performance, mean over three seeds (two for TotalSegmentator localization). MED is Mean Euclidean Distance in px; Perm is full permutation accuracy; RelPos, Orient and Symm are the relative position, orientation and bilateral symmetry probes. Accuracies in %. † Finetuned end-to-end. Bold marks the best value per column on WORD; no value is marked for symmetry, where all configurations sit at chance.

Model	Macro	RelPos	Orient	Symm	Size	Dist
		(n=1162)	(n=1050)	(n=444)	(n=712)	(n=170)
LLaVA-1.5-7B	34.9 $\pm$ 0.4	50.1 $\pm$ 1.1	20.2 $\pm$ 0.2	50.2 $\pm$ 0.4	0 7.7 $\pm$ 0.4	46.5 $\pm$ 3.1
Qwen2-VL-2B-Instruct	31.3 $\pm$ 0.3	49.5 $\pm$ 1.3	19.1 $\pm$ 0.4	20.5 $\pm$ 0.2	15.2 $\pm$ 0.9	52.2 $\pm$ 1.2
MedGemma-4B-IT	46.8 $\pm$ 2.4	50.0 $\pm$ 0.0	0 8.5 $\pm$ 2.0	50.5 $\pm$ 0.0	71.3 $\pm$ 10.3	53.7 $\pm$ 0.3
Qwen2-VL-7B-Instruct	50.6 $\pm$ 0.2	50.0 $\pm$ 0.0	20.0 $\pm$ 0.0	46.7 $\pm$ 0.7	82.7 $\pm$ 0.5	53.5 $\pm$ 0.0
CNN-MLP (best enc.)	78.5	99.8	93.6	29.9	82.1	87.2

Table 2: SPAR-VQA. Level 3 questions posed to four open-weight MLLMs (mean

\pm

std, 3 seeds, phrasing resampled per seed). All values in % (

\uparrow

). Encoder rows give the best and worst configuration from Table 1 for reference. Counting is omitted. Relative position is near ceiling for the encoders and at chance for every MLLM.

Trained from scratch
Model	Liver	Spln	K-L	K-R	Stom	GB	Esph	Panc	Duod	Colon	Intes	Adrn
CNN-MLP	8.3	8.0	7.2	7.1	10.3	9.1	8.6	9.1	8.1	11.5	10.9	9.5
CNN-Trans	10.7	10.6	8.3	8.8	14.1	11.4	9.0	11.9	9.6	16.8	15.0	12.1
ViT-MLP	11.6	10.6	8.5	8.4	14.3	11.3	7.3	11.1	10.0	17.6	15.3	11.2
ViT-Trans	11.2	10.2	8.0	8.3	13.8	10.7	10.1	10.8	9.6	17.6	16.1	11.0
ViT-Comb	11.1	10.0	8.2	8.2	13.5	11.6	8.5	10.9	9.6	17.2	15.1	11.0

Table 3: Per-organ localization error (MED, px, mean over three seeds) on WORD. Lower is better. Kidneys are easiest; colon and intestine hardest across configurations.

Trained from scratch
Model	Liver	Spln	K-L	K-R	Stom	GB	Esph	Panc	Duod	Colon	Intes	Adrn
CNN-MLP	98.5	96.6	97.2	97.5	92.6	85.3	53.8	88.7	88.1	94.9	87.3	92.4
CNN-Trans	98.3	94.3	96.2	95.9	87.8	70.2	10.3	79.0	78.4	93.3	74.9	87.7
ViT-MLP	98.1	95.4	96.8	97.3	84.9	81.8	30.8	84.1	82.8	89.9	78.1	92.6
ViT-Trans	99.2	97.4	98.7	98.9	92.5	88.2	79.5	94.9	90.5	96.9	87.7	96.8
ViT-Comb	97.0	93.5	95.4	96.2	78.4	73.7	48.7	81.6	79.9	85.2	75.1	87.7

Table 4: Per-organ permutation accuracy (patch accuracy, %, mean over three seeds) on WORD. Higher is better. Frozen MLP heads collapse on variable organs, most sharply the esophagus, while a Transformer head on the same features recovers them.

Trained from scratch
Model	RelPos	Orient	Symm	Size	Dist
CNN-MLP	68.8	9.1	30.9	38.4	48.2
CNN-Trans	67.5	9.3	31.4	34.3	49.1
ViT-MLP	67.8	14.8	31.8	32.7	46.9
ViT-Trans	59.6	18.9	29.7	34.7	54.2
ViT-Comb	68.9	8.3	31.1	31.5	49.1

Table 5: Full zero-shot transfer to TotalSegmentator, Level 3 probes (%, mean over three seeds; RadDINO-Trans † two seeds). No retraining. Symmetry stays near its 33.3% chance level for every configuration while other probes shift.

Explore similar work

A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans
Aug 21, 2026Simon Vincent Abel, Heiko Hillenhagen, Michael Götz +3Medical Image Analysis Medical VQA
Visual Computing Group, Institute of Media Informatics, Ulm University, Germany · Diagnostic and Interventional Radiology, Ulm University Hospital, Germany · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany +1
Sparse Concept Channels in Frozen 3D CT Vision Encoders
Jul 23, 2026Farhad Nooralahzadeh, Lea Bogensperger, Christian Bluethgen +1Radiology Report Generation 3D Medical Imaging
Department of Quantitative Biomedicine, University of Zurich, Switzerland · Center for Artificial Intelligence in Medicine and Imaging, Stanford University, USA
SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models
Jul 30, 2026Satvik Tripathi, Mustafa Ege Seker, Kristian Quevada +8Radiology VLMs Medical Image Analysis
Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA · Department of Radiology, University of Wisconsin–Madison School of Medicine and Public Health, Madison, Wisconsin, USA · Department of Radiology, Cooper University Hospital, Cooper Medical School of Rowan University, Camden, New Jersey, USA +5

Aug 21, 2026J/K move · Enter open · S save

A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans

Simon Vincent Abel, Heiko Hillenhagen, Michael Götz, Timo Ropinski+2

Visual Computing Group, Institute of Media Informatics, Ulm University, Germany · Diagnostic and Interventional Radiology, Ulm University Hospital, Germany · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany+1

Medical Image Analysis Medical VQA Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

Reliable spatial understanding is an important prerequisite for future medical vision-language systems that aim to support radiological report generation and structured image understanding. While modern vision-language models (VLMs) show promising performance on many medical imaging tasks, recent evidence suggests they remain weak in controlled spatial reasoning and often fail to reliably ground spatial relations in image evidence. Given that radiological reasoning hinges on understanding the relative positions of anatomical structures and findings, this spatial weakness poses risks to diagnostic accuracy. We present a modular medical imaging agent for binary spatial relation verification in axial CT slices. Instead of directly predicting spatial answers end-to-end, the system decomposes the task into explicit stages: language parsing, anatomical localization, and deterministic geometric verification. Natural-language queries are converted into structured relation tuples, queried organs are localized with a YOLO-based detector, and the final spatial decision is computed from object centers using deterministic geometric rules. We evaluate the approach on the held-out MIRP spatial QA benchmark and compare it against representative end-to-end VLM baselines. The best-performing hybrid configuration reaches 94.1% accuracy and 94.2% F1, outperforming direct Qwen2-VL prompting by 42.5 percentage points in accuracy, while preserving interpretable intermediate representations and auditable reasoning stages. The results suggest that explicit modular spatial verification can serve as a promising building block for future report-oriented medical imaging agents.

Abstract

Explore similar work

A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans

Sparse Concept Channels in Frozen 3D CT Vision Encoders

SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models