Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning

Xueqi Ma; Shuo Yang; Yanbei Jiang; Shu Liu; Zhenzhen Liu; Jiayang Ao; Xingjun Ma; Sarah Monazam Erfani

cs.AIMar 21, 2026

Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning

Authors: Xueqi Ma, Shuo Yang, Yanbei Jiang, Shu Liu, Zhenzhen Liu, Jiayang Ao, Xingjun Ma, Sarah Monazam Erfani,

Organizations: The University of Melbourne · Australian National University · Fudan University · Monash University

Abstract

Despite remarkable advances in large Vision-Language Models (VLMs), spatial reasoning remains a persistent challenge. In this work, we investigate how attention heads within VLMs contribute to spatial reasoning by analyzing their functional roles through a mechanistic interpretability lens. We introduce CogVSR, a dataset that decomposes complex spatial reasoning questions into step-by-step subquestions designed to simulate human-like reasoning via a chain-of-thought paradigm, with each subquestion linked to specific cognitive functions such as spatial perception or relational reasoning. Building on CogVSR, we develop a probing framework to identify and characterize attention heads specialized for these functions. Our analysis across diverse VLM families reveals that these functional heads are universally sparse, vary in number and distribution across functions. Notably, spatially specialized heads are fewer than those for other cognitive functions, highlighting their scarcity. We propose methods to activate latent spatial heads, improving spatial understanding. Intervention experiments further demonstrate their critical role in spatial reasoning: removing functional heads leads to performance degradation, while emphasizing them enhances accuracy. This study provides new interpretability driven insights into how VLMs attend to space and paves the way for enhancing complex spatial reasoning in multimodal models.

Figures & tables

Figure 1: To solve a complex question, the human brain engages multiple regions to perform distinct cognitive functions necessary for generating a response. We explore whether there are specific attention heads in VLMs that play functional roles in producing answers.

Main / Image	Subquestion	Answer	Cognitive Function Label
Main Question: Judge whether the text is true or false: “The dog is facing the horse.” Main Answer: True.	1. Which animals appear in the image?	A dog and a horse.	High-level Visual Perception
	2. Where is the dog located relative to the horse?	The dog is on the right and the horse is on the left.	Spatial Perception, High-level Visual Perception
	3. Which way is the dog’s head oriented?	The dog’s head is facing left.	Spatial Perception, High-level Visual Perception
	4. Is the direction the dog is facing pointed toward the horse’s position?	Yes.	Relational Reasoning
	5. Based on the dog’s orientation toward the horse, is the statement “The dog is facing the horse” true or false?	True.	Decision Making

Table 1: An example from the CogVSR dataset, illustrating a main question, its final answer, and a breakdown into subquestions with corresponding answers and multi-label cognitive function annotations.

Figure 2: The existence of cognitive heads in Qwen2.5-VL-7B responsible for eight distinct functions in complex reasoning tasks. The x-axis represents the head index, while the y-axis indicates the layer index. The values denote head importance scores.

Model	Inter_Head	Spatial Functions				Other Functions
		Spatial		Relational		Low-Level		High-Level		Info		Recall		Math		Decision
		llm	acc	llm	acc	llm	acc	llm	acc	llm	acc	llm	acc	llm	acc	llm	acc
Intern2B	random	61.37	56.82	63.25	58.13	65.48	60.91	69.72	64.35	71.19	66.04	69.87	64.11	59.44	54.27	64.03	58.76
Intern2B	cognitive	37.92	34.37	31.56	26.88	13.43	18.21	34.27	28.65	39.82	33.17	37.11	31.49	23.68	18.02	21.54	16.43
Intern8B	random	72.84	68.12	74.33	69.57	76.05	71.44	81.26	76.09	83.17	78.03	81.92	76.48	73.66	68.40	77.41	72.05
Intern8B	cognitive	46.39	42.07	52.84	48.51	36.12	31.76	59.73	53.28	65.04	58.69	62.91	56.33	49.82	44.15	46.37	40.88

Table 2: Intervention results (%) of cognitive heads vs. random heads across 8 functions: Spatial Perception, Relational Reasoning, Low-level Visual Perception, High-level Visual Perception, Language Info rmation Extraction and Understanding, Knowledge Recall , Math Reasoning, and Decision Making. For model names, Intern2B: InternVL3-2B, Intern8B: InternVL3-8B, Llama11B: Llama3.2-11B-Vision, Llama90B: Llama3.2-90B-Vision, Qwen3B: Qwen2.5-VL-3B-Instruct, Qwen7B: Qwen2.5-VL-7B-Instruct. Lower values indicate more effective intervention outcomes, suggesting that the corresponding heads play a greater role in the cognitive function.

Figure 3: Objects with applied masks and bounding boxes as input.

Figure 4: Comparison of spatial perception and relational reasoning heads before and after spatial head activation across various models.

Model	Input	Spatial		Relational
Model	Input	llm	acc	llm	acc
Intern2B	Original	61.37	56.82	63.25	58.13
	+BBox	65.84	58.94	64.95	62.39
	+Mask	59.70	59.80	66.82	56.47
	+BBox+Mask (Ours)	71.82	68.64	70.12	67.04
Llama90B	Original	86.21	82.07	88.03	83.92

Table 3: Effect of spatial input augmentation on spatial and relational functions. “Original” uses only the raw image; “+BBox” and “+Mask” add bounding-box and instance-mask cues respectively; “+BBox+Mask (Ours)” incorporates both.

Dataset	Inter_Head	Intern8B	Qwen7B	Llama11B
VSR	before	64.35	62.47	65.82
VSR	after	24.13	21.76	17.94
Spatial457	before	41.23	38.71	32.18
Spatial457	after	14.97	12.43	9.86
SpatialEval	before	47.92	45.33	39.61
SpatialEval	after	20.08	18.57	15.19

Table 4: Negative intervention on visual spatial reasoning benchmarks. Scores are LLM-Judge accuracy (%). “before” uses the original model; “after” applies intervention.

Model	Inter_Head	In Domain								Out of Domain
Model	Inter_Head	Spatial	Relational	Low	High	Info	Recall	Math	Decision	VSR	Spatial457	SpatialEval	3DSRBench
Qwen7B	before	80.04	82.11	84.39	88.27	90.02	88.39	82.76	86.51	62.47	38.71	45.33	34.16
Qwen7B	after	81.89	83.76	84.10	90.63	92.41	90.92	82.90	87.74	63.72	39.15	45.81	34.95
Llama11B	before	74.95	76.68	78.37	83.29	85.06	83.64	75.92	79.18	65.82	32.18	39.61	28.87
Llama11B	after	75.62	77.40	79.10	83.75	86.20	83.10	75.40	79.65	66.91	32.40	39.90	29.40
Intern8B	before	72.84	74.33	76.05	81.26	83.17	81.92	73.66	77.41	64.35	41.23	47.92	36.58

Table 5: Positive intervention on spatial reasoning benchmarks (200 samples per dataset). Scores are LLM-Judge accuracy (%). For out-of-domain data, we positively intervene on spatial perception and relational reasoning heads.

Figure 5: The existence of cognitive heads in Intern3VL-2B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 6: The existence of cognitive heads in LLaMA3.2-11B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 7: The existence of cognitive heads in LLaMA3.2-90B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 8: The existence of cognitive heads in Qwen2.5VL-3B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 9: The existence of cognitive heads in Intern3VL-8B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 10: The performance of Qwen2.5-VL-7B after masking out top K cognitive heads vs K random heads on inference, high-level visual reception, low-level visual reception, and decion-masking.

Token_use	Spatial(llm)	Spatial(acc)	Relational(llm)	Relational(acc)
first	70.42	72.54	75.28	66.46
last	84.33	83.26	78.54	70.18
full	67.46	59.28	70.28	63.33
top-k	57.69	53.42	63.85	59.57

Table 6: Performance of different token usage strategies for Qwen2.5-VL-7B. Lower is Better.

Dataset	InternVL3-2B	InternVL3-8B	Qwen2.5VL-7B	Qwen2.5VL-3B	LLaMA3.2-11B	LLaMA3.2-90B
CogVSR	78.74	75.88	82.15	84.22	89.64	83.44

Table 7: The test accuracy (%) of probing method on different LLMs.

Figure 11: A detailed case in CogVSR dataset.

Figure 12: A qualitative example of turn-based maze reasoning. The model first needs to decompose the path into cardinal directions, counts direction changes, classifies the type of turn (right vs. left), and finally determines the total number of right turns along the path.

Figure 13: A qualitative example of grid-based counting. The model need to first counts rabbits row by row and then aggregates these subcounts to obtain the total number of rabbit-containing blocks in the grid.

Cognitive Skill	Count	Percentage
Spatial Perception	1,026	27.29%
High-level Visual Perception	941	25.03%
Relational Reasoning	576	15.32%
Decision Making	498	13.25%
Knowledge Recall	272	7.24%
Information Extraction and Understanding	198	5.27%

Table 8: Distribution of cognitive skills across 3,759 subquestions.

Figure 14: Importance curve for eight functions of Qwen2.5-VL-7B.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
Aug 24, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Visual Spatial Reasoning Spatial Reasoning in Language Models
Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool
When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning
Feb 25, 2026Muku Akasaka, Soyeon Caren HanVisual Spatial Reasoning VLM Reasoning
The University of Melbourne Melbourne, Victoria, Australia
Soft Spatial Reasoning
Sep 30, 2026Rafi Ibn Sultan, Md. Sajid Alam Chowdhury, Saleh Zare Zade +4Visual Spatial Reasoning VLM Reasoning
Department of Computer Science, Wayne State University · Department of Radiation Oncology, Henry Ford Health · Department of Electrical and Computer Engineering, The Ohio State University +1

Aug 24, 2026J/K move · Enter open · S save

Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?

Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li+4

Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool

Visual Spatial Reasoning Spatial Reasoning in Language Models Vision-Language Grounding VLM Interpretability

Open paper AskPDF arXiv

Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.

Abstract

Visual Spatial Reasoning

Explore similar work

Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?

When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning

Soft Spatial Reasoning