Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
Figures & tables
Figure 1: (a) and (b): The geometry of the 1D absolute position embedding in LLaVA-1.5-7B. The visualization is performed via t-SNE for dimensionality reduction. The labels are set to row IDs and column IDs, respectively. (c) and (d): The PCA results of the direction vectors for left of / right of / in front of / behind with and without position embedding in Qwen2.5-7B.
Figure 2: Examples in the curated What’s Up dataset.
Models:
LLaVA-1.5 7B
LLaVA-1.5 13B
Qwen2.5-VL 7B
Strategy
Token (%)
Loc-T (%)
Loc-R (%)
Rel. (%)
Loc-T (%)
Loc-R (%)
Rel. (%)
Token (%)
Loc-T (%)
Loc-R (%)
Rel. (%)
Baseline
0
10.15
14.81
24.82
29.91
39.03
40.23
0
84.53
86.64
96.11
− 1 Padding
1.77
8.79 ↓ 1.36
14.33 ↓ 0.48
24.61 ↓ 0.21
23.51 ↓ 6.40
39.25 ↑ 0.22
40.07 ↓ 0.16
2.24
84.58 ↑ 0.05
86.75 ↑ 0.11
95.93 ↓ 0.18
Target
4.94
4.78 ↓ 5.37
14.71 ↓ 0.10
24.59 ↓ 0.23
2.01 ↓ 27.90
39.90 ↑ 0.87
40.05 ↓ 0.18
4.23
19.60 ↓ 64.93
86.32 ↓ 0.32
94.79 ↓ 1.32
+ 1 Padding
9.52
0.38 ↓ 9.77
17.59 ↑ 2.78
24.57 ↓ 0.25
0.12 ↓ 29.79
36.91 ↓ 2.12
27.85 ↓ 12.38
6.71
2.55 ↓ 81.98
84.69 ↓ 1.95
89.74 ↓ 6.37
− 1 Padding
2.59
10.53 ↑ 0.38
14.12 ↓ 0.69
24.75 ↓ 0.07
29.64 ↓ 0.27
31.38 ↓ 7.65
39.90 ↓ 0.33
3.26
84.26 ↓ 0.27
88.60 ↑ 1.96
96.42 ↑ 0.31
Table 1: Performance under token ablation. Baseline removes no token. Target and reference ablations replace visual tokens selected from the corresponding object masks with a global average visual embedding; padding variants shrink or dilate the object-token mask by one token. Token (%) is the average percentage of image tokens replaced. Loc-T (Target) and Loc-R (Reference) report localization accuracy averaged over IoU @ 0.5/0.7/0.9, and Rel. reports spatial relation accuracy.
Figure 3: Layer-wise positional decoding and logit lens. (a) Joint two-dimensional position-probe accuracy across the LLM decoder layers of LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-7B. (b) Axis-specific position-probe accuracy. (c) Spatial-relation answer accuracy obtained at each layer by applying a training-free logit lens to the final prompt-token representation. Error bars and shaded regions denote the standard error across samples.
Figure 4: Performance after attention knockout. We block attention to object-aligned visual tokens within grouped decoder layers and across all layers. Lines and bars report localization and spatial-relation accuracy, respectively, under the Padding=0 (P0) and Padding=1 (P1) settings. Object-token knockout is compared with a size-matched random-token control. Solid and dash-dotted horizontal lines indicate the localization and spatial-relation accuracy of the unmodified model.
Figure 5: Causal mediation via activation patching. We compare three model runs: (a) the source clean run, where the object is present and the model produces the correct answer; (b) the base counterfactual run, where the target object is removed and the model fails; and (c) the patched counterfactual run, where we transfer hidden activations from a selected attention head in the source run into the base run. Improvements in the patched prediction indicate that the transferred head carries task-relevant information. This procedure is repeated head-wise to quantify each head’s causal contribution.
Figure 6: Head-level causal mediation analysis. Mediation Fraction (MF) scores for every attention head across the decoder layers, shown separately for the localization and spatial reasoning tasks.
Figure 7: Performance under cumulative ablation of attention heads. (a) Localization and (b) spatial-relation accuracy as increasing fractions of localization-important, relation-important, unimportant, and negative-MF heads are ablated.
Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.
Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig
Goethe University Frankfurt, Germany · The Hessian Center for AI, Germany
Large Vision-Language Models (LVLMs) have made remarkable progress across visual perception tasks, yet spatial reasoning remains a persistent weakness, especially for questions that require reasoning over visual space. Recent spatial-reasoning methods incorporate generated grounding, where models predict bounding boxes, masks, or other localization outputs for task-relevant objects as part of their reasoning trace. However, these approaches typically optimize final-answer correctness alone, allowing correct answers to be rewarded even when the model does not reason from confidently localized task-relevant objects. We introduce SpatialCORE (Spatially COnfident REasoning), a post-training framework that turns the model's own confidence in generated grounding into a learning signal for spatial reasoning. Its central idea is to reinforce grounding that is both accurate and confident, encouraging the model to reason from confidently localized task-relevant objects. SpatialCORE realizes this through a self-regulating spatial reward that weights each predicted bounding box's matching quality by its coordinate-token confidence. An answer gate further ties grounding optimization to final-answer correctness. SpatialCORE achieves state-of-the-art results among open-source and specialized spatial reasoning models across diverse benchmarks, and transfers effectively in zero-shot settings to unseen data distributions. The source code is available at https://github.com/rafiibnsultan/SpatialCORE.
Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury +4
Department of Computer Science, Wayne State University · Department of Radiation Oncology, Henry Ford Health · Department of Electrical and Computer Engineering, The Ohio State University +1
Spatial understanding is fundamental to embodied intelligence, underpinning applications such as robotic manipulation, embodied navigation, and autonomous driving. Although recent vision-language models (VLMs) have achieved impressive performance on spatial reasoning benchmarks, state-of-the-art approaches typically rely on additional spatial encoders or architectural modifications during inference, increasing computational cost. We introduce Space Tokens, a lightweight, architecture-agnostic framework that equips VLMs with explicit continuous spatial representations without requiring additional inference-time modules. By distilling scene-level 3D geometry and object-centric spatial attributes into continuous latent tokens, our method enables these modalities to be directly incorporated into a chain-of-thought reasoning process, thereby improving the VLM's spatial reasoning capabilities. At the same time, the learned representations can be explicitly decoded to verify that they encode meaningful geometric information, while the unified token interface remains extensible to additional modalities. Experiments on VSI-Bench improve Qwen3-VL-8B by 4.3% and SenseNova-SI-1.3 by 1.3%, while achieving state-of-the-art performance on object size (79.2%) and room size estimation (75.7%). These results demonstrate that continuous spatial tokens provide an effective, interpretable, and computationally efficient mechanism for integrating geometric reasoning into large vision-language models.
Hunter Schofield, Mohammed Elmahgiubi, Mohammad Mahdavian +4
York University · Huawei Technologies Canada · University of Toronto