Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?

Xiwei Liu; Yulong Li; Xinlin Zhuang; Xuhui Li; Zhixiang Lu; Haolin Yang; Imran Razzak; Yutong Xie

cs.CVAug 24, 2026

Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?

Authors: Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li, Zhixiang Lu, Haolin Yang, Imran Razzak, Yutong Xie

Organizations: Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool

Abstract

Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.

Figures & tables

Figure 1: (a) and (b): The geometry of the 1D absolute position embedding in LLaVA-1.5-7B. The visualization is performed via t-SNE for dimensionality reduction. The labels are set to row IDs and column IDs, respectively. (c) and (d): The PCA results of the direction vectors for left of / right of / in front of / behind with and without position embedding in Qwen2.5-7B.

Figure 2: Examples in the curated What’s Up dataset.

Models:	LLaVA-1.5 7B				LLaVA-1.5 13B			Qwen2.5-VL 7B
Strategy	Token (%)	Loc-T (%)	Loc-R (%)	Rel. (%)	Loc-T (%)	Loc-R (%)	Rel. (%)	Token (%)	Loc-T (%)	Loc-R (%)	Rel. (%)
Baseline	0	10.15	14.81	24.82	29.91	39.03	40.23	0	84.53	86.64	96.11
− 1 Padding	1.77	8.79 $\downarrow$ 1.36	14.33 $\downarrow$ 0.48	24.61 $\downarrow$ 0.21	23.51 $\downarrow$ 6.40	39.25 $\uparrow$ 0.22	40.07 $\downarrow$ 0.16	2.24	84.58 $\uparrow$ 0.05	86.75 $\uparrow$ 0.11	95.93 $\downarrow$ 0.18
Target	4.94	4.78 $\downarrow$ 5.37	14.71 $\downarrow$ 0.10	24.59 $\downarrow$ 0.23	2.01 $\downarrow$ 27.90	39.90 $\uparrow$ 0.87	40.05 $\downarrow$ 0.18	4.23	19.60 $\downarrow$ 64.93	86.32 $\downarrow$ 0.32	94.79 $\downarrow$ 1.32
+ 1 Padding	9.52	0.38 $\downarrow$ 9.77	17.59 $\uparrow$ 2.78	24.57 $\downarrow$ 0.25	0.12 $\downarrow$ 29.79	36.91 $\downarrow$ 2.12	27.85 $\downarrow$ 12.38	6.71	2.55 $\downarrow$ 81.98	84.69 $\downarrow$ 1.95	89.74 $\downarrow$ 6.37
− 1 Padding	2.59	10.53 $\uparrow$ 0.38	14.12 $\downarrow$ 0.69	24.75 $\downarrow$ 0.07	29.64 $\downarrow$ 0.27	31.38 $\downarrow$ 7.65	39.90 $\downarrow$ 0.33	3.26	84.26 $\downarrow$ 0.27	88.60 $\uparrow$ 1.96	96.42 $\uparrow$ 0.31

Table 1: Performance under token ablation. Baseline removes no token. Target and reference ablations replace visual tokens selected from the corresponding object masks with a global average visual embedding; padding variants shrink or dilate the object-token mask by one token. Token (%) is the average percentage of image tokens replaced. Loc-T (Target) and Loc-R (Reference) report localization accuracy averaged over IoU

@

0.5/0.7/0.9, and Rel. reports spatial relation accuracy.

Figure 3: Layer-wise positional decoding and logit lens. (a) Joint two-dimensional position-probe accuracy across the LLM decoder layers of LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-7B. (b) Axis-specific position-probe accuracy. (c) Spatial-relation answer accuracy obtained at each layer by applying a training-free logit lens to the final prompt-token representation. Error bars and shaded regions denote the standard error across samples.

Figure 4: Performance after attention knockout. We block attention to object-aligned visual tokens within grouped decoder layers and across all layers. Lines and bars report localization and spatial-relation accuracy, respectively, under the Padding=0 (P0) and Padding=1 (P1) settings. Object-token knockout is compared with a size-matched random-token control. Solid and dash-dotted horizontal lines indicate the localization and spatial-relation accuracy of the unmodified model.

Figure 5: Causal mediation via activation patching. We compare three model runs: (a) the source clean run, where the object is present and the model produces the correct answer; (b) the base counterfactual run, where the target object is removed and the model fails; and (c) the patched counterfactual run, where we transfer hidden activations from a selected attention head in the source run into the base run. Improvements in the patched prediction indicate that the transferred head carries task-relevant information. This procedure is repeated head-wise to quantify each head’s causal contribution.

Figure 6: Head-level causal mediation analysis. Mediation Fraction (MF) scores for every attention head across the decoder layers, shown separately for the localization and spatial reasoning tasks.

Figure 7: Performance under cumulative ablation of attention heads. (a) Localization and (b) spatial-relation accuracy as increasing fractions of localization-important, relation-important, unimportant, and negative-MF heads are ablated.

Explore similar work

Mechanisms of Object Localization in Vision-Language Models
May 19, 2026Timothy Schaumlöffel, Martina G. Vilas, Gemma RoigVision-Language Models VLM Interpretability
Goethe University Frankfurt, Germany · The Hessian Center for AI, Germany
SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models
Sep 30, 2026Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury +4Visual Spatial Reasoning Large Vision-Language Models
Department of Computer Science, Wayne State University · Department of Radiation Oncology, Henry Ford Health · Department of Electrical and Computer Engineering, The Ohio State University +1
Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
Aug 10, 2026Hunter Schofield, Mohammed Elmahgiubi, Mohammad Mahdavian +4Geometric Representation Learning Vision-Language Grounding
York University · Huawei Technologies Canada · University of Toronto

May 19, 2026, cs.CVJ/K move · Enter open · S save

Mechanisms of Object Localization in Vision-Language Models

Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig

Goethe University Frankfurt, Germany · The Hessian Center for AI, Germany

Vision-Language Models VLM Interpretability Vision-Language Grounding

Open paper AskPDF arXiv

Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.

Abstract

Explore similar work

Mechanisms of Object Localization in Vision-Language Models

SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models

Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models