Vision-language models are known to encode spatial information in their hidden states, yet often fail to use it when answering. However, it remains unclear when and where this encoded information reaches the answer. We address this with direction patching, a class-conditioned causal intervention applied across layers, token positions, and prompt formats. Using spatial-ID directions constructed following prior encoding evidence, we find that causal influence on answer logits emerges only at mid-to-deep depths. Text chain-of-thought suppresses immediate object-word argmax-level transport in most models, while visually grounded prompts keep it open. Positive target-logit gain can remain below the argmax threshold, and transport can re-emerge at the final prefix token or at the answer step in deeper layers. Across the ten VLMs we study, these local effects form descriptive transport patterns. Complementary experiments characterize how these patterns shift across datasets, attributes, and encoding amplitudes. Together, these results reframe the encoding-grounding gap as a problem of conditional transport in VLMs.
Figures & tables
Figure 1: Direction patching reveals conditional transport. Spatial encoding is broadly present across VLMs, but whether it reaches the answer depends on prompt type, token position, and layer depth. Text CoT can block object-word transport, whereas visual grounding prompts keep this pathway open.
Model
Family
Params
Layers
Qwen2.5-VL-7B
Qwen
7B
28
Qwen2.5-VL-32B
Qwen
32B
64
LLaVA-OneVision-7B
LLaVA
7B
28
InternVL2.5-8B
InternVL
8B
32
InternVL3-8B
InternVL
8B
28
InternVL3-14B
InternVL
14B
48
Table 1: Ten evaluated VLMs spanning five families, including three within-family scale pairs: Qwen 7B/32B, InternVL3 8B/14B, and Gemma-3 4B/12B/27B. Row tints group rows by family.
Figure 2: Layer-wise Δ argmax for ten models (obj_word, answer_only, α=5 ). Argmax-level transport activates only at mid-to-deep layers, although the Kang-style spatial-ID direction is constructed from early-layer representation evidence.
Model
CoT
Vis-CoT
Ans-only
Qwen-7B
+.081
+.136
+.169
Qwen-32B
+.006
+.117 ∗
+.089
LLaVA
.000
+.103
+.158
InternVL2.5
.000
+.153 ∗
+.114
InternVL3-8B
.000
+.164
+.236
InternVL3-14B
.000
+.117 ∗
+.097
Table 2: Δ argmax at obj_word at each model’s visual_cot peak layer (RefCOCO, α=5 ). ∗ : visual_cot exceeds answer_only. All ten models recover positive transport under visual_cot, including eight suppressed under text CoT.
Figure 3: Position-dependent transport (GQA Spatial, three models). Left: InternVL3-8B shows the relocation pattern (obj_word L12 → prefix_last L20; CoT gated only at obj_word). Center: Janus concentrates at obj_word. Right: Qwen-7B transports at both positions under both prompts.
Figure 4: Per-model CoT gating signature (obj_word, RefCOCO, α=5 ). Eight of ten models fall below the descriptive 0.01 threshold: six are exact-zero and two are near-zero/borderline; Qwen-7B retains substantial CoT transport. Numerical values in App. B .
Pattern
Models
Obj. route
Reloc.
Open
Qwen-7B
Open
Yes
Reduced supp.,reloc.
LLaVA , IV2.5 , IV3-8B , IV3-14B
Supp./red.
Yes
Suppressed, non-reloc.
Gemma-3 (4/12/27B), Janus
Supp.
No
Prompt- selective
Qwen-32B
Near-zero
Sel.
Table 3: Four descriptive transport patterns across ten models. “Reloc.”: whether transport reappears at prefix_last (Yes / Sel. = prompt-selective / No). Colors match Figure 4 .
Figure 5: Attention head knockout at L27 of InternVL3-8B ( n=100 , RefCOCO, α=5 ). (a) Under CoT, knocking out H21 recovers +0.063Δ argmax, the only head above the significance threshold among all 32 heads. (b) Under answer-only, H21 has no effect ( +0.007 ). H21 is a CoT-specific gating candidate in this scan, rather than a general transport component.
Prediction
Match
Note
CoT suppression at obj_word
9/10
Qwen-7B : remains open
Peak layer ± 8L
6/6
original 6 models
pfx_last deeper than obj_word
8/10
Janus , Gem-27B anomalous
Table 4: Structural transfer from RefCOCO to GQA Spatial. Three transport-map predictions tested with independent images, scene graphs, and object distributions.
Model
Spatial peak
Color peak
Shift
Qwen-7B
L8
L4
−4
Qwen-32B
L24
L16
−8
LLaVA
L8
L4
−4
InternVL2.5
L8
L4
−4
InternVL3-8B
L12
L4
−8
InternVL3-14B
L12
L12
0
Table 5: Obj_word peak layer on GQA Spatial vs. GQA Color (answer_only), all ten models. Color peaks ∼4 – 8 layers earlier than spatial in 9/10 models; InternVL3-14B is the lone exception (same peak layer).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Format
Instruction suffix
cot
“Let’s think step by step.”
brief_reason
“Give a brief reason, then answer with the letter.”
answer_first
“First, answer with only the letter (A–D). Then explain briefly.”
final_tag
“Think about it, then give your final answer as Final Answer: X”
answer_only
“Answer with only the letter (A–D).”
visual_cot
“First, carefully look at the image and describe where you see the object mentioned above. Then answer with only the letter.”
Appendix
Table 6: Verbatim instruction suffixes appended to a shared question stem. The first five suffixes ( cot through answer_only ) span a reasoning-length spectrum and are used in the prompt ladder of § 4.1 ; the last two ( visual_cot , visual_direct ) are visual-grounding prompts evaluated in § 3.3 .
Model
CoT
Ans-only
Gated?
Qwen-7B
+0.120 L16
+0.156 L16
No
InternVL2.5
+0.042 L12
+0.168 L12
Weak
LLaVA
+0.006 L10
+0.168 L16
Yes
InternVL3-8B
0.000 all
+0.236 L16
Yes
InternVL3-14B
0.000 all
+0.097 L24
Yes
Gemma-3-4B
0.000 all
+0.178 L12
Yes
Appendix
Table 7: Peak obj_word Δ argmax under CoT and answer-only (RefCOCO, α=5 ), with the layer at which each peak is attained. The sample count is n=50 /quadrant for 7B/8B models and n=30 /quadrant for Qwen-32B, Gemma-3, and Janus; each source sample is paired with the three non-source target quadrants. Top two rows (above the midrule) are open or partially suppressed; the bottom eight rows fall at or below 0.006 .
Figure 6: Wrong-axis and non-object patching produce near-zero Δ argmax, confirming direction and position specificity.
Model
Lyr
Target
Wrong
Non-obj
Qwen-7B
16
+.156
+.026
−.008
LLaVA
16
+.168
−.018
+.002
IV2.5
12
+.168
+.018
−.004
IV3-8B
16
+.236
+.028
−.012
IV3-14B
24
+.097
+.006
+.003
Gem-4B
12
+.178
−.003
+.006
Appendix
Table 8: Direction and position controls (RefCOCO, ans-only, α=5 ). Wrong: perpendicular direction. Non-obj: correct direction at a non-object token.
Model
Pmt
Lyr
Δargmax
95% CI
Qwen-7B
ans
16
+.156
[+.122,+.192]
Qwen-7B
cot
16
+.120
[+.090,+.150]
LLaVA
ans
16
+.168
[+.134,+.202]
LLaVA
cot
16
+.002
[+.000,+.006]
IV2.5
ans
12
+.168
[+.134,+.204]
IV2.5
cot
12
+.042
[+.026,+.060]
Appendix
Table 9: 95% sample-cluster bootstrap CIs (2000 resamples, n=50 /quadrant) for obj_word Δ argmax under each prompt at each model’s canonical layer (RefCOCO, α=5 ). IV3-8B CoT is exactly zero with zero-width CI; LLaVA CoT lower bound touches zero.
Model
Pmt
Lyr
gain
95% CI
Qwen-7B
ans
16
+1.252
[+1.130,+1.386]
Qwen-7B
cot
16
+0.500
[+0.417,+0.589]
LLaVA
ans
16
+0.855
[+0.753,+0.964]
LLaVA
cot
16
+0.146
[+0.075,+0.217]
IV2.5
ans
12
+1.527
[+1.367,+1.679]
IV2.5
cot
12
+0.311
[+0.263,+0.362]
Appendix
Table 10: 95% bootstrap CIs for the raw shift in the target-letter logit ( n=50 /quadrant, 2000 resamples, RefCOCO, α=5 ). Target logit gain is positive even when Δ argmax is zero (IV3-8B CoT +0.421 ), showing that CoT gating suppresses the argmax flip rather than erasing the directional signal.
Figure 7: Alpha sweep at obj_word (RefCOCO, ten models). Answer-only: increasing dose-response across the tested range. CoT: near-zero for the suppressed majority, with LLaVA and Qwen-32B as borderline cases.
Model
Lyr
α=1
α=2
α=5
α=10
Answer-only
Qwen-7B
16
+.011
+.039
+.119
+.144
LLaVA
16
+.031
+.072
+.125
+.131
IV2.5
12
+.008
+.022
+.133
+.153
IV3-8B
16
+.017
+.025
+.164
+.167
Gem-4B
12
.000
+.047
+.161
+.125
Appendix
Table 11: Obj_word Δ argmax across α∈{1,2,5,10} at each model’s canonical obj_word layer (RefCOCO). Answer-only grows across the dose range; the suppressed models remain at or near zero. LLaVA and Qwen-32B are borderline near-zero cases, while Qwen-7B and InternVL2.5 retain larger CoT effects.
Table 13: Prefix_last Δ argmax under five prompt suffixes at each model’s prefix_last layer (RefCOCO, α=10 ). Qwen-7B is prompt-stable; Qwen-32B is prompt-selective (final_tag +0.431 vs. ans-only +0.019 ); IV3-14B, Gem-12B, Gem-27B are zero at every layer and every prompt. Dashes: condition not run.
Model
Pmt
Lyr
Pos
Base
Steer
Δ acc
Δparse
Qwen-7B
ans
16
obj
74.2
53.6
− 20.6
+.125
vis
16
obj
72.5
57.8
− 14.7
+.114
cot
16
obj
20.0
18.6
− 1.4
− .006
Qwen-32B
ans
32
obj
77.5
61.1
− 16.4
+.072
vis
32
obj
70.8
54.7
− 16.1
+.044
cot
32
obj
9.2
6.9
− 2.2
− .019
Appendix
Table 14: Full 128 -token greedy-generation accuracy (%, RefCOCO, n=30 /quadrant, 120 source samples and 360 source–target intervention records per cell) before steering (Base) and after steering toward an incorrect quadrant (Steer). Pos: obj = obj_word. Δ acc is the change in parsed-answer accuracy in percentage points; Δparse is the intervention-minus-noise contrast in the final parsed target rate. It is not the fixed-state next-token Δ argmax. Pmt: ans = answer-only, vis = visual_cot, cot = text CoT.
Model
Cond
L0
L4
L8
L12
L16
L20
L24
L28
Qwen-7B
o,a
0
+.083
+.137
+.095
+.112
+.002
+.002
0
o,c
+.003
+.083
+.142
+.113
+.100
+.007
−.007
0
p,a
−.002
+.012
+.055
+.088
+.267
+.322
+.073
−.020
p,c
+.005
+.060
+.163
+.137
+.213
+.185
+.030
−.017
LLaVA
o,a
+.002
+.085
+.117
+.090
+.050
−.003
0
0
o,c
0
0
+.002
+.003
+.002
0
0
0
Appendix
Table 15: GQA Spatial layer sweep (4 models). o = obj_word, p = prefix_last, a = ans-only, c = CoT. Bold: peak. IV3-8B: relocation (obj L12, pfx L20) with position-specific gating. Janus: obj peaks at L12; pfx negative.
Model
L0
L4
L8
L12
L16
L20
L24
L28
Qwen-7B
0
+.082
+.040
+.028
+.010
0
−.002
0
LLaVA
+.002
+.103
+.062
+.018
+.007
−.002
+.002
0
IV2.5
+.003
+.102
+.098
+.023
+.018
+.005
0
−.002
IV3-8B
+.002
+.107
+.073
+.033
−.002
0
0
0
Gem-4B
+.047
+.017
+.078
+.018
+.005
0
−.002
+.002
Janus
+.015
+.077
+.082
+.052
+.002
0
0
0
Appendix
Table 16: Layer-wise Δ argmax at obj_word on GQA Color (answer-only, α=5 , n=50 /quadrant). Bold: per-model peak layer. Color peaks at L4–L8, approximately four layers earlier than the spatial-attribute peaks at L8–L16.
Obj_word peak
Prefix_last peak
Model
CoT
Vis-C
Vis-D
Ans
Model
CoT
Vis-C
Vis-D
Ans
Qwen-7B
+.081
+.136
+.111
+.169
Qwen-7B
—
+.122
+.169
—
Qwen-32B
+.006
+.117
+.097
+.089
Qwen-32B
—
+.217
+.200
—
LLaVA
.000
+.103
+.117
+.158
LLaVA
—
+.108
+.097
—
IV2.5
.000
+.153
+.128
+.114
IV2.5
—
+.058
+.047
—
IV3-8B
.000
+.164
+.144
+.236
IV3-8B
—
+.144
+.156
—
Appendix
Table 17: Peak Δ argmax under four prompt types at obj_word (left) and prefix_last (right), RefCOCO, α=5 . Vis-C = visual_cot, Vis-D = visual_direct, Ans = answer_only. Both visual prompts produce positive Δ argmax in every gated model. Dashes: condition not run.
Model
steps
n /q
best step
best Δ argmax
final Δ
empty
Qwen-7B
32
20
19
+.008
+.017
.82/.80
Qwen-7B
64
10
13
+.008
− .025
.63/.48
Qwen-7B
128
20
65
+.031
+.025
.41/.35
LLaVA
32
20
3
+.013
+.042
.25/.37
LLaVA
64
10
60
+.030
+.025
.07/.10
LLaVA
128
20
109
+.200
+.104
.02/.01
Appendix
Table 18: Stepwise CoT probe. “best step”: generation index with maximal target-logit gain; “best Δ argmax”: target-option argmax-rate contrast at that step. “final Δ ”: change in parsed-answer hit rate (intervention − noise). “empty”: fraction of unparseable outputs (intervention/noise).
Model
answer-found n
L8
L12
L16
L20
L24
LLaVA-OV-7B
80
+.100
+.280
+.430
+.080
−.019
Qwen2.5-VL-7B
59
+.221
+.204
+.309
+.111
−.000
InternVL3-8B
72
+.649
+.763
+.538
+.176
−.081
Appendix
Table 19: Target-logit gain at the model’s actual generated answer step under an obj_word direction patch (RefCOCO, text CoT, α=5 , five-fold crossfit). Values use all answer-found samples; the answer step is the position immediately before the generated A/B/C/D letter. The corresponding Δ argmax at L16 is +.125 , +.011 , and −.005 for LLaVA, Qwen, and InternVL3, respectively, so the continuous gain is the primary timing-control readout.
Model
Ax
CoT Δ
Ans Δ
CoT g
Ans g
Qwen-7B
H
+.220
+.210
+1.04
+0.34
LLaVA
H
.000
+.170
+0.44
+0.40
Qwen-7B
V
+.070
+.060
+0.35
+0.17
LLaVA
V
.000
−.010
+0.16
+0.06
Appendix
Table 20: Axis-specific transport on COCO-Spatial binary MCQs ( n=50 /class, α=5 , L16, 5-fold heldout centroids). Ax: H = horizontal (left/right), V = vertical (above/below). Δ=Δ argmax; g= target-letter logit gain. Horizontal axis reproduces the RefCOCO CoT-zero / ans-positive pattern; vertical axis is systematically weaker.
Model
Pmt
Eval
Img
n /q
Contrast
gain
Qwen-7B
cot
gen
orig
10
−.033
—
LLaVA
cot
gen
orig
10
+.042
—
Qwen-7B
ans
logit
blank
20
−.042
+.064
LLaVA
ans
logit
blank
20
+.042
−.094
Appendix
Table 21: Blank-image and full-CoT sanity controls ( α=5 , L16). Eval: gen = full 128 -token generation hit rate, logit = next-token Δ argmax. Img: orig = true image, blank = uniform-grey canvas. Blank-image patching collapses the original-image effect to within noise; the full-CoT rows are intentionally underpowered ( n=10 /q) and cross-checked by Table 18 .
Run
Pmt
Lyr
Δargmax
95% CI
gain
n=30 /q layer sub-sweep
cot
16
−.003
—
+.057
cot
24
+.003
—
+.095
cot
32
+.003
—
+.108
ans
16
+.019
—
+.165
ans
24
+.036
—
+.360
Appendix
Table 22: Qwen-32B obj_word Δ argmax and target logit gain across layers (top: n=30 /q sub-sweep without CIs; bottom: n=168 rerun with 95% bootstrap CIs). Answer-only Δ argmax grows monotonically from L16 to L32 while CoT stays at zero, mirroring Qwen-7B’s late-emergence pattern at a deeper layer index.
KO Layer
Baseline Δ
Best Head
Recovery
L17
+0.072
H14
+0.032
L20
+0.078
H21
+0.028
L24
+0.080
H18
+0.006
L27
+0.080
H21
+0.062
L27 confirmation ( n=100 )
L27 (CoT)
+0.095
H21
+0.063
Appendix
Table 23: InternVL3-8B head knockout (steer L16, obj_word, RefCOCO, α=5 ). KO Layer is the layer at which a single attention head is zeroed out; Baseline Δ is the CoT Δ argmax without knockout; Recovery is the change after knockout (positive value = gate is opened); AO = answer-only. Bold values mark the only head and condition above the +0.03 significance threshold.
Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.
Vision-language models can describe an image with remarkable accuracy, yet a more fundamental question remains unanswered: what visual information actually drives their answers? In this work, we investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region. Extending the analysis to larger vision-language models reveals a similar pattern across models and corruption settings, suggesting that strong multimodal performance does not necessarily imply spatially localized causal representations. We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures. Together, our experiments expose a gap between seeing, using, and reasoning over visual structure, and provide a causal framework for studying how visual information is transformed, preserved, and ultimately used by modern vision-language models.
Naren Kumar S, Tirth Bhatt, Mayank Singh
LINGO Research Group, Indian Institute of Technology Gandhinagar, India
Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
Xiwei Liu, Yulong Li, Xinlin Zhuang +5
Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong · University Of Liverpool