cs.CLAug 24, 2026
SaveKnowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?
Organizations: Tsinghua University Beijing, China
Abstract
Vision-language models are known to encode spatial information in their hidden states, yet often fail to use it when answering. However, it remains unclear when and where this encoded information reaches the answer. We address this with direction patching, a class-conditioned causal intervention applied across layers, token positions, and prompt formats. Using spatial-ID directions constructed following prior encoding evidence, we find that causal influence on answer logits emerges only at mid-to-deep depths. Text chain-of-thought suppresses immediate object-word argmax-level transport in most models, while visually grounded prompts keep it open. Positive target-logit gain can remain below the argmax threshold, and transport can re-emerge at the final prefix token or at the answer step in deeper layers. Across the ten VLMs we study, these local effects form descriptive transport patterns. Complementary experiments characterize how these patterns shift across datasets, attributes, and encoding amplitudes. Together, these results reframe the encoding-grounding gap as a problem of conditional transport in VLMs.
Figures & tables
Figure 1: Direction patching reveals conditional transport. Spatial encoding is broadly present across VLMs, but whether it reaches the answer depends on prompt type, token position, and layer depth. Text CoT can block object-word transport, whereas visual grounding prompts keep this pathway open.
| Model | Family | Params | Layers |
| Qwen2.5-VL-7B | Qwen | 7B | 28 |
| Qwen2.5-VL-32B | Qwen | 32B | 64 |
| LLaVA-OneVision-7B | LLaVA | 7B | 28 |
| InternVL2.5-8B | InternVL | 8B | 32 |
| InternVL3-8B | InternVL | 8B | 28 |
| InternVL3-14B | InternVL | 14B | 48 |
Table 1: Ten evaluated VLMs spanning five families, including three within-family scale pairs: Qwen 7B/32B, InternVL3 8B/14B, and Gemma-3 4B/12B/27B. Row tints group rows by family.
Figure 2: Layer-wise argmax for ten models (obj_word, answer_only, ). Argmax-level transport activates only at mid-to-deep layers, although the Kang-style spatial-ID direction is constructed from early-layer representation evidence.
| Model | CoT | Vis-CoT | Ans-only |
| Qwen-7B | +.081 | +.136 | +.169 |
| Qwen-32B | +.006 | +.117 ∗ | +.089 |
| LLaVA | .000 | +.103 | +.158 |
| InternVL2.5 | .000 | +.153 ∗ | +.114 |
| InternVL3-8B | .000 | +.164 | +.236 |
| InternVL3-14B | .000 | +.117 ∗ | +.097 |
Table 2: argmax at obj_word at each model’s visual_cot peak layer (RefCOCO, ). ∗ : visual_cot exceeds answer_only. All ten models recover positive transport under visual_cot, including eight suppressed under text CoT.
Figure 3: Position-dependent transport (GQA Spatial, three models). Left: InternVL3-8B shows the relocation pattern (obj_word L12 prefix_last L20; CoT gated only at obj_word). Center: Janus concentrates at obj_word. Right: Qwen-7B transports at both positions under both prompts.
Figure 4: Per-model CoT gating signature (obj_word, RefCOCO, ). Eight of ten models fall below the descriptive threshold: six are exact-zero and two are near-zero/borderline; Qwen-7B retains substantial CoT transport. Numerical values in App. B .
| Pattern | Models | Obj. route | Reloc. |
| Open | Qwen-7B | Open | Yes |
| Reduced supp.,reloc. | LLaVA , IV2.5 , IV3-8B , IV3-14B | Supp./red. | Yes |
| Suppressed, non-reloc. | Gemma-3 (4/12/27B), Janus | Supp. | No |
| Prompt- selective | Qwen-32B | Near-zero | Sel. |
Table 3: Four descriptive transport patterns across ten models. “Reloc.”: whether transport reappears at prefix_last (Yes / Sel. = prompt-selective / No). Colors match Figure 4 .
Figure 5: Attention head knockout at L27 of InternVL3-8B ( , RefCOCO, ). (a) Under CoT, knocking out H21 recovers argmax, the only head above the significance threshold among all 32 heads. (b) Under answer-only, H21 has no effect ( ). H21 is a CoT-specific gating candidate in this scan, rather than a general transport component.
| Prediction | Match | Note |
| CoT suppression at obj_word | 9/10 | Qwen-7B : remains open |
| Peak layer 8L | 6/6 | original 6 models |
| pfx_last deeper than obj_word | 8/10 | Janus , Gem-27B anomalous |
Table 4: Structural transfer from RefCOCO to GQA Spatial. Three transport-map predictions tested with independent images, scene graphs, and object distributions.
| Model | Spatial peak | Color peak | Shift |
| Qwen-7B | L8 | L4 | |
| Qwen-32B | L24 | L16 | |
| LLaVA | L8 | L4 | |
| InternVL2.5 | L8 | L4 | |
| InternVL3-8B | L12 | L4 | |
| InternVL3-14B | L12 | L12 |
Table 5: Obj_word peak layer on GQA Spatial vs. GQA Color (answer_only), all ten models. Color peaks – layers earlier than spatial in 9/10 models; InternVL3-14B is the lone exception (same peak layer).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Format | Instruction suffix |
| cot | “Let’s think step by step.” |
| brief_reason | “Give a brief reason, then answer with the letter.” |
| answer_first | “First, answer with only the letter (A–D). Then explain briefly.” |
| final_tag | “Think about it, then give your final answer as Final Answer: X” |
| answer_only | “Answer with only the letter (A–D).” |
| visual_cot | “First, carefully look at the image and describe where you see the object mentioned above. Then answer with only the letter.” |
Table 6: Verbatim instruction suffixes appended to a shared question stem. The first five suffixes ( cot through answer_only ) span a reasoning-length spectrum and are used in the prompt ladder of § 4.1 ; the last two ( visual_cot , visual_direct ) are visual-grounding prompts evaluated in § 3.3 .
| Model | CoT | Ans-only | Gated? |
| Qwen-7B | +0.120 L16 | +0.156 L16 | No |
| InternVL2.5 | +0.042 L12 | +0.168 L12 | Weak |
| LLaVA | +0.006 L10 | +0.168 L16 | Yes |
| InternVL3-8B | 0.000 all | +0.236 L16 | Yes |
| InternVL3-14B | 0.000 all | +0.097 L24 | Yes |
| Gemma-3-4B | 0.000 all | +0.178 L12 | Yes |
Table 7: Peak obj_word argmax under CoT and answer-only (RefCOCO, ), with the layer at which each peak is attained. The sample count is /quadrant for 7B/8B models and /quadrant for Qwen-32B, Gemma-3, and Janus; each source sample is paired with the three non-source target quadrants. Top two rows (above the midrule) are open or partially suppressed; the bottom eight rows fall at or below .
Figure 6: Wrong-axis and non-object patching produce near-zero argmax, confirming direction and position specificity.
| Model | Lyr | Target | Wrong | Non-obj |
| Qwen-7B | 16 | +.156 | +.026 | |
| LLaVA | 16 | +.168 | +.002 | |
| IV2.5 | 12 | +.168 | +.018 | |
| IV3-8B | 16 | +.236 | +.028 | |
| IV3-14B | 24 | +.097 | +.006 | +.003 |
| Gem-4B | 12 | +.178 | +.006 |
Table 8: Direction and position controls (RefCOCO, ans-only, ). Wrong: perpendicular direction. Non-obj: correct direction at a non-object token.
| Model | Pmt | Lyr | 95% CI | |
| Qwen-7B | ans | 16 | +.156 | |
| Qwen-7B | cot | 16 | +.120 | |
| LLaVA | ans | 16 | +.168 | |
| LLaVA | cot | 16 | +.002 | |
| IV2.5 | ans | 12 | +.168 | |
| IV2.5 | cot | 12 | +.042 |
Table 9: 95% sample-cluster bootstrap CIs (2000 resamples, /quadrant) for obj_word argmax under each prompt at each model’s canonical layer (RefCOCO, ). IV3-8B CoT is exactly zero with zero-width CI; LLaVA CoT lower bound touches zero.
| Model | Pmt | Lyr | gain | 95% CI |
| Qwen-7B | ans | 16 | +1.252 | |
| Qwen-7B | cot | 16 | +0.500 | |
| LLaVA | ans | 16 | +0.855 | |
| LLaVA | cot | 16 | +0.146 | |
| IV2.5 | ans | 12 | +1.527 | |
| IV2.5 | cot | 12 | +0.311 |
Table 10: 95% bootstrap CIs for the raw shift in the target-letter logit ( /quadrant, 2000 resamples, RefCOCO, ). Target logit gain is positive even when argmax is zero (IV3-8B CoT ), showing that CoT gating suppresses the argmax flip rather than erasing the directional signal.
Figure 7: Alpha sweep at obj_word (RefCOCO, ten models). Answer-only: increasing dose-response across the tested range. CoT: near-zero for the suppressed majority, with LLaVA and Qwen-32B as borderline cases.
| Model | Lyr | ||||
| Answer-only | |||||
| Qwen-7B | 16 | +.011 | +.039 | +.119 | +.144 |
| LLaVA | 16 | +.031 | +.072 | +.125 | +.131 |
| IV2.5 | 12 | +.008 | +.022 | +.133 | +.153 |
| IV3-8B | 16 | +.017 | +.025 | +.164 | +.167 |
| Gem-4B | 12 | .000 | +.047 | +.161 | +.125 |
Table 11: Obj_word argmax across at each model’s canonical obj_word layer (RefCOCO). Answer-only grows across the dose range; the suppressed models remain at or near zero. LLaVA and Qwen-32B are borderline near-zero cases, while Qwen-7B and InternVL2.5 retain larger CoT effects.
| Model | Lyr | ||||
| Answer-only | |||||
| Qwen-7B | 20 | +.014 | +.047 | +.150 | +.275 |
| LLaVA | 20 | +.014 | +.031 | +.075 | +.214 |
| IV2.5 | 24 | +.003 | +.008 | +.017 | +.039 |
| IV3-8B | 20 | +.008 | +.025 | +.097 | +.461 |
| IV3-14B | 36 | +.028 | +.044 | +.086 | +.344 |
Table 12: Prefix_last alpha sweep (RefCOCO). Qwen-7B CoT non-monotonic (peak : ). IV3-14B: hard-zero at prefix_last.
Figure 8: Prompt ladder at prefix_last (RefCOCO, ). Qwen-7B prompt-stable; Qwen-32B prompt-selective.
| Model | Lyr | CoT | Brief | Ans-1st | Final | Ans |
| Qwen-7B | 20 | +.351 | +.148 | +.277 | +.395 | — |
| Qwen-7B | 28 | +.840 | +.808 | +.800 | +.796 | +.790 |
| Qwen-32B | 46 | +.003 | +.044 | +.042 | +.122 | +.025 |
| Qwen-32B | 55 | +.181 | +.133 | +.050 | +.431 | +.019 |
| LLaVA | 24 | +.014 | +.182 | +.401 | +.415 | — |
| LLaVA | 28 | +.315 | +.451 | +.451 | +.401 | — |
Table 13: Prefix_last argmax under five prompt suffixes at each model’s prefix_last layer (RefCOCO, ). Qwen-7B is prompt-stable; Qwen-32B is prompt-selective (final_tag vs. ans-only ); IV3-14B, Gem-12B, Gem-27B are zero at every layer and every prompt. Dashes: condition not run.
| Model | Pmt | Lyr | Pos | Base | Steer | acc | |
| Qwen-7B | ans | 16 | obj | 74.2 | 53.6 | 20.6 | +.125 |
| vis | 16 | obj | 72.5 | 57.8 | 14.7 | +.114 | |
| cot | 16 | obj | 20.0 | 18.6 | 1.4 | .006 | |
| Qwen-32B | ans | 32 | obj | 77.5 | 61.1 | 16.4 | +.072 |
| vis | 32 | obj | 70.8 | 54.7 | 16.1 | +.044 | |
| cot | 32 | obj | 9.2 | 6.9 | 2.2 | .019 |
Table 14: Full -token greedy-generation accuracy (%, RefCOCO, /quadrant, source samples and source–target intervention records per cell) before steering (Base) and after steering toward an incorrect quadrant (Steer). Pos: obj obj_word. acc is the change in parsed-answer accuracy in percentage points; is the intervention-minus-noise contrast in the final parsed target rate. It is not the fixed-state next-token argmax. Pmt: ans answer-only, vis visual_cot, cot text CoT.
| Model | Cond | L0 | L4 | L8 | L12 | L16 | L20 | L24 | L28 |
| Qwen-7B | o,a | 0 | +.083 | +.137 | +.095 | +.112 | +.002 | +.002 | 0 |
| o,c | +.003 | +.083 | +.142 | +.113 | +.100 | +.007 | 0 | ||
| p,a | +.012 | +.055 | +.088 | +.267 | +.322 | +.073 | |||
| p,c | +.005 | +.060 | +.163 | +.137 | +.213 | +.185 | +.030 | ||
| LLaVA | o,a | +.002 | +.085 | +.117 | +.090 | +.050 | 0 | 0 | |
| o,c | 0 | 0 | +.002 | +.003 | +.002 | 0 | 0 | 0 |
Table 15: GQA Spatial layer sweep (4 models). o obj_word, p prefix_last, a ans-only, c CoT. Bold: peak. IV3-8B: relocation (obj L12, pfx L20) with position-specific gating. Janus: obj peaks at L12; pfx negative.
| Model | L0 | L4 | L8 | L12 | L16 | L20 | L24 | L28 |
| Qwen-7B | 0 | +.082 | +.040 | +.028 | +.010 | 0 | 0 | |
| LLaVA | +.002 | +.103 | +.062 | +.018 | +.007 | +.002 | 0 | |
| IV2.5 | +.003 | +.102 | +.098 | +.023 | +.018 | +.005 | 0 | |
| IV3-8B | +.002 | +.107 | +.073 | +.033 | 0 | 0 | 0 | |
| Gem-4B | +.047 | +.017 | +.078 | +.018 | +.005 | 0 | +.002 | |
| Janus | +.015 | +.077 | +.082 | +.052 | +.002 | 0 | 0 | 0 |
Table 16: Layer-wise argmax at obj_word on GQA Color (answer-only, , /quadrant). Bold: per-model peak layer. Color peaks at L4–L8, approximately four layers earlier than the spatial-attribute peaks at L8–L16.
| Obj_word peak | Prefix_last peak | ||||||||
| Model | CoT | Vis-C | Vis-D | Ans | Model | CoT | Vis-C | Vis-D | Ans |
| Qwen-7B | +.081 | +.136 | +.111 | +.169 | Qwen-7B | — | +.122 | +.169 | — |
| Qwen-32B | +.006 | +.117 | +.097 | +.089 | Qwen-32B | — | +.217 | +.200 | — |
| LLaVA | .000 | +.103 | +.117 | +.158 | LLaVA | — | +.108 | +.097 | — |
| IV2.5 | .000 | +.153 | +.128 | +.114 | IV2.5 | — | +.058 | +.047 | — |
| IV3-8B | .000 | +.164 | +.144 | +.236 | IV3-8B | — | +.144 | +.156 | — |
Table 17: Peak argmax under four prompt types at obj_word (left) and prefix_last (right), RefCOCO, . Vis-C visual_cot, Vis-D visual_direct, Ans answer_only. Both visual prompts produce positive argmax in every gated model. Dashes: condition not run.
| Model | steps | /q | best step | best argmax | final | empty |
| Qwen-7B | 32 | 20 | 19 | +.008 | +.017 | .82/.80 |
| Qwen-7B | 64 | 10 | 13 | +.008 | .025 | .63/.48 |
| Qwen-7B | 128 | 20 | 65 | +.031 | +.025 | .41/.35 |
| LLaVA | 32 | 20 | 3 | +.013 | +.042 | .25/.37 |
| LLaVA | 64 | 10 | 60 | +.030 | +.025 | .07/.10 |
| LLaVA | 128 | 20 | 109 | +.200 | +.104 | .02/.01 |
Table 18: Stepwise CoT probe. “best step”: generation index with maximal target-logit gain; “best argmax”: target-option argmax-rate contrast at that step. “final ”: change in parsed-answer hit rate (intervention noise). “empty”: fraction of unparseable outputs (intervention/noise).
| Model | answer-found | L8 | L12 | L16 | L20 | L24 |
| LLaVA-OV-7B | 80 | +.100 | +.280 | +.430 | +.080 | |
| Qwen2.5-VL-7B | 59 | +.221 | +.204 | +.309 | +.111 | |
| InternVL3-8B | 72 | +.649 | +.763 | +.538 | +.176 |
Table 19: Target-logit gain at the model’s actual generated answer step under an obj_word direction patch (RefCOCO, text CoT, , five-fold crossfit). Values use all answer-found samples; the answer step is the position immediately before the generated A/B/C/D letter. The corresponding argmax at L16 is , , and for LLaVA, Qwen, and InternVL3, respectively, so the continuous gain is the primary timing-control readout.
| Model | Ax | CoT Δ | Ans Δ | CoT g | Ans g |
| Qwen-7B | H | +.220 | +.210 | +1.04 | +0.34 |
| LLaVA | H | .000 | +.170 | +0.44 | +0.40 |
| Qwen-7B | V | +.070 | +.060 | +0.35 | +0.17 |
| LLaVA | V | .000 | +0.16 | +0.06 |
Table 20: Axis-specific transport on COCO-Spatial binary MCQs ( /class, , L16, 5-fold heldout centroids). Ax: H horizontal (left/right), V vertical (above/below). argmax; target-letter logit gain. Horizontal axis reproduces the RefCOCO CoT-zero / ans-positive pattern; vertical axis is systematically weaker.
| Model | Pmt | Eval | Img | /q | Contrast | gain |
| Qwen-7B | cot | gen | orig | 10 | — | |
| LLaVA | cot | gen | orig | 10 | +.042 | — |
| Qwen-7B | ans | logit | blank | 20 | +.064 | |
| LLaVA | ans | logit | blank | 20 | +.042 |
Table 21: Blank-image and full-CoT sanity controls ( , L16). Eval: gen full -token generation hit rate, logit next-token argmax. Img: orig true image, blank uniform-grey canvas. Blank-image patching collapses the original-image effect to within noise; the full-CoT rows are intentionally underpowered ( /q) and cross-checked by Table 18 .
| Run | Pmt | Lyr | 95% CI | gain | |
| /q layer sub-sweep | |||||
| cot | 16 | — | +.057 | ||
| cot | 24 | +.003 | — | +.095 | |
| cot | 32 | +.003 | — | +.108 | |
| ans | 16 | +.019 | — | +.165 | |
| ans | 24 | +.036 | — | +.360 | |
Table 22: Qwen-32B obj_word argmax and target logit gain across layers (top: /q sub-sweep without CIs; bottom: rerun with 95% bootstrap CIs). Answer-only argmax grows monotonically from L16 to L32 while CoT stays at zero, mirroring Qwen-7B’s late-emergence pattern at a deeper layer index.
| KO Layer | Baseline | Best Head | Recovery |
| L17 | +0.072 | H14 | +0.032 |
| L20 | +0.078 | H21 | +0.028 |
| L24 | +0.080 | H18 | +0.006 |
| L27 | +0.080 | H21 | +0.062 |
| L27 confirmation ( ) | |||
| L27 (CoT) | +0.095 | H21 | +0.063 |
Table 23: InternVL3-8B head knockout (steer L16, obj_word, RefCOCO, ). KO Layer is the layer at which a single attention head is zeroed out; Baseline is the CoT argmax without knockout; Recovery is the change after knockout (positive value gate is opened); AO answer-only. Bold values mark the only head and condition above the significance threshold.
More in this topic
VLM Interpretability
Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.
See all 268 papers