When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Figures & tables
Figure 2: Left: concrete banana example with the max-variant rule: at each layer, we take the best-matching blue and yellow token forms, then define MAC as the first stable layer where visual ( v ) exceeds prior ( p ), verified by persistence at the next layer. Right: MAC logit trajectories (color, 493 samples). Each subplot shows per-layer visual (blue) and prior (orange) logits with sample traces and mean ± 1 std shading. Green dashed line: MAC crossover. “V wins” indicates the percentage of samples where the visual logit exceeds the prior logit at the final layer. Top: three primary models (7–8B). Bottom: scaled-up variants. See Appendix Section C.1 for all ten models.
Model
L
Color
Size
Latent (Color)
MAC
R%
D%
MAC
R%
D%
25%
50%
75%
Cos@75
Primary models (7–8B)
DeepSeek-Janus
30
L10.8
91%
36%
L14.6
70%
49%
14.29
63.26
178.60
0.961
LLaVA-Next
32
L12.7
82%
40%
L15.0
64%
47%
0.24
1.73
3.80
0.977
LLaVA-v1.5
32
L13.5
91%
42%
L16.9
75%
53%
1.07
9.60
31.67
0.935
LLaVA-OneVision
28
L16.8
96%
60%
L17.1
92%
61%
3.80
20.81
53.04
0.899
Table 1: MAC crossover and latent encoding (Color: 493 samples; Size: 727). Color / Size : MAC = mean crossover layer; R% = % samples where visual logit wins at final layer; D% = MAC depth as % of total layers L . Latent (Color) : L2 distance at 25%, 50%, and 75% of mean MAC depth (higher = stronger encoding); Cos@75 = cosine similarity at 75% depth. Per-model rankings partially reverse between Color and Size; scaled variants show 2 – 5× stronger L2 at 75%.
Model
VISUAL
PRIOR
Full-seq. patch
Token-type
L2
n
L2
n
Ratio
p (MW)
Chg
Flip
Cond.
Full
Img
Txt
Ret.
Primary models (7–8B)
DeepSeek-Janus
178.84
434
176.85
59
1.01 ×
0.946
99
82
94
82
83
0
101%
LLaVA-Next
3.82
357
3.75
136
1.02 ×
0.559
96
60
82
60
58
0
97%
LLaVA-v1.5
32.02
460
26.75
33
1.20 ×
0.002
99
78
94
78
77
1
99%
LLaVA-OneVision
53.04
471
53.20
22
1.00 ×
0.916
98
68
88
68
67
0
99%
Table 2: Left: encoding at 75% MAC depth (Color, 493 samples per model): L2 between counterfactual vs. standard hidden states for samples where the model answered with the visual vs. prior token; Ratio = VISUAL/PRIOR mean L2; p (MW) = Mann–Whitney p . Ratio ≈1 favors arbitration failure over perceptual blindness. Right: activation patching at the MAC probe layer (100 Color samples per model): Chg / Flip / Cond. = % any change, % visual → prior flips, % flips conditioned on baseline-visual; Full / Img / Txt = flip counts; Ret. = Img / Full . BLIP-2: patching not run (—). Last-token patching 0–1% flip; see Appendix E .
Linear Steering
SAE Steering
Model
Base.
Layer
Δ Acc.
Acc.
Imp
Deg
Layer
Δ Acc.
Acc.
Imp
Deg
LLaVA-v1.5
86.0
L12
+1.4
87.4
6
2
L4
+2.0
88.1
6
0
InternVL2
83.6
L3
+3.4
87.0
16
6
L3
+3.8
87.4
14
3
Qwen2-VL
88.4
L6
+3.4
91.8
11
1
L3
+2.0
90.4
8
2
Table 3: Steering results at each model’s best layer (293 eval samples, Color attribute). Base. and Acc. : visual-grounding accuracy (%) before and after steering. Δ Acc. : net accuracy change (pp). Imp / Deg : number of samples improved/degraded.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Probe AUC
P(visual) @ 10%
Model
∼ 10%
∼ 50%
∼ 97%
Succ
Fail
Δ
InternVL2
1.000
1.000
1.000
.958
.961
− .003
DeepSeek-Janus
1.000
1.000
.998
.990
.991
− .001
LLaVA-OneVision
.995
1.000
.998
.892
.915
− .022
Qwen2-VL
.995
.999
.999
.905
.909
− .004
BLIP-2
.976
.990
.992
.812
.797
+ .015
Appendix
Table 4: Linear probe AUC (5-fold CV, 493 samples) for decoding visual attributes from hidden states at three depths. Right columns: mean probe confidence for success vs. failure samples at ∼ 10% depth.
Vision-language models must reconcile visual evidence with memorized world knowledge when the two conflict. How they resolve this conflict shapes the reliability of multimodal systems, yet prior work characterizes it behaviorally without a component-level causal account. We combine activation patching across three granularities (residual stream, attention heads, and MLP sublayers) with model-component ablation studies and mechanistic analysis. Across three VLM families, we find that visual grounding emerges by default, whereas prior grounding depends on a small set of causally necessary attention heads (2.5-4.8%) concentrated in the second half of the network. These heads enable answers from stored world knowledge (e.g., "red" for a strawberry) despite conflicting visual input. Ablating them flips predictions from knowledge-grounded to visually grounded answers in 68-96% of cases under prior-knowledge prompts, but changes only 0.8-7.5% of visually grounded predictions, establishing an asymmetric causal structure. The identified heads decompose into routing heads, which modulate information flow, and writing heads, which directly project answer tokens into the residual stream. This structure is consistent across model families and scales, revealing a sparse causal circuit underlying perception-knowledge conflict in VLMs.
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.