When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.
Figures & tables
Figure 1: Tri-Layer Hallucination Diagnostic Framework. Each (question, image) pair is queried under full / blind / Gaussian-noise conditions with a set of refusal anchors. Three per-sample metrics are extracted—LAD (perception; refusal-prob. gap blind vs. full), VNS (dependency; KL between full and blind output distributions), CS (alignment; answer-vs-refusal log-ratio)—and cascaded into four failure categories (P.B., L.SC, V.S., R.R.; rule table: Table 1 ).
Figure 2: Layer-wise blind-vs-full linear separability at the last text-token position, 9 models overlaid against relative depth (balanced 5-fold CV, N=1,000 /condition). Eight grey lines saturate at ≥0.996 from layer 1. Llama-3.2-11B-Vision (red) is the sole outlier (layer-1 acc. 0.19, reaching 1.00 by layer 4)—consistent with its cross-attention architecture. Yet the same samples emit hallucinations at SC blind rates of 1.1% – 99.4% (Table 2 )—a decoding-time override of fully-preserved internal evidence.
Category
LAD (Perc.)
VNS (Dep.)
CS (Align.)
Diagnostic Interpretation
Perceptual Blindness
≤τLAD
–
–
Encoder failure: model cannot distinguish blind from natural input; downstream metrics undefined.
Language Shortcut
>τLAD
≤τVNS
–
Visual neglect: Model detects anomaly but ignores visual signal.
Visual Sycophancy
>τLAD
>τVNS
>τCS
Split beliefs: Perceives anomaly but hallucinates to satisfy instruction.
Robust Refusal
>τLAD
>τVNS
≤τCS
Ideal behavior: Visual grounding overrides language priors.
Table 1: Diagnostic Taxonomy based on the Tri-Layer Framework. “–” indicates inapplicable metrics. Threshold selection is detailed in Section 3 .
Model
Acc.
SC blind
SC noise
SC conf
Qwen3-VL-32B
75.8
1.1
0.3
14.3
Qwen3-VL-8B
75.1
4.7
2.6
12.2
Qwen2.5-VL-72B
73.5
40.4
9.4
9.7
Qwen2.5-VL-7B
72.1
45.8
67.9
15.1
Molmo2-4B
71.3
99.4
79.1
37.1
Llama-3.2-11B
69.9
45.8
57.8
37.8
Table 2: Response-level results (%), N=1000 per model. Acc. (higher is better) under full condition; SC x = shortcut rate under condition x (lower is more grounded).
Model
P.B. ↓
L.SC ↓
V.S. ↓
Molmo2-4B
42.8
25.2
32.0
Phi-3.5-Vision
1.6
39.8
58.6
LLaVA-NeXT-7B
0.0
29.7
70.3
Qwen2.5-VL-7B
0.1
27.5
72.4
Llama-3.2-11B
0.0
21.2
78.8
Pixtral-12B
5.3
14.8
79.9
Table 3: Taxonomy distribution (%, blind condition). Robust Refusal is 0% (omitted). Per-category accuracies (P.B./L.SC/V.S./Overall: 65.9/69.2/70.9/70.7%) confirm P.B. is the only category warranting abstention.
Category
−1.0
−0.5
0.0∗
0.5
1.0
P.B.
7.1
7.1
7.1
7.1
7.1
L.SC
23.3
23.3
23.3
23.3
23.3
V.S.
69.6
69.6
69.6
69.3
59.2
R.R.
0.0
0.0
0.0
0.4
10.5
Table 4: Taxonomy shares (%) under τCS sweep ( τLAD=1.5 , τVNS=1.0 fixed; ∗ =default). R.R. stays at 0% across two orders of magnitude of the natural probabilistic boundary. Full τLAD / τVNS sweeps: Appendix B .
Model
VNS ↑
LAD ↑
CS ↓
Qwen3-VL-32B
2.81 ± 1.02
5.32 ± 1.06
1.99 ± .56
Qwen3-VL-8B
2.70 ± 1.09
4.97 ± .77
2.18 ± .53
Qwen2.5-VL-72B
2.59 ± 1.09
4.17 ± .82
1.91 ± .39
Pixtral-12B
2.06 ± 1.16
3.12 ± .96
3.04 ± .75
Llama-3.2-11B
1.97 ± 1.08
3.24 ± .45
0.87 ± .33
Qwen2.5-VL-7B
1.75 ± .97
3.85 ± .77
1.42 ± .50
Table 5: Per-model Tri-Layer metrics (mean ± std, blind condition). Noise-variant columns are reported in Appendix C .
Model
VQAv2
HB (abs τ )
HB (pct τ )
Qwen2.5-VL-7B
72.4
73.5
72.0
Qwen3-VL-8B
96.6
99.0
96.0
Pixtral-12B
79.9
66.5
79.0
Table 6: Cross-dataset V.S. share (%) on VQAv2 vs HallusionBench under absolute and percentile-anchored thresholds. Pixtral is the diagnostic case; the other two are intrinsically stable.
Task
N
Acc. ↑
SC blind ↓
SC conf ↓
P.B. ↓
L.SC ↓
V.S. ↓
Halluc.
3150
80.1
43.8
24.1
6.1
11.4
82.5
Spatial
2250
69.4
48.7
46.9
5.1
25.4
69.6
Complex
2250
68.9
47.4
52.6
4.4
24.3
71.3
Count.
1350
63.0
46.4
18.5
7.0
14.6
78.4
Table 7: Task-wise results (%), N pooled across 9 models. R.R. is 0% across all tasks (omitted). Abbreviations follow Table 2 .
Model
POPE Acc
POPE F1
Qwen3-VL-8B
88.5
88.1
LLaVA-NeXT-7B
88.0
87.8
Qwen3-VL-32B
87.0
86.5
Qwen2.5-VL-72B
86.8
85.1
Llama-3.2-11B
86.3
87.2
Qwen2.5-VL-7B
86.3
84.4
Table 8: POPE overall results (%, average over Random/Popular/Adversarial subsets, 600 queries/model). All models in 79.5–88.5% range. Per-subset breakdown in Appendix E .
Model
Acc.
VNS
LAD
L.SC
V.S.
Qwen2.5-VL-7B
72.1
1.75
3.85
27.5
72.4
Qwen2.5-VL-72B
73.5
2.59
4.17
4.7
95.3
Qwen3-VL-8B
75.1
2.70
4.97
3.4
96.6
Qwen3-VL-32B
75.8
2.81
5.32
0.9
98.9
Table 9: Scaling within Qwen-VL (%). The three axes (within-gen 7B → 72B, gen upgrade 7B → 8B, within-gen 8B → 32B) reduce L.SC and amplify V.S. monotonically; LAD climbs in parallel. SC blind /CS in Table 2 / Appendix C .
Model
Base
Rand
LAD
VNS
Ours
Δ↑
Qwen3-VL-32B
75.8
75.2
79.0
84.6
82.6
+6.8
Qwen3-VL-8B
75.1
74.8
77.4
84.8
84.0
+8.9
Qwen2.5-VL-72B
73.5
73.2
73.4
79.2
77.8
+4.3
Qwen2.5-VL-7B
72.1
72.6
80.8
78.8
81.6
+9.5
Molmo2-4B
71.3
72.2
72.6
74.0
76.6
+5.3
Llama-3.2-11B
69.9
68.2
70.0
69.0
69.8
≈ 0.0
Table 10: Selective prediction at 50% coverage. Base = full-coverage accuracy; Rand = random 50% subset (control, seed 42); LAD / VNS = single-metric ranking; Ours = category-conditional routing. Δ = Ours − Base. Underlined = best per row.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Decision Point
Model
Resp. Len
Blind
Conflict
Noise
Molmo2-4B
47.1
17.4
14.0
15.3
Phi-3.5-Vision
15.6
4.4
4.6
4.2
LLaVA-NeXT-7B
38.9
11.7
12.1
11.7
Qwen2.5-VL-7B
52.8
15.2
9.6
16.5
Llama-3.2-11B
22.0
8.5
8.3
8.8
Appendix
Table 11: Per-token VNS statistics. Resp. Len = mean response length (tokens); Decision Point = mean token index with maximum KL divergence between full-image and blind/conflict/noise conditions. Visual influence consistently peaks in the first half of generation across all conditions, validating our top- k aggregation strategy.
Category
0.5
1.0
1.5∗
2.0
2.5
P.B.
0.2
1.0
7.1
16.0
26.6
L.SC
26.3
26.0
23.3
19.5
15.6
V.S.
73.5
73.1
69.6
64.5
57.9
Robust
0.0
0.0
0.0
0.0
0.0
Appendix
Table 12: Taxonomy proportions (%) under τLAD sweep ( τVNS=1.0 , τCS=0 fixed; ∗ = default). V.S. remains the dominant category across all settings.
Category
0.5
1.0∗
1.25
1.5
2.0
P.B.
7.1
7.1
7.1
7.1
7.1
L.SC
9.0
23.3
31.1
39.0
54.0
V.S.
83.9
69.6
61.8
53.9
38.8
Robust
0.0
0.0
0.0
0.0
0.0
Appendix
Table 13: Taxonomy proportions (%) under τVNS sweep ( τLAD=1.5 , τCS=0 fixed; ∗ = default). V.S. remains the dominant category across all but the most extreme setting; maximum deviation from default: 69.6% − 38.8% = 30.8 pp.
Model
VNS noise↑
LAD noise↑
CS noise↓
Qwen3-VL-32B
3.28 ± 1.15
2.78 ± .88
4.10 ± .96
Qwen3-VL-8B
2.77 ± 1.16
3.04 ± .61
3.22 ± .57
Qwen2.5-VL-72B
2.08 ± .88
3.39 ± .69
2.17 ± .24
Pixtral-12B
2.22 ± 1.27
2.22 ± .51
3.12 ± .55
Llama-3.2-11B
2.13 ± 1.15
2.39 ± .38
1.39 ± .28
Qwen2.5-VL-7B
1.62 ± .86
3.25 ± .78
2.21 ± .27
Appendix
Table 14: Tri-Layer metrics under the Gaussian noise condition (mean ± std). VNS noise correlates with VNS (Table 5 ) at r=0.866 per-sample. LAD noise trails LAD (mean 2.38 vs. 3.09): black images are recognized as more anomalous than noise (which provides texture the encoder partially processes).
Random
Popular
Adversarial
Overall
Model
Acc
F1
Yes
Acc
F1
Yes
Acc
F1
Yes
Acc
F1
Qwen3-VL-32B
91.5
90.7
41.5
88.0
87.4
45.0
81.5
81.8
51.5
87.0
86.5
Qwen3-VL-8B
92.0
91.4
43.0
88.5
88.1
46.5
85.0
85.0
50.0
88.5
88.1
Qwen2.5-VL-72B
87.5
85.7
37.5
86.5
84.8
38.5
86.5
84.8
38.5
86.8
85.1
Qwen2.5-VL-7B
87.0
85.1
37.0
86.0
84.1
38.0
86.0
84.1
38.0
86.3
84.4
Molmo2-4B
90.0
90.5
55.0
83.5
85.2
61.5
77.5
80.8
67.5
83.7
85.3
Appendix
Table 15: POPE benchmark results (200 samples per subset × 3 subsets = 600 queries per model). Acc = accuracy (%), F1 = positive-class F1 (%), Yes = %-yes rate. Standard decoding only. Best per column in bold .
Absolute τ (1.5, 1.0, 0)
Percentile-anchored τ
Model
Dataset
N
P.B.
L.SC
V.S.
R.R.
P.B.
L.SC
V.S.
R.R.
Qwen2.5-VL-7B
VQAv2
1000
0.1
27.5
72.4
0.0
0.1
27.5
72.4
0.0
HallusionBench
200
0.0
26.5
73.5
0.0
0.5
27.5
72.0
0.0
Qwen3-VL-8B
VQAv2
1000
0.0
3.4
96.6
0.0
0.1
3.4
96.5
0.0
HallusionBench
200
0.0
1.0
99.0
0.0
0.5
3.5
96.0
0.0
Pixtral-12B
VQAv2
1000
5.3
14.8
79.9
0.0
5.3
14.8
79.9
0.0
Appendix
Table 16: Cross-dataset taxonomy stability on VQAv2 (1,000 samples; our main eval) vs HallusionBench (200 yes/no samples). Absolute τ : paper-default (1.5,1.0,0) applied directly. Percentile-anchored τ : τLAD and τVNS are set to the per-(model, dataset) local LAD/VNS values at the percentile ranks that the absolute thresholds occupy in VQAv2; τCS=0 is retained. Pixtral-12B is the diagnostic case: under absolute τ , the V.S. share drifts 79.9 % → 66.5 % across datasets; percentile-anchoring restores it to 79.0 %, within 1pp of the VQAv2 reference. Qwen2.5-VL-7B and Qwen3-VL-8B are intrinsically stable under both regimes.
Figure 3: Per-layer balanced accuracy of a logistic probe trained on full-condition hidden states with randomly assigned binary labels (5-fold CV; layers normalised to relative depth so models with different L overlay). All three models track the chance line ( 0.50 ) across the entire stack, in sharp contrast to the ≥0.99 peak balanced accuracy of the blind-vs-full probe on the same activations.
Model
L
mean
max
min
Qwen2.5-VL-7B
28
0.486
0.516
0.451
Qwen3-VL-8B
36
0.485
0.530
0.449
Pixtral-12B
40
0.478
0.513
0.450
Appendix
Table 17: Probing control with randomly assigned binary labels on full-condition hidden states (1,000 samples, 5-fold CV). Per-layer balanced accuracy is at chance ( ≈ 0.50) for every layer of every model, confirming that the near-perfect blind-vs-full separability reported in Figure 2 reflects condition information rather than trivial sample-identity separability.
Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning. Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns? We investigate this via VisualSwap, an image-swap probing framework: after a model reasons over an image, we replace it with a visually similar but semantically different one and test whether the model notices. We introduce VS-Bench, 800 image pairs curated from MathVista, MathVerse, MathVision, and MMMU-Pro. Experiments on Qwen3-VL, Kimi-VL, and ERNIE-VL reveal a striking failure: models overwhelmingly miss the swap, with accuracy dropping by up to 60%. Counterintuitively, thinking models are nearly 3x more vulnerable than their instructed counterparts, and scaling offers no mitigation. Multi-turn user instructions restore visual grounding, but self-generated reflective statements during continuous generation do not. Attention analysis explains why: user instructions substantially elevate attention to visual tokens, whereas self-reflection does not. Current VLMs tend to say rather than actually see when claiming to perform visual re-examination. Our code and dataset are available at the project page: https://visualswap.github.io
Chufan Shi, Cheng Yang, Yaokang Wu +4
University of Southern California · University of California San Diego · Carnegie Mellon Univer +1
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5