To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

Rui Hong; Shuxue Quan

cs.CVMar 19, 2026

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

Authors: Rui Hong, Shuxue Quan

Organizations: George Mason University, Fairfax, VA, USA · Independent Researcher

Abstract

When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.

Figures & tables

Figure 1: Tri-Layer Hallucination Diagnostic Framework. Each (question, image) pair is queried under full / blind / Gaussian-noise conditions with a set of refusal anchors. Three per-sample metrics are extracted—LAD (perception; refusal-prob. gap blind vs. full), VNS (dependency; KL between full and blind output distributions), CS (alignment; answer-vs-refusal log-ratio)—and cascaded into four failure categories (P.B., L.SC, V.S., R.R.; rule table: Table 1 ).

Figure 2: Layer-wise blind-vs-full linear separability at the last text-token position, 9 models overlaid against relative depth (balanced 5-fold CV,

N{=}1{,}000

/condition). Eight grey lines saturate at

\geq 0.996

from layer 1. Llama-3.2-11B-Vision (red) is the sole outlier (layer-1 acc. 0.19, reaching 1.00 by layer 4)—consistent with its cross-attention architecture. Yet the same samples emit hallucinations at SC blind rates of

1.1\%

–

99.4\%

(Table 2 )—a decoding-time override of fully-preserved internal evidence.

Category	LAD (Perc.)	VNS (Dep.)	CS (Align.)	Diagnostic Interpretation
Perceptual Blindness	$\leq\tau_{LAD}$	–	–	Encoder failure: model cannot distinguish blind from natural input; downstream metrics undefined.
Language Shortcut	$>\tau_{LAD}$	$\leq\tau_{VNS}$	–	Visual neglect: Model detects anomaly but ignores visual signal.
Visual Sycophancy	$>\tau_{LAD}$	$>\tau_{VNS}$	$>\tau_{CS}$	Split beliefs: Perceives anomaly but hallucinates to satisfy instruction.
Robust Refusal	$>\tau_{LAD}$	$>\tau_{VNS}$	$\leq\tau_{CS}$	Ideal behavior: Visual grounding overrides language priors.

Table 1: Diagnostic Taxonomy based on the Tri-Layer Framework. “–” indicates inapplicable metrics. Threshold selection is detailed in Section 3 .

Model	Acc.	SC blind	SC noise	SC conf
Qwen3-VL-32B	75.8	1.1	0.3	14.3
Qwen3-VL-8B	75.1	4.7	2.6	12.2
Qwen2.5-VL-72B	73.5	40.4	9.4	9.7
Qwen2.5-VL-7B	72.1	45.8	67.9	15.1
Molmo2-4B	71.3	99.4	79.1	37.1
Llama-3.2-11B	69.9	45.8	57.8	37.8

Table 2: Response-level results (%),

N{=}1000

per model. Acc. (higher is better) under full condition; SC x = shortcut rate under condition

x

(lower is more grounded).

Model	P.B. $\downarrow$	L.SC $\downarrow$	V.S. $\downarrow$
Molmo2-4B	42.8	25.2	32.0
Phi-3.5-Vision	1.6	39.8	58.6
LLaVA-NeXT-7B	0.0	29.7	70.3
Qwen2.5-VL-7B	0.1	27.5	72.4
Llama-3.2-11B	0.0	21.2	78.8
Pixtral-12B	5.3	14.8	79.9

Table 3: Taxonomy distribution (%, blind condition). Robust Refusal is 0% (omitted). Per-category accuracies (P.B./L.SC/V.S./Overall: 65.9/69.2/70.9/70.7%) confirm P.B. is the only category warranting abstention.

Category	$\mathbf{-1.0}$	$\mathbf{-0.5}$	$\mathbf{0.0}^{*}$	$\mathbf{0.5}$	$\mathbf{1.0}$
P.B.	7.1	7.1	7.1	7.1	7.1
L.SC	23.3	23.3	23.3	23.3	23.3
V.S.	69.6	69.6	69.6	69.3	59.2
R.R.	0.0	0.0	0.0	0.4	10.5

Table 4: Taxonomy shares (%) under

\tau_{CS}

sweep (

\tau_{LAD}{=}1.5

\tau_{VNS}{=}1.0

fixed; ∗ =default). R.R. stays at 0% across two orders of magnitude of the natural probabilistic boundary. Full

\tau_{LAD}

\tau_{VNS}

sweeps: Appendix B .

Model	VNS $\uparrow$	LAD $\uparrow$	CS $\downarrow$
Qwen3-VL-32B	2.81 $\pm$ 1.02	5.32 $\pm$ 1.06	1.99 $\pm$ .56
Qwen3-VL-8B	2.70 $\pm$ 1.09	4.97 $\pm$ .77	2.18 $\pm$ .53
Qwen2.5-VL-72B	2.59 $\pm$ 1.09	4.17 $\pm$ .82	1.91 $\pm$ .39
Pixtral-12B	2.06 $\pm$ 1.16	3.12 $\pm$ .96	3.04 $\pm$ .75
Llama-3.2-11B	1.97 $\pm$ 1.08	3.24 $\pm$ .45	0.87 $\pm$ .33
Qwen2.5-VL-7B	1.75 $\pm$ .97	3.85 $\pm$ .77	1.42 $\pm$ .50

Table 5: Per-model Tri-Layer metrics (mean

\pm

std, blind condition). Noise-variant columns are reported in Appendix C .

Model	VQAv2	HB (abs $\tau$ )	HB (pct $\tau$ )
Qwen2.5-VL-7B	72.4	73.5	72.0
Qwen3-VL-8B	96.6	99.0	96.0
Pixtral-12B	79.9	66.5	79.0

Table 6: Cross-dataset V.S. share (%) on VQAv2 vs HallusionBench under absolute and percentile-anchored thresholds. Pixtral is the diagnostic case; the other two are intrinsically stable.

Task	N	Acc. $\uparrow$	SC blind $\downarrow$	SC conf $\downarrow$	P.B. $\downarrow$	L.SC $\downarrow$	V.S. $\downarrow$
Halluc.	3150	80.1	43.8	24.1	6.1	11.4	82.5
Spatial	2250	69.4	48.7	46.9	5.1	25.4	69.6
Complex	2250	68.9	47.4	52.6	4.4	24.3	71.3
Count.	1350	63.0	46.4	18.5	7.0	14.6	78.4

Table 7: Task-wise results (%),

N

pooled across 9 models. R.R. is 0% across all tasks (omitted). Abbreviations follow Table 2 .

Model	POPE Acc	POPE F1
Qwen3-VL-8B	88.5	88.1
LLaVA-NeXT-7B	88.0	87.8
Qwen3-VL-32B	87.0	86.5
Qwen2.5-VL-72B	86.8	85.1
Llama-3.2-11B	86.3	87.2
Qwen2.5-VL-7B	86.3	84.4

Table 8: POPE overall results (%, average over Random/Popular/Adversarial subsets, 600 queries/model). All models in 79.5–88.5% range. Per-subset breakdown in Appendix E .

Model	Acc.	VNS	LAD	L.SC	V.S.
Qwen2.5-VL-7B	72.1	1.75	3.85	27.5	72.4
Qwen2.5-VL-72B	73.5	2.59	4.17	4.7	95.3
Qwen3-VL-8B	75.1	2.70	4.97	3.4	96.6
Qwen3-VL-32B	75.8	2.81	5.32	0.9	98.9

Table 9: Scaling within Qwen-VL (%). The three axes (within-gen 7B

\to

72B, gen upgrade 7B

\to

8B, within-gen 8B

\to

32B) reduce L.SC and amplify V.S. monotonically; LAD climbs in parallel. SC blind /CS in Table 2 / Appendix C .

Model	Base	Rand	LAD	VNS	Ours	$\Delta\uparrow$
Qwen3-VL-32B	75.8	75.2	79.0	84.6	82.6	+6.8
Qwen3-VL-8B	75.1	74.8	77.4	84.8	84.0	+8.9
Qwen2.5-VL-72B	73.5	73.2	73.4	79.2	77.8	+4.3
Qwen2.5-VL-7B	72.1	72.6	80.8	78.8	81.6	+9.5
Molmo2-4B	71.3	72.2	72.6	74.0	76.6	+5.3
Llama-3.2-11B	69.9	68.2	70.0	69.0	69.8	$\approx$ 0.0

Table 10: Selective prediction at 50% coverage. Base = full-coverage accuracy; Rand = random 50% subset (control, seed 42); LAD / VNS = single-metric ranking; Ours = category-conditional routing.

\Delta

= Ours

-

Base. Underlined = best per row.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

		Decision Point
Model	Resp. Len	Blind	Conflict	Noise
Molmo2-4B	47.1	17.4	14.0	15.3
Phi-3.5-Vision	15.6	4.4	4.6	4.2
LLaVA-NeXT-7B	38.9	11.7	12.1	11.7
Qwen2.5-VL-7B	52.8	15.2	9.6	16.5
Llama-3.2-11B	22.0	8.5	8.3	8.8

Appendix

Table 11: Per-token VNS statistics. Resp. Len = mean response length (tokens); Decision Point = mean token index with maximum KL divergence between full-image and blind/conflict/noise conditions. Visual influence consistently peaks in the first half of generation across all conditions, validating our top-

k

aggregation strategy.

Category	$\mathbf{0.5}$	$\mathbf{1.0}$	$\mathbf{1.5}^{*}$	$\mathbf{2.0}$	$\mathbf{2.5}$
P.B.	0.2	1.0	7.1	16.0	26.6
L.SC	26.3	26.0	23.3	19.5	15.6
V.S.	73.5	73.1	69.6	64.5	57.9
Robust	0.0	0.0	0.0	0.0	0.0

Appendix

Table 12: Taxonomy proportions (%) under

\tau_{\text{LAD}}

sweep (

\tau_{\text{VNS}}=1.0

\tau_{\text{CS}}=0

fixed; ∗ = default). V.S. remains the dominant category across all settings.

Category	$\mathbf{0.5}$	$\mathbf{1.0}^{*}$	$\mathbf{1.25}$	$\mathbf{1.5}$	$\mathbf{2.0}$
P.B.	7.1	7.1	7.1	7.1	7.1
L.SC	9.0	23.3	31.1	39.0	54.0
V.S.	83.9	69.6	61.8	53.9	38.8
Robust	0.0	0.0	0.0	0.0	0.0

Appendix

Table 13: Taxonomy proportions (%) under

\tau_{\text{VNS}}

sweep (

\tau_{\text{LAD}}=1.5

\tau_{\text{CS}}=0

fixed; ∗ = default). V.S. remains the dominant category across all but the most extreme setting; maximum deviation from default: 69.6%

-

38.8%

=

30.8 pp.

Model	VNS ${}_{noise}\uparrow$	LAD ${}_{noise}\uparrow$	CS ${}_{noise}\downarrow$
Qwen3-VL-32B	3.28 $\pm$ 1.15	2.78 $\pm$ .88	4.10 $\pm$ .96
Qwen3-VL-8B	2.77 $\pm$ 1.16	3.04 $\pm$ .61	3.22 $\pm$ .57
Qwen2.5-VL-72B	2.08 $\pm$ .88	3.39 $\pm$ .69	2.17 $\pm$ .24
Pixtral-12B	2.22 $\pm$ 1.27	2.22 $\pm$ .51	3.12 $\pm$ .55
Llama-3.2-11B	2.13 $\pm$ 1.15	2.39 $\pm$ .38	1.39 $\pm$ .28
Qwen2.5-VL-7B	1.62 $\pm$ .86	3.25 $\pm$ .78	2.21 $\pm$ .27

Appendix

Table 14: Tri-Layer metrics under the Gaussian noise condition (mean

\pm

std). VNS noise correlates with VNS (Table 5 ) at

r{=}0.866

per-sample. LAD noise trails LAD (mean 2.38 vs. 3.09): black images are recognized as more anomalous than noise (which provides texture the encoder partially processes).

	Random			Popular			Adversarial			Overall
Model	Acc	F1	Yes	Acc	F1	Yes	Acc	F1	Yes	Acc	F1
Qwen3-VL-32B	91.5	90.7	41.5	88.0	87.4	45.0	81.5	81.8	51.5	87.0	86.5
Qwen3-VL-8B	92.0	91.4	43.0	88.5	88.1	46.5	85.0	85.0	50.0	88.5	88.1
Qwen2.5-VL-72B	87.5	85.7	37.5	86.5	84.8	38.5	86.5	84.8	38.5	86.8	85.1
Qwen2.5-VL-7B	87.0	85.1	37.0	86.0	84.1	38.0	86.0	84.1	38.0	86.3	84.4
Molmo2-4B	90.0	90.5	55.0	83.5	85.2	61.5	77.5	80.8	67.5	83.7	85.3

Appendix

Table 15: POPE benchmark results (200 samples per subset

\times

3 subsets = 600 queries per model). Acc = accuracy (%), F1 = positive-class F1 (%), Yes = %-yes rate. Standard decoding only. Best per column in bold .

			Absolute $\tau$ (1.5, 1.0, 0)				Percentile-anchored $\tau$
Model	Dataset	N	P.B.	L.SC	V.S.	R.R.	P.B.	L.SC	V.S.	R.R.
Qwen2.5-VL-7B	VQAv2	1000	0.1	27.5	72.4	0.0	0.1	27.5	72.4	0.0
	HallusionBench	200	0.0	26.5	73.5	0.0	0.5	27.5	72.0	0.0
Qwen3-VL-8B	VQAv2	1000	0.0	3.4	96.6	0.0	0.1	3.4	96.5	0.0
	HallusionBench	200	0.0	1.0	99.0	0.0	0.5	3.5	96.0	0.0
Pixtral-12B	VQAv2	1000	5.3	14.8	79.9	0.0	5.3	14.8	79.9	0.0

Appendix

Table 16: Cross-dataset taxonomy stability on VQAv2 (1,000 samples; our main eval) vs HallusionBench (200 yes/no samples). Absolute

\tau

: paper-default

(1.5,1.0,0)

applied directly. Percentile-anchored

\tau

\tau_{LAD}

and

\tau_{VNS}

are set to the per-(model, dataset) local LAD/VNS values at the percentile ranks that the absolute thresholds occupy in VQAv2;

\tau_{CS}{=}0

is retained. Pixtral-12B is the diagnostic case: under absolute

\tau

, the V.S. share drifts 79.9 %

\rightarrow

66.5 % across datasets; percentile-anchoring restores it to 79.0 %, within 1pp of the VQAv2 reference. Qwen2.5-VL-7B and Qwen3-VL-8B are intrinsically stable under both regimes.

Figure 3: Per-layer balanced accuracy of a logistic probe trained on full-condition hidden states with randomly assigned binary labels (5-fold CV; layers normalised to relative depth so models with different

L

overlay). All three models track the chance line (

0.50

) across the entire stack, in sharp contrast to the

\geq 0.99

peak balanced accuracy of the blind-vs-full probe on the same activations.

Model	L	mean	max	min
Qwen2.5-VL-7B	28	0.486	0.516	0.451
Qwen3-VL-8B	36	0.485	0.530	0.449
Pixtral-12B	40	0.478	0.513	0.450

Appendix

Table 17: Probing control with randomly assigned binary labels on full-condition hidden states (1,000 samples, 5-fold CV). Per-layer balanced accuracy is at chance (

\approx

0.50) for every layer of every model, confirming that the near-perfect blind-vs-full separability reported in Figure 2 reflects condition information rather than trivial sample-identity separability.

Explore similar work

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
May 15, 2026Chufan Shi, Cheng Yang, Yaokang Wu +4VLM Evaluation VLM Robustness
University of Southern California · University of California San Diego · Carnegie Mellon Univer +1
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Faithfulness in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM Evaluation Visual Question Answering
NIT Silchar · MBZUAI · BML Munjal University +1

May 15, 2026J/K move · Enter open · S save

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin+3

University of Southern California · University of California San Diego · Carnegie Mellon Univer+1

VLM Evaluation VLM Robustness VLM Interpretability VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning. Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns? We investigate this via VisualSwap, an image-swap probing framework: after a model reasons over an image, we replace it with a visually similar but semantically different one and test whether the model notices. We introduce VS-Bench, 800 image pairs curated from MathVista, MathVerse, MathVision, and MMMU-Pro. Experiments on Qwen3-VL, Kimi-VL, and ERNIE-VL reveal a striking failure: models overwhelmingly miss the swap, with accuracy dropping by up to 60%. Counterintuitively, thinking models are nearly 3x more vulnerable than their instructed counterparts, and scaling offers no mitigation. Multi-turn user instructions restore visual grounding, but self-generated reflective statements during continuous generation do not. Attention analysis explains why: user instructions substantially elevate attention to visual tokens, whereas self-reflection does not. Current VLMs tend to say rather than actually see when claiming to perform visual re-examination. Our code and dataset are available at the project page: https://visualswap.github.io

Abstract

Explore similar work

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread