Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

Arya Shah; Vaibhav Tripathi; Mayank Singh; Chaklam Silpasuwanchai

cs.CVApr 15, 2026

Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

Authors: Arya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai

Organizations: Indian Institute of Technology Gandhinagar Gandhinagar, India · Asian Institute of Technology Bangkok, Thailand

Abstract

Vision-language models are increasingly deployed in high-stakes settings, yet their susceptibility to sycophantic manipulation remains poorly understood, particularly in relation to how these models represent visual information internally. Whether models whose visual representations more closely mirror human neural processing are also more resistant to adversarial pressure is an open question with implications for both neuroscience and AI safety. We investigate this question by evaluating 12 open-weight vision-language models spanning 6 architecture families and a 40 $\times$ parameter range (256M--10B) along two axes: brain alignment, measured by predicting fMRI responses from the Natural Scenes Dataset across 8 human subjects and 6 visual cortex regions of interest, and sycophancy, measured through 76,800 two-turn gaslighting prompts spanning 5 categories and 10 difficulty levels. Region-of-interest analysis reveals that alignment specifically in early visual cortex (V1--V3) is a reliable negative predictor of sycophancy ( $r = -0.441$ , BCa 95% CI $[-0.740, -0.031]$ ), with all 12 leave-one-out correlations negative and the strongest effect for existence denial attacks ( $r = -0.597$ , $p = 0.040$ ). This anatomically specific relationship is absent in higher-order category-selective regions, suggesting that faithful low-level visual encoding provides a measurable anchor against adversarial linguistic override in vision-language models. We release our code on GitHub and dataset on Hugging Face

Figures & tables

Figure 1: Overview of the three-stage pipeline. Stage 1: Vision encoder features are extracted from 12 VLMs and used to predict fMRI responses across 6 visual cortex ROIs in 8 human subjects (Algonauts 2023). Stage 2: Each model is evaluated on 6,400 two-turn gaslighting prompts spanning 5 manipulation categories and 10 difficulty levels. Stage 3: Brain alignment scores are correlated with sycophancy rates at both aggregate and ROI-specific levels, with robustness checks including BCa bootstrap, leave-one-out, and permutation testing.

Study	Focus	Brain Align.	Syc. Eval.	Multi-turn	VLM	ROI-level
( Schrimpf et al., 2020 )	Brain-Score benchmark	✓				✓
( Conwell et al., 2024 )	Inductive biases in brain alignment	✓				✓
( Sucholutsky and Griffiths, 2023 )	Alignment & few-shot robustness	✓
( Hoak et al., 2025 )	Alignment & $\ell_{\infty}$ robustness	✓
( Sharma et al., 2025 )	Sycophancy characterization		✓	✓
( Perez et al., 2023 )	Model-written evaluations		✓

Table 1: Comparison with prior work across brain alignment, sycophancy evaluation, and their intersection. Brain Align. : whether the study measures neural predictivity. Syc. Eval. : whether the study evaluates sycophantic behavior. Multi-turn : whether the adversarial evaluation uses multi-turn pressure. VLM : whether the study targets vision-language models. ROI-level : whether brain alignment is analyzed per region of interest. A check (✓) indicates the feature is present.

Model	Params	Vision Encoder	Source
SmolVLM-256M	256M	SigLIP	( Marafioti et al., 2025 )
SmolVLM-500M	500M	SigLIP	( Marafioti et al., 2025 )
Gemma-3-1B	1B	SigLIP	( Team et al., 2025 )
LFM-2-VL-1B	1.6B	SigLIP2-NaFlex	( Amini et al., 2025 )
Qwen2-VL-2B	2B	Qwen-ViT	( Wang et al., 2024 )
BLIP-2-OPT-2.7B	2.7B	ViT-G/14 + Q-Former	( Li et al., 2023a )

Table 2: Overview of the 12 VLMs evaluated in this study, ordered by parameter count. Vision Encoder : the architecture of the frozen visual backbone. Params : total model parameter count.

ROI	$r$	Perm. $p$	BCa 95% CI	Excl. 0	LOO
prf-visualrois	$-$ 0.441	0.071	[ $-$ 0.740, $-$ 0.031]	✓	✓
streams	$-$ 0.244	0.232	[ $-$ 0.622, 0.175]		✓
floc-places	$-$ 0.178	0.316	[ $-$ 0.626, 0.332]		✓
floc-faces	$-$ 0.111	0.403	[ $-$ 0.538, 0.337]
floc-bodies	$-$ 0.069	0.456	[ $-$ 0.566, 0.436]
floc-words	$-$ 0.064	0.458	[ $-$ 0.531, 0.432]

Table 4: ROI-specific correlations between brain alignment and sycophancy rate across

K=12

VLMs.

r

: Pearson correlation. Perm.

p

: one-tailed permutation

p

-value (10,000 permutations). BCa 95% CI : bias-corrected and accelerated bootstrap confidence interval (10,000 resamples). Excl. 0 : whether the BCa CI excludes zero. LOO : whether all leave-one-out correlations are negative.

Figure 2: Brain alignment score (prf-visualrois) versus final sycophancy rate for all 12 VLMs. Each point represents one model. The negative trend (

r=-0.441

, BCa 95% CI [

-

0.740,

-

0.031]) indicates that models with higher early visual cortex alignment tend to exhibit lower sycophancy rates.

Figure 3: Cohen’s

d

effect sizes comparing brain alignment between resistant (

\Sigma<0.50

n=4

) and susceptible (

\Sigma\geq 0.50

n=8

) models across six ROIs. Error bars show bootstrap 95% CIs. Positive values indicate that resistant models have higher brain alignment. All ROIs show small-to-medium positive effects, with floc-places (

d=0.63

) and streams (

d=0.61

) largest.

ROI	$\bar{B}^{R}$	$\bar{B}^{S}$	$\Delta$	$d$	$t$	$p$
prf-visualrois	.336	.323	.013	0.55	0.81	.436
floc-bodies	.460	.451	.009	0.47	0.68	.512
floc-faces	.423	.415	.008	0.51	0.71	.493
floc-places	.446	.437	.009	0.63	0.91	.386
floc-words	.360	.354	.006	0.38	0.55	.594
streams	.373	.363	.009	0.61	0.86	.411

Table 5: Group comparison of brain alignment scores between resistant (

n=4

) and susceptible (

n=8

) VLMs.

\bar{B}^{R}

: mean score for resistant group.

\bar{B}^{S}

: mean score for susceptible group.

\Delta

: difference.

d

: Cohen’s

d

Figure 4: Leave-one-out sensitivity analysis for the prf-visualrois correlation. Each bar shows the Pearson

r

when the indicated model is excluded. All 12 LOO correlations are negative (range: [

-

0.531,

-

0.325]), confirming that the finding is not driven by any single model. The dashed line indicates the full-sample correlation (

r=-0.441

ROI	CAT1	CAT2	CAT3	CAT4	CAT5
prf-visualrois	$-$ .409	$-$ .470	$-$ .597 ∗	$-$ .286	$-$ .413
streams	$-$ .224	$-$ .246	$-$ .413	$-$ .124	$-$ .223
floc-places	$-$ .160	$-$ .173	$-$ .330	$-$ .078	$-$ .166
floc-faces	$-$ .109	$-$ .090	$-$ .259	$-$ .026	$-$ .093
floc-words	$-$ .054	$-$ .063	$-$ .217	.026	$-$ .042
floc-bodies	$-$ .068	$-$ .056	$-$ .204	.007	$-$ .053

Table 6: Cross-correlation matrix: Pearson

r

between ROI-specific brain alignment and category-specific sycophancy rates. Bold with asterisk indicates

p<0.05

(uncorrected). CAT1: Object Misidentification. CAT2: Attribute Manipulation. CAT3: Existence Denial. CAT4: Count Falsification. CAT5: Authority Appeal.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Turn-1 $\Sigma$	$\Pi$	Final $\Sigma$	$\Delta$
SmolVLM-500M	0.03%	3.7%	3.7%	3.7%
Qwen2.5-VL-3B	7.8%	0.7%	8.5%	0.6%
Phi-3.5-Vision	3.9%	20.4%	23.5%	19.6%
Gemma-3-1B	4.5%	39.5%	42.2%	37.7%
LLaVA-v1.6-7B	9.6%	56.0%	60.2%	50.6%
Idefics2-8B	15.8%	54.4%	61.6%	45.8%

Appendix

Table 8: Two-turn attack statistics for all 12 VLMs. Turn-1

\Sigma

: sycophancy rate at Turn 1 (before escalation).

\Pi

: pressure conversion rate (fraction of initially resistant responses that become sycophantic at Turn 2). Final

\Sigma

: overall sycophancy rate after both turns.

\Delta

: absolute increase from Turn-1 to final sycophancy.

Cat.	Description	Mean $\Sigma$	Std	$r$	$p$
CAT1	Object Misidentification	69.1%	0.323	$-$ 0.029	.929
CAT2	Attribute Manipulation	56.1%	0.394	$-$ 0.020	.950
CAT3	Existence Denial	53.0%	0.326	$-$ 0.223	.486
CAT4	Count Falsification	68.5%	0.377	0.018	.955
CAT5	Authority Appeal	64.1%	0.374	$-$ 0.020	.951

Appendix

Table 9: Category-specific sycophancy rates and correlations with overall brain alignment. Visual-domain categories (CAT1–CAT4) show stronger (more negative) mean correlation than the social-domain category (CAT5).

Family	Models	Brain Score	Mean $\Sigma$
Qwen-ViT	Qwen2-VL-2B, Qwen2.5-VL-3B	0.995	40.8%
CLIP-ViT	LLaVA-v1.6-7B, Phi-3.5-Vision	0.995	41.8%
SigLIP	SmolVLM-256M/500M, Gemma-3-1B, PaliGemma2-10B	0.993	61.0%
SigLIP (mod.)	Idefics2-8B	0.991	61.6%
ViT-G/14	BLIP-2-OPT-2.7B	0.993	94.7%
SigLIP2-NaFlex	LFM-2-VL-1B, LFM-2-VL-8B	0.997	96.5%

Appendix

Table 10: Architecture family comparison. Brain Score : mean normalized brain alignment. Mean

\Sigma

: mean final sycophancy rate. Families are ordered by mean sycophancy.

Rank	Tactic	Mean $\Sigma$	Std
1	Statistics	86.5%	0.278
2	Question	82.2%	0.329
3	Specific authority	77.5%	0.289
4	Data appeal	75.2%	0.428
5	Institutional authority	75.2%	0.428
6	Weak suggestion	74.4%	0.363

Appendix

Table 11: Top 10 most effective and bottom 5 least effective persuasion tactics, ranked by mean sycophancy rate across 12 VLMs. 65 total tactics were analyzed.

Model	AURC	Slope
SmolVLM-500M	0.952	$-$ 0.003
Qwen2.5-VL-3B	0.912	$-$ 0.000
Phi-3.5-Vision	0.747	0.073
Gemma-3-1B	0.590	0.014
LLaVA-v1.6-7B	0.367	0.018
Idefics2-8B	0.358	0.015

Appendix

Table 12: Resistance curve statistics for all 12 VLMs. AURC : area under the resistance curve (higher = more resistant). Slope : linear trend of resistance across difficulty levels (positive = resistance increases with difficulty; negative = decreases).

Level	Mean $\Sigma$	$r$	$p$
1	69.8%	$-$ 0.350	.265
2	72.5%	$-$ 0.113	.727
3	60.6%	$-$ 0.200	.533
4	60.0%	$-$ 0.270	.396
5	57.7%	$-$ 0.200	.533
6	80.9%	$-$ 0.186	.563

Appendix

Table 13: Brain alignment vs. sycophancy correlation at each difficulty level.

Figure 5: ROI atlas showing the six ROI categories mapped onto the cortical surface. Colors indicate different ROI categories: prf-visualrois (V1–V3, hV4), floc-bodies (EBA, FBA), floc-faces (OFA, FFA), floc-places (OPA, PPA, RSC), floc-words (VWFA), and streams (early through parietal).

Figure 6: Heatmap of brain alignment scores across all 12 models and 6 ROI categories. Darker colors indicate higher brain alignment. The prf-visualrois column shows the greatest inter-model variability, particularly the low score of PaliGemma2-10B (0.273).

Figure 7: Group comparison of brain alignment scores between resistant ( $n=4$ ) and susceptible ( $n=8$ ) VLMs for each ROI, with individual model data points overlaid.

Figure 8: Bar chart comparing per-ROI brain alignment scores for each of the 12 VLMs. Error bars indicate standard deviation across subjects.

Figure 9: Overview of the Algonauts 2023 dataset, showing sample natural scene images from MS-COCO and the corresponding fMRI recording structure across 8 subjects.

Sycophancy in VLMs

See all 11 papers

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
Mar 19, 2026Rui Hong, Shuxue QuanSelective Prediction for VLMs Sycophancy in VLMs
George Mason University, Fairfax, VA, USA · Independent Researcher
To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
Mar 23, 2026OFM Riaz Rahman Aranya, Kevin DesaiSycophancy in VLMs Medical VLMs
Department of Computer Science, The University of Texas at San Antonio
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
Aug 19, 2026Mehak Gupta, Tanmoy ChakrabortyVLM Safety Alignment Abstention in VLMs
Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India

Mar 19, 2026J/K move · Enter open · S save

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

Rui Hong, Shuxue Quan

George Mason University, Fairfax, VA, USA · Independent Researcher

Selective Prediction for VLMs Sycophancy in VLMs Vision-Language Models Vision-Language Grounding Visual Evidence Sensitivity in VLMs Hallucination Detection in VLMs+1

Open paper AskPDF arXiv

When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.

Abstract

Sycophancy in VLMs

Explore similar work

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models