Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

Muhammad Zeeshan Karamat; Syed Talal Hassan; Christiana Chamon Garcia

cs.CVOct 23, 2026

Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

Authors: Muhammad Zeeshan Karamat, Syed Talal Hassan, Christiana Chamon Garcia

Organizations: Virginia Tech

Abstract

Vision-Language Models (VLMs) are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth. Causal interventions show that early vision layers support strong bidirectional refusal control through activation steering along a learned safety direction, with unsafe rejection spanning 6%-96%, whereas later layers respond more to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off. Deeper layers remain well separated along the safety direction yet are largely insensitive to steering, distinguishing representational separability from causal controllability. InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry despite differences in layer location and magnitude. Our results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.

Figures & tables

Figure 1: Layer-wise Mean Angular Divergence for Qwen3-VL-4B. (a,c) The language tower exhibits strong early directional separation that generally decreases with depth, with SU–UU remaining larger than SU–SS across most layers. (b,d) The vision encoder exhibits non-linear, heterogeneous divergence with distinct safety-sensitive regions around Layers 7–10 and 18–22.

Figure 2: Layer-wise Mean Angular Divergence for InternVL2.5-4B. (a,c) The language tower shows stronger early angular divergence that generally decreases with depth, qualitatively matching Qwen3-VL-4B. (b) Vision SU–SS divergence is stronger in early and intermediate layers and decreases toward later depth. (d) Vision SU–UU transitions from negative to positive divergence across depth, again qualitatively resembling Qwen3-VL-4B. The precise layer locations and divergence magnitudes differ across architectures, suggesting recurring depth-dependent safety geometry with model-dependent localization.

Figure 3: Projection of CLS token activations onto the learned safety direction (

\mathbf{E}_{1}

) at Layer 20 of Qwen3-VL-4B. Safe and unsafe inputs form clearly shifted distributions with limited overlap, indicating strong representational separability along a safety-related direction in deeper vision layers. Subsequent intervention results show that such separability does not necessarily imply equivalent directional controllability.

$\alpha$	Vision 7–10		Vision 18–22		Language
$\alpha$	URR	SRR	URR	SRR	URR	SRR
0.85	100	99	79	43	37	8
0.90	95	89	94	72	80	49
1.00	72	26	72	26	72	26
1.15	72	23	96	91	97	77
1.20	72	24	88	89	98	98

Table 1: Parameter scaling results on Unsafe-bench (Qwen3-VL-4B). URR denotes Unsafe Rejection Rate and SRR denotes Safe Rejection Rate.

$\alpha$	Vision 7–10	Vision 18–22	Language
0.85	100	16	0
0.90	100	16	0
1.00	0	0	0
1.15	9	8	6
1.20	7	8	5

Table 2: Parameter scaling results on Google Conceptual Captions (Qwen3-VL-4B). SRR denotes Safe Rejection Rate (%).

	Layers 7–10				Layers 18–20
$\alpha$	Focused		Global		Focused		Global
	URR	SRR	URR	SRR	URR	SRR	URR	SRR
Baseline ( $\alpha=0$ )	84	32	84	32	84	32	84	32
0.5	84	32	84	24	82	26	82	28
1.0	84	28	94	60	82	30	82	32
2.0	84	32	70	44	82	30	84	38

Table 3: Causal mediation analysis with activation steering on Qwen3-VL-4B. URR denotes Unsafe Rejection Rate and SRR denotes Safe Rejection Rate.

VLM Safety

See all 32 papers

Visual Grounding Safety in Vision-Language Models
Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM Evaluation VLM Safety Alignment
University of California, Riverside · Work done during internship at Apple. · Apple
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
Aug 19, 2026Mehak Gupta, Tanmoy ChakrabortyVLM Safety Alignment Abstention in VLMs
Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India

Oct 5, 2026J/K move · Enter open · S save

Visual Grounding Safety in Vision-Language Models

Erfan Shayegani, Kundan Krishna, Yue Dong, Nael Abu-Ghazaleh+2

University of California, Riverside · Work done during internship at Apple. · Apple

VLM Evaluation VLM Safety Alignment Vision-Language Grounding Visual Grounding VLM Safety

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.

Abstract

VLM Safety

Explore similar work

Visual Grounding Safety in Vision-Language Models

Do VLMs Share Safety Neurons Across Modalities?

When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models