cs.CVAug 19, 2026

When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models

Authors: Mehak Gupta, Tanmoy Chakraborty

Organizations: Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Aligned vision-language models (VLMs) are designed to balance grounded visual reasoning with safe generation behavior. However, we observe a striking phenomenon: under safety-constrained instruction, models frequently abstain from answering questions that remain correctly answerable under default instruction despite receiving identical image-question inputs. This raises a fundamental question: does safety alignment suppress perceptual grounding itself, or does visual evidence remain internally available while generation is redirected toward abstention? In this work, we investigate the internal decoding dynamics underlying safety-induced abstention in aligned VLMs. Across multiple architectures and multimodal benchmarks, we show that abstained generations remain consistently influenced by visual evidence throughout decoding, indicating that perceptual grounding is largely preserved despite refusal behavior. We further demonstrate that, although the representational organization of refusal differs substantially across architectures, safety-constrained instruction consistently alters late-stage hidden-state dynamics toward refusal-oriented decoding. Finally, through targeted activation-level interventions, we show that suppressing refusal-related representations reliably restores grounded answering behavior across models without retraining or modifying visual inputs. Together, these findings reveal a previously underexplored failure mode in aligned VLMs: safety alignment can override grounded visual expression even when perceptual evidence remains internally preserved.

Figures & tables

Figure 1: Illustration of the central concept studied in this work. Given the same image and question, a VLM produces a grounded answer under default instruction but abstains under safety-constrained instruction. Although the visual evidence remains unchanged and recoverable, alignment redirects generation toward refusal behavior. This raises a key question: does safety alignment suppress perceptual grounding itself or is visual evidence internally preserved while decoding trajectories shift toward refusal-aligned representations?
ModelVQAv2GQAPOPE
DefaultRestrictiveDefaultRestrictiveDefaultRestrictive
Qwen3-VL-8B-Instruct0.00.320.00.280.00.56
LLaVA-v1.6-Vicuna-7B0.00.130.00.030.00.04
Phi-3-Vision-128K-Instruct0.070.440.040.400.060.36
Table 1: Abstention rates (%) across three vision-language models under default and safety-constrained instruction on VQAv2, GQA and POPE. Safety-constrained instruction consistently increases abstention behavior revealing the sensitivity of aligned VLMs to safety-oriented instruction framing, even for benign visual question answering tasks.
Figure 2: Layerwise vision influence scores for Qwen3-VL, LLaVA-v1.6 and Phi-3-Vision on VQA-v2 under default and safety-constrained instruction. Each subplot contains paired layerwise trajectories for answered and abstained samples across instructions. Across architectures, abstained samples retain vision influence throughout decoding, although the evolution differs across architectures.
Figure 3: Layer-wise projection difference onto the learned refusal direction between default and safety-constrained prompting across aligned VLMs on VQA-v2 which shows the evolution of refusal-direction alignment across decoding layers for answered and abstained generations. Qwen3-VL exhibits strong separation between answered and abstained trajectories, LLaVA-v1.6 shows moderate separation, whereas Phi-3-Vision displays largely overlapping trajectories despite measurable changes in refusal alignment suggesting that safety-induced abstention is associated with architecture-dependent representational dynamics.
Figure 4: Joint representational evolution of visual grounding and refusal alignment during decoding. Contour densities show abstained generations progressively migrate toward stronger refusal-aligned regimes in deeper layers while retaining substantial visual grounding. Detailed layer-wise analysis is shown in Figures 8 , 9 , 10 .
  OriginalRefusal SuppressionRefusal Amplification
ModelDatasetCorrectAbstainIncorrectCorrectAbstainIncorrectCorrectAbstainIncorrect
Qwen3-VL-8B-InstructVQA-v20.630.320.050.810.090.100.510.450.04
Qwen3-VL-8B-InstructGQA0.530.280.190.630.050.320.450.390.16
Qwen3-VL-8B-InstructPOPE0.410.560.030.680.010.330.340.650.01
LLaVA-v1.6-Vicuna-7b-hfVQA-v20.780.130.090.810.090.100.740.170.09
LLaVA-v1.6-Vicuna-7b-hfGQA0.730.030.240.750.010.240.710.050.24
Table 2: Effect of refusal-aligned interventions under safety-constrained instruction across models and datasets. Refusal suppression substantially reduces abstention and recovers grounded answering behavior, while refusal amplification increases conservative refusal behavior. Hallucination increases remain comparatively modest relative to the reduction in abstention.
Figure 5: Layerwise abstention rate under refusal suppression and refusal amplification across models on VQA-v2. Interventions applied to deeper decoding layers produce substantially larger behavioral changes than interventions applied to earlier layers.
Figure 6: Layerwise top-2 logit gap across Qwen3-VL, LLaVA-v1.6 and Phi-3-Vision under default and safety-constrained instruction. Answered and abstained samples exhibit broadly comparable logit gap dynamics across transformer layers, suggesting that abstention behavior is not solely explained by differences in token-level output confidence.
Figure 7: Layerwise vocabulary entropy across Qwen3-VL, LLaVA-v1.6 and Phi-3-Vision under default and safety-constrained instruction. Both answered and abstained samples exhibit similar entropy dynamics across layers indicating that predictive uncertainty alone is insufficient to explain abstention behavior in aligned vision-language models.
Figure 8: Interaction between layerwise vision influence and refusal projection dynamics in Qwen3-VL. Abstained samples under safety-constrained instruction exhibit progressively stronger alignment with the refusal direction accompanied by weakened vision influence in deeper layers.
Figure 9: Layerwise interaction between vision influence and refusal projection dynamics in LLaVA-v1.6. Abstained samples increasingly shift toward refusal-aligned representations while exhibiting progressively weaker vision influence in deeper layers.
Figure 10: Interaction between vision influence and refusal projection dynamics in Phi-3-Vision across layers. Unlike other models, Phi-3-Vision shows limited disentanglement between refusal alignment and visual grounding, with abstained and answered samples exhibiting similar projection trends across layers.
Figure 11: Sensitivity of refusal-direction interventions to scaling coefficient α\alpha across Qwen3-VL, LLaVA-v1.6 and Phi-3-Vision. Positive intervention strengths progressively increase alignment with the learned refusal direction and amplify abstention behavior, while negative interventions suppress refusal-aligned features and recover answer generation. The smooth behavioral transition across α\alpha values suggests that refusal behavior is governed by coherent and controllable representation directions.
Table 3: Qualitative examples on VQAv2 for Qwen3-VL.
Table 4: Qualitative examples on GQA for Qwen3-VL.
Table 5: Qualitative examples on POPE for Qwen3-VL.
Table 6: Qualitative examples on VQAv2 for LLaVA-v1.6.
Table 7: Qualitative examples on GQA for LLaVA-v1.6.
Table 8: Qualitative examples on POPE for LLaVA-v1.6.
Table 9: Qualitative examples on VQAv2 for Phi-3-vision.
Table 10: Qualitative examples on GQA for Phi-3-vision.
Table 11: Qualitative examples on POPE for Phi-3-vision.
More in this topic

VLM Safety Alignment

See all 15 papers

Explore similar work

CardsList
Oct 5, 2026

Visual Grounding Safety in Vision-Language Models

Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.

Erfan Shayegani, Kundan Krishna, Yue Dong +3

University of California, Riverside · Work done during internship at Apple. · Apple

VLM EvaluationVLM Safety AlignmentVision-Language GroundingVisual GroundingVLM Safety
AskPDF
Sep 3, 2026

SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models

Existing safety alignment methods for vision-language models usually modify the model behavior globally: once the safety parameters are trained or loaded, they participate in both unsafe and already-safe generations. This always-on intervention can unnecessarily perturb the model's original reasoning path and degrade general multimodal capabilities. We argue that safety alignment should be an on-demand intervention rather than a permanent modification to every decoding trajectory. To this end, we propose a streaming recognition and gated LoRA framework for intrinsic VLM safety. During autoregressive generation, a lightweight recognizer estimates whether the current pre-token generation state is safe or unsafe. Its output updates the LoRA gate for the following decoding step; otherwise, generation follows the frozen-backbone policy. The LoRA module is trained from unsafe prefixes, transition statements, and safe continuations, so that it learns to redirect unsafe generations back to safe responses after activation. Experiments across multiple safety and general-purpose benchmarks demonstrate the effectiveness of our method in post-alignment settings.

Caoyuan Ma, Tian Gu, Wenpu Liu +11

The University of Tokyo · Wuhan University · Shanghai AI Laboratory +4

Vision-Language ModelsVLM Safety AlignmentLarge Vision-Language ModelsVLM Safety
AskPDF
Apr 11, 2026

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Large Vision-Language Models (LVLMs) rely on attention-based retrieval of safety instructions to maintain alignment during generation. Existing attacks typically optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model's safety-retrieval mechanism. We propose Attention-Guided Visual Jailbreaking, which circumvents rather than overpowers safety alignment by directly manipulating attention patterns. Our method introduces two simple auxiliary objectives: (1) suppressing attention to alignment-relevant prefix tokens and (2) anchoring generation on adversarial image features. This simple yet effective push-pull formulation reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. At tighter perturbation budgets (ε=8/255ε=8/255), we maintain 59.0% ASR compared to 45.7% for standard methods. Mechanistic analysis reveals a failure mode we term safety blindness: successful attacks suppress system-prompt attention by 80%, causing models to generate harmful content not by overriding safety rules, but by failing to retrieve them.

Jingru Li, Wei Ren, Tianqing Zhu

China University of Geosciences, Wuhan · City University of Macau

Jailbreak AttacksAdversarial Attacks on VLMsVLM SafetyLarge Vision-Language ModelsAttention-Based Adversarial AttacksVLM Safety Alignment+1
AskPDF