Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Jingru Li; Wei Ren; Tianqing Zhu

cs.CVApr 11, 2026

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Authors: Jingru Li, Wei Ren, Tianqing Zhu

Organizations: China University of Geosciences, Wuhan · City University of Macau

Abstract

Large Vision-Language Models (LVLMs) rely on attention-based retrieval of safety instructions to maintain alignment during generation. Existing attacks typically optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model's safety-retrieval mechanism. We propose Attention-Guided Visual Jailbreaking, which circumvents rather than overpowers safety alignment by directly manipulating attention patterns. Our method introduces two simple auxiliary objectives: (1) suppressing attention to alignment-relevant prefix tokens and (2) anchoring generation on adversarial image features. This simple yet effective push-pull formulation reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. At tighter perturbation budgets ( $ε=8/255$ ), we maintain 59.0% ASR compared to 45.7% for standard methods. Mechanistic analysis reveals a failure mode we term safety blindness: successful attacks suppress system-prompt attention by 80%, causing models to generate harmful content not by overriding safety rules, but by failing to retrieve them.

Figures & tables

Figure 1: Overview of Attention-Guided Visual Jailbreaking. (A) Standard attacks face gradient conflict between adversarial and safety objectives. (B) Our method suppresses attention to safety instructions, aligning gradients. (C) This induces safety blindness , achieving 95.5% ASR vs. 70.0% baseline.

Figure 2: Gradient Conflict Analysis. (a) Optimization dynamics: baseline (blue) exhibits oscillation from gradient conflict, while our method (orange) maintains smooth convergence. (b) Our method reduces severe conflict instances (cos

<-0.5

) by 45%. (c) Distribution shift toward positive gradient alignment.

Figure 3: Method Overview. (A) Standard optimization suffers from gradient conflict: adversarial gradients oppose safety-retrieval signals, causing slow convergence. (B) Our push-pull mechanism applies binary position selectors to extract the TGT $\to$ SYS and TGT $\to$ IMG blocks from $\bar{A}$ , steering attention routing toward image tokens ( $A_{\text{prefix}}$ : 0.60 $\rightarrow$ 0.06; $A_{\text{img}}$ : 0.16 $\rightarrow$ 0.80) via loss-driven backpropagation without modifying the model’s forward computation.

Method	$\boldsymbol{\epsilon}$	Qwen-VL								LLaVA-1.5								InternVL								MiniGPT-4 †
		AdvBench		StrongREJECT		HarmBench		Jailbreak		AdvBench		StrongREJECT		HarmBench		Jailbreak		AdvBench		StrongREJECT		HarmBench		Jailbreak		AdvBench		StrongREJECT		HarmBench		Jailbreak
		D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G	D	G
Baselines
VAE-JB	32	1.9	68.8	2.9	55.6	1.5	70.0	7.0	61.0	0.6	57.9	1.0	51.8	4.5	76.0	2.0	69.0	0.0	1.0	0.0	3.8	0.0	0.5	0.0	2.0	0.0	75.4	1.0	64.5	1.0	78.5	0.0	72.0
BAP	32	0.0	4.2	0.6	9.3	0.0	14.5	0.0	6.0	0.0	54.8	0.0	47.0	1.5	68.5	1.0	64.0	0.0	0.8	0.0	3.5	0.0	0.5	0.0	1.0	0.4	75.8	0.6	64.3	2.5	75.5	0.0	73.0
JB-Pcs	–	0.0	1.5	0.3	5.8	0.0	7.0	0.0	3.0	0.2	58.7	0.6	50.2	0.5	73.0	2.0	72.0	0.0	0.6	0.0	4.5	0.0	1.0	0.0	0.0	0.8	62.7	1.0	63.9	1.0	74.5	2.0	62.0

Table 1: Attack Success Rate (ASR, %) on Standard Benchmarks across four models. Judges: D = Detoxify, G = Llama Guard 3. White-box: Qwen-VL, LLaVA-1.5, InternVL; transfer: MiniGPT-4 † . Bold : best per column.

Scenario	LLaVA-1.5			Qwen-VL
Scenario	VAE	BAP	Ours	VAE	BAP	Ours
01 Illegal Activ.	43.30	74.23	77.32	40.21	1.03	57.73
02 Hate Speech	24.54	38.65	60.12	28.22	1.23	58.90
03 Malware Gen.	31.82	22.73	52.27	25.00	0.00	50.00
04 Physical Harm	33.33	44.44	56.25	29.86	1.39	54.17
05 Economic Harm	15.57	22.13	27.05	18.85	1.64	41.80

Table 2: Attack Success Rate (ASR, %) on MM-SafetyBench . Bold : best per row.

Configuration	$\alpha$	$\beta$	Adv.	Str.	Harm.	JB.	Avg.
$\mathcal{L}_{\text{target}}$ only	0	0	55.0	47.0	70.5	69.0	60.4
$+\,\mathcal{L}_{\text{suppress}}$	10	0	63.3	70.0	72.0	73.0	69.6
$+\,\mathcal{L}_{\text{anchor}}$	0	5	57.5	44.1	72.5	70.0	61.0
Full (Ours)	10	5	77.5	78.0	84.0	84.0	80.9

Table 3: Component ablation on LLaVA-1.5-7B (

\epsilon{=}32/255

). Suppression and anchoring exhibit synergistic interaction.

Figure 4: Attention Redistribution: System Suppression & Image Amplification. (a) Baseline: Strong system-prompt attention (SYS, red bands) maintains safety. (b) Ours: System attention suppressed by 80%, image attention amplified $4.1\times$ . (c) Difference map (Ours $-$ Baseline): Blue = suppression, red = amplification.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	LLaVA-1.5	Qwen-VL	InternVL2	MiniGPT-4
Vision Encoder	CLIP ViT-L/14	OpenCLIP ViT-G	InternViT-6B	EVA-CLIP ViT-G
Image Resolution	336 $\times$ 336	448 $\times$ 448	448 $\times$ 448	224 $\times$ 224
Image Tokens ( $N_{v}$ )	576	256	$\sim$ 256	32
Projector	2-layer MLP	Cross-attention	MLP Projector	Q-Former + Linear
LLM Backbone	Vicuna-7B	Qwen-7B	InternLM2-7B	Vicuna-7B
# Layers ( $L$ )	32	32	32	32

Appendix

Table 4: Architectural configurations of the evaluated LVLMs.

Figure 5: Comprehensive ablation analysis on LLaVA-1.5-7B. (a) Main comparison: Our method converges to near-zero loss while PGD baseline plateaus. (b) Layer ablation: Targeting last

K{=}8

layers achieves optimal performance. (c) Weight ablation: Both suppression (

\alpha

) and anchoring (

\beta

) components are necessary. (d) Perturbation budget:

\epsilon{=}16/255

balances attack efficacy and imperceptibility. (e) Attention dynamics: System-prompt attention (blue) decreases while image attention (red) increases during optimization.

Layers	Adv.	Str.	Harm.	JB.	Avg.
Last-1	52.9	50.8	61.5	53.0	54.6
Last-2	72.1	71.9	74.5	76.0	73.6
Last-3	72.1	62.0	85.5	80.0	74.9
Last-4	73.8	59.4	81.0	77.0	72.8
Last-6	73.7	57.5	78.5	81.0	72.7
Last-8	81.2	70.0	84.5	82.0	79.4

Appendix

Table 5: Layer depth ablation on LLaVA-1.5-7B (

\epsilon{=}32/255

Panel of Table 6 — Table 6: Average ASR (%) across weight combinations.

$\boldsymbol{\epsilon}$	Adv.	Str.	Harm.	JB.	Avg.
4/255	52.5	46.0	70.5	60.0	57.3
8/255	52.9	45.7	69.5	68.0	59.0
12/255	62.5	50.5	72.0	66.0	62.8
16/255	72.5	64.5	76.5	78.0	72.9
20/255	63.7	57.8	72.0	66.0	64.9
24/255	67.3	57.2	70.0	74.0	67.1

Appendix

Table 7: Perturbation budget ablation on LLaVA-1.5-7B.

Method	TGT $\rightarrow$ SYS	TGT $\rightarrow$ IMG	TGT $\rightarrow$ TXT	ASR
Clean	0.342	0.089	0.156	–
VAE	0.601	0.160	0.149	56.5%
Ours	0.061	0.805	0.144	75.0%
Change from Clean
VAE	$-$ 12.9%	$+$ 25.8%	$-$ 4.5%
Ours	$-$ 82.2%	$+$ 429.2%	$-$ 7.7%

Appendix

Table 8: Regional attention analysis on LLaVA-1.5-7B.

	GPT-4o		Claude-3.5		Gemini-1.5
Benchmark	VAE	Ours	VAE	Ours	VAE	Ours
AdvBench	14.2	18.5	8.5	12.1	16.4	30.3
StrongREJECT	5.8	15.2	3.2	28.4	7.1	38.9
HarmBench	11.5	15.8	7.9	41.5	13.2	58.7
JailbreakBench	8.3	18.6	4.1	36.2	9.8	11.4
Average	10.0	17.0	5.9	29.6	11.6	34.8

Appendix

Table 9: Transfer ASR (%) to closed-source models.

Transform	Parameter	Baseline	Ours
None	original	61.0 $\pm$ 4.9	92.0 $\pm$ 2.7
JPEG	Q=90	5.0 $\pm$ 2.2	6.0 $\pm$ 2.4
JPEG	Q=50	6.0 $\pm$ 2.4	6.0 $\pm$ 2.4
Blur	3 $\times$ 3	8.0 $\pm$ 2.7	9.0 $\pm$ 2.9
Blur	5 $\times$ 5	2.0 $\pm$ 1.4	3.0 $\pm$ 1.7
Resize	0.9 $\times$	7.0 $\pm$ 2.6	10.0 $\pm$ 3.0

Appendix

Table 10: Transformation robustness on Qwen-VL (JailbreakBench). ASR is reported as mean

\pm

std using the main ASR column. Bold indicates the higher ASR in each row.

Figure 6: Adversarial image examples. Top row: original image and adversarial versions at different $\epsilon$ budgets. Bottom row: perturbations magnified 10 $\times$ for visibility. At $\epsilon$ =16/255, perturbations are nearly imperceptible (SSIM=0.990).

$\boldsymbol{\epsilon}$	SSIM $\uparrow$	PSNR $\uparrow$	L ∞	L 2
8/255	0.998	34.2	9.0	4.97
16/255	0.990	28.2	17.0	9.92
16/255(VAE)	0.990	28.2	17.0	9.61
32/255	0.963	22.5	33.0	19.2
32/255(VAE)	0.963	22.7	33.0	19.1

Appendix

Table 11: Perceptual quality of adversarial images.

Defense	ASR $\downarrow$	$\Delta$ ASR	Latency
None (Baseline)	88.0%	–	1.00 $\times$
Monitoring ( $\tau{=}0.15$ )	88.0% ∗	0.0%	1.01 $\times$
Steering ( $b{=}0.5$ )	76.0%	$-$ 12.0%	0.94 $\times$
Steering ( $b{=}1.0$ )	84.0%	$-$ 4.0%	0.83 $\times$
Steering ( $b{=}2.0$ )	26.0%	$-$ 62.0%	$\approx$ 1.0 $\times$ †

Appendix

Table 12: Defense effectiveness on Qwen-VL. ∗ Monitoring detects but does not prevent attacks; ASR reflects successful jailbreaks before filtering. † Steering with high bias reduces output length, affecting latency measurement.

Figure 7: Qualitative examples. Our method responses successful jailbreak into toxic output.

Figure 8: Qualitative examples. Our method responses successful jailbreak into toxic output.

Jailbreak Attacks

Jailbreak attacks exploit prompts, context, or model representations to bypass AI safeguards and elicit disallowed outputs.

See all 73 papers

Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
Feb 16, 2026In Chong Choi, Jiacheng Zhang, Feng Liu +1Adversarial Attacks on VLMs Large Vision-Language Models
The University of Melbourne · The University of Adelaide
InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
Dec 4, 2023Xunguang Wang, Pingchuan Ma, Zhenlan Ji +3Adversarial Attacks on VLMs Large Vision-Language Models
The Hong Kong University of Science and Technology, Hong Kong, China

May 1, 2026J/K move · Enter open · S save

Jailbreaking Vision-Language Models Through the Visual Modality

Aharon Azulay, Jan Dubiński, Zhuoyun Li, Atharv Mittal+1

Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland+3

Adversarial Attacks on VLMs Vision-Language Models VLM Safety Alignment Modality Gap in VLMs Vision-Language Alignment VLM Safety

Open paper AskPDF arXiv

The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb -> banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9% attack success on Claude-Haiku-4.5 versus 10.7% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.

Abstract

Jailbreak Attacks

Explore similar work

Jailbreaking Vision-Language Models Through the Visual Modality

Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models