Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints

Chenxi Li; Xianggan Liu; Dake Shen; Yaosong Du; Zhibo Yao; Hao Jiang; Linyi Jiang; Chengwei Cao

cs.CVMar 8, 2026

Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints

Authors: Chenxi Li, Xianggan Liu, Dake Shen, Yaosong Du, Zhibo Yao, Hao Jiang, Linyi Jiang, Chengwei Cao,

Organizations: DAIL Tech · NLP & KG Lab, Huazhong University of Science and Technology

Abstract

Despite the rapid progress of Large Vision-Language Models (LVLMs), the integration of visual modalities introduces new safety vulnerabilities that adversaries can exploit to elicit biased or malicious outputs. In this paper, we demonstrate an underexplored vulnerability via semantic slot filling, where LVLMs complete missing slot values with unsafe content even when the slot types are deliberately crafted to appear benign. Building on this finding, we propose StructAttack, a simple yet effective single-query jailbreak framework under black-box settings. StructAttack decomposes a harmful query into a central topic and a set of benign-looking slot types, then embeds them as structured visual prompts (e.g., mind maps, tables, or sunburst diagrams) with small random perturbations. Paired with a completion-guided instruction, LVLMs automatically recompose the concealed semantics and generate unsafe outputs without triggering safety mechanisms. Although each slot appears benign in isolation (local benignness), StructAttack exploits LVLMs' reasoning to assemble these slots into coherent harmful semantics. Extensive experiments on multiple models and benchmarks show the efficacy of our proposed StructAttack.

Figures & tables

Figure 2: The framework of our proposed StructAttack . Based on the vulnerability of semantic slot filling, we first employ a Semantic Slot Decomposition module to decompose the target instruction into malicious and distractor slot types. Then, we embed the decomposed slot types into structured visual prompts that can be combined with a completion-guided text prompt to jailbreak LVLMs.

Panel of Figure 3 — Figure 3: Illustration of the (a) zero-shot semantic slot filling (SSF) process in natural language understanding (NLU), that assigns slot types to the given text segments. (b) The textual SSF attack exploits the model’s SSF vulnerability, inducing the LLM to fill in malicious slot types with harmful content. (c) The visual SSF attack extends this paradigm to LVLMs.

Figure 4: Comparison of jailbreak performance using the semantic slot filling vulnerability in textual and visual attacks (e.g., Mind Map, Table, and Sunburst Diagram).

Attack Method	GPT - 4o		Gemini - 2.0 - Flash		Gemini - 2.5 - Flash		Qwen3 - VL - Flash		Qwen2.5VL - 7B		InternVL - 3 - 9B
Attack Method	HF $\uparrow$	ASR% $\uparrow$	HF $\uparrow$	ASR% $\uparrow$	HF $\uparrow$	ASR% $\uparrow$	HF $\uparrow$	ASR% $\uparrow$	HF $\uparrow$	ASR% $\uparrow$	HF $\uparrow$	ASR% $\uparrow$
Advbench - M
Vanilla	0.0	0.0	1.3	0.5	1.3	0.5	1.5	0.9	1.6	1.9	1.3	2.3
FigStep 2023	0.8	0.0	2.8	7.4	1.8	0.9	2.9	3.2	3.0	2.8	2.4	15.7
FigStep - Pro 2023	1.7	10.7	5.8	62.5	2.5	8.3	4.1	48.6	5.0	64.4	5.5	86.1
HADES 2024	1.5	10.2	1.7	5.1	1.8	7.9	2.1	5.1	3.3	22.7	2.1	24.5

Table 1: Jailbreak performance comparison on Advbench - M and SafeBench. Typographic - based methods and OOD - based methods are shaded accordingly. StructAttack v1, v2, and v3 correspond to Mind Map, Table, and Sunburst Diagram, respectively. Best results within each dataset

\times

model column are in bold .

Figure 5: Performance comparison of harmfulness scores across different categories on GPT-4o in Advbench-M.

Figure 6: Case study result on GPT-4o showing that semantic slot–based visual-structural attacks can reliably bypass LVLM safety filters, demonstrating the effectiveness of StructAttack .

Setting	Jailbroken Samples
Setting	BE	D	FW	HI	KS	SV	S	All
Vanilla	0	0	0	0	0	0	0	0
Tree Map	6	7	7	7	9	6	5	47
Mind Map	7	7	9	6	9	5	7	48
Table	5	7	8	8	7	5	5	45
Pyramid Diagram	6	6	8	6	9	7	8	50

Table 2: Ablation study results of different structured visual prompts with identical slot types.

Setting	Jailbroken Samples
Setting	BE	D	FW	HI	KS	SV	S	All
Vanilla	0	0	0	0	0	0	0	0
+SSD	2	4	9	7	7	4	5	38
+VSI	5	5	8	6	9	7	4	44
+Random Perturbation	6	6	7	8	9	5	7	48

Table 3: Ablation study results of the semantic slot decomposition, visual-structural injection, and random perturbation.

Number of Distractor Slot	Jailbroken Samples
Number of Distractor Slot	BE	D	FW	HI	KS	SV	S	All
0 DS	3	6	8	7	6	6	5	41
1 DS	5	7	8	7	7	6	3	44
2 DS	6	6	7	8	9	5	7	48

Table 4: Comparison of jailbroken samples across different number of distractor slot types.

System Prompt Defense	FigStep 2023		FigStep-Pro 2023		HADES 2024		JOOD 2025		SI-Attack 2025		StructAttack
System Prompt Defense	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$
$\times$	0.9	0%	1.5	7.1%	1.4	14.3%	1.8	22.9%	2.6	28.6%	5.3	65.7%
$\checkmark$	0.8	0%	1.2	0%	1.0	0%	1.0	4.3%	1.0	2.9%	3.3	47.2%

Table 5: Effect of attack methods for jailbreaking against the system prompt defense

Method	SI-Attack	JOOD	StructAttack
Iterations per Sample $\downarrow$	10	45	1
ASR $\uparrow$	37.1%	38.6%	65.7 %
Refusal Rate $\downarrow$	28.6%	27.1%	7.1 %

Table 6: Efficiency results of ASR, refusal rate, and iteration numbers per sample compared with OOD-based methods.

Panel of Figure 7 — Figure 7: Visualization of model representations for vanilla inputs and our proposed StructAttack on two open-source models.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Layout Rand.	FigStep-Pro 2024		SI-Attack 2025		StructAttack
	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$	HF $\uparrow$	ASR $\uparrow$
$\times$	1.5	7.1%	2.6	28.6%	4.9	62.9%
$\checkmark$	1.8	8.6%	3.0	32.9%	5.3	67.1%

Appendix

Table 7: Effect analysis of layout random perturbation on GPT-4o.

Evaluator	ASR $\uparrow$	Agreement w/ Human $\uparrow$
Llama-Guard-3-8B (Binary Classification)	67.1%	82.9%
GPT-4o-mini (HF $\geq$ 4)	64.3%	85.7%
GPT-4o-mini (HF $\geq$ 5)	57.1%	84.3%
Human (reference)	61.4%	–

Appendix

Table 8: Consistency analysis on 70 Advbench-M samples.

Method	Victim LVLM		Auxiliary LLM		Total Tokens	Time
Method	queries $\downarrow$	tokens $\downarrow$	queries $\downarrow$	tokens $\downarrow$	(per sample)	(seconds)
JOOD	$45$	$21847\pm 13742$	$0$	$0$	$21847\pm 13742$	$193.44\pm 84.23$
SI-Attack	$10$	$2791\pm!980$	$0$	$0$	$2791\pm 980$	$25.48\pm 10.92$
StructAttack	$1$	$1652\pm 651$	$2$	$568\pm 4$	$2220\pm 655$	$22.79\pm 9.56$

Appendix

Table 9: Per-sample cost analysis of queries, tokens, and time

Words (Per Branch)	BE	D	FW	HI	KS	SV	S	All
100	6.1	5.2	7.3	6.0	4.9	3.3	4.4	5.3
300	6.9	4.5	6.5	7.0	6.6	3.5	4.6	5.6
500	6.6	6.2	6.4	7.0	6.3	2.4	3.5	5.5
1000	5.3	6.0	6.0	6.8	4.3	3.5	4.7	5.2

Appendix

Table 10: Harmfulness score under different word budgets per branch.

Figure 8: A visual structure embedded in a natural scene.

Panel of Figure 9 — Figure 9: Radar plots of ASR on Advbench-M across seven safety categories (BE, D, FW, HI, KS, SV, S) for different jailbreak attacks. Left: Qwen2.5VL-7B. Right: InternVL3-9B.

Panel of Figure 10 — Figure 10: Radar plots of ASR on Advbench-M across seven safety categories (BE, D, FW, HI, KS, SV, S) for different jailbreak attacks. Left: Gemini-2.5-Flash. Right: GPT-4o.

Panel of Figure 11 — Figure 11: Radar plots of ASR on SafeBench across seven safety categories (IA, HS, MG, PH, F, AC, PV) for different jailbreak attacks. Left: Qwen2.5VL-7B. Right: Qwen3-VL-Flash.

Panel of Figure 12 — Figure 12: Radar plots of ASR on SafeBench across seven safety categories (IA, HS, MG, PH, F, AC, PV) for different jailbreak attacks. Left: Gemini-2.5-Flash. Right: GPT-4o.

Figure 13: Jailbreak case result on GPT-4o.

Figure 14: Jailbreak case result on Gemini-2.0-Flash.

Figure 15: Jailbreak case result on Gemini-2.5-Flash.

Figure 16: Jailbreak case result on Qwen3VL-Flash.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 179 papers

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
Feb 16, 2026In Chong Choi, Jiacheng Zhang, Feng Liu +1Adversarial Attacks on VLMs Large Vision-Language Models
The University of Melbourne · The University of Adelaide

Apr 11, 2026J/K move · Enter open · S save

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Jingru Li, Wei Ren, Tianqing Zhu

China University of Geosciences, Wuhan · City University of Macau

Jailbreak Attacks Adversarial Attacks on VLMs VLM Safety Large Vision-Language Models Attention-Based Adversarial Attacks VLM Safety Alignment+1

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) rely on attention-based retrieval of safety instructions to maintain alignment during generation. Existing attacks typically optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model's safety-retrieval mechanism. We propose Attention-Guided Visual Jailbreaking, which circumvents rather than overpowers safety alignment by directly manipulating attention patterns. Our method introduces two simple auxiliary objectives: (1) suppressing attention to alignment-relevant prefix tokens and (2) anchoring generation on adversarial image features. This simple yet effective push-pull formulation reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. At tighter perturbation budgets (

ε=8/255

), we maintain 59.0% ASR compared to 45.7% for standard methods. Mechanistic analysis reveals a failure mode we term safety blindness: successful attacks suppress system-prompt attention by 80%, causing models to generate harmful content not by overriding safety rules, but by failing to retrieve them.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Jailbreaking Vision-Language Models Through the Visual Modality

Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models