Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs
Authors: Tongxi Wu, Yuyang Cheng, Jiacheng Xian, Hu Huang, Zhihao Wu, Menglin Yang, Jian Zhang, Yang Gao
Organizations: Nanjing University · The Hong Kong University of Science and Technology (Guangzhou) · Independent Researcher · China Academy of Launch Vehicle Technology · Beihang University
Existing jailbreak attacks have exposed serious safety weaknesses in multimodal large language models (MLLMs). Most prior attacks rely on specially constructed visual inputs to carry or support harmful intent. In this paper, we uncover a different vulnerability: task-relevant scene semantics alone can shift an MLLM from refusal to compliance, even when the visual input introduces no additional attack instructions and the original task intent remains unchanged. We formalize this failure as a scene-grounded visual safety vulnerability and introduce SceneSafetyBench to investigate it across eight risk categories and nine white- and black-box MLLMs. Our analysis reveals that this failure is associated with how task-relevant visual evidence influences downstream safety representations, suggesting that visual grounding alone cannot fully explain the observed safety shift. Building on this finding, we propose Scene-Grounded Attacks (SGA), which preserve the original task while constructing validated scene-grounded queries and combining their independent responses. Across nine target MLLMs, SGA achieves an average attack success rate (ASR) of 86.17%, outperforming the strongest baseline by 8.55 percentage points and achieving the highest ASR on eight models. Our results identify scene semantics as an overlooked attack surface in MLLMs and highlight a growing tension between stronger visual grounding and robust multimodal safety alignment.
Figures & tables
Figure 1: Representative examples of scene-grounded safety failures in ChatGPT 5.6. It refuses the original harmful request and an intent-preserving textual rewrite, but exhibits different safety behavior after the task is grounded in a semantically relevant visual scene.
Figure 2: Representative examples of scene-grounded safety failures. Both models refuse the original harmful request and an intent-preserving textual rewrite, but exhibit different safety behavior after the task is grounded in a semantically relevant visual scene.
No.
Safety Category
# Image–Question Pairs
1
Illegal Activity
87
2
Hate Speech
30
3
Malware Generation
110
4
Physical Harm
78
5
Economic Harm
36
6
Fraud
28
Table 1: Composition of SceneSafetyBench across its eight safety categories.
Model
Condition
ASR
VSFR
95% CI
Qwen3-VL
Original text (A)
4.0%
–
–
Changed text (B)
0.0%
–
–
Item (C)
40.0%
40.0%
[27.6, 53.8]
Multi-item (D)
66.0%
66.0%
[52.2, 77.6]
Scene (E)
76.0%
76.0%
[62.6, 85.7]
LLaVA
Original text (A)
50.0%
–
–
Table 2: Behavioral characterization under increasingly structured visual context. ASR denotes the overall compliance rate under each condition. VSFR denotes the fraction of samples refused under the changed-request text baseline (B) that flip to compliance under the corresponding visual condition. Bold p -values indicate p<0.05 .
Figure 3: Representative TAM visualizations across increasingly structured visual conditions. The top example is from LLaVA-v1.5-13B and the bottom example is from Qwen3-VL-8B-Instruct.
Figure 4: Overview of the scene-grounded attack ( SGA ) pipeline. Given an original request, the framework constructs a semantic blueprint and decomposes it into task-related sub-questions. A proxy model specifies and validates the required visual evidence, while an image generator instantiates and iteratively refines the corresponding scenes. Uncovered semantic requirements are supplemented before the target model processes each validated image–question pair independently. The resulting local responses are then synthesized and evaluated against the original task.
Target model
Illegal Activity
Hate Speech
Malware Generation
Physical Harm
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
Qwen3-VL-8B-Instruct
5.70
4.45
4.26
79.31
5.20
4.50
3.97
86.67
6.58
4.55
3.95
76.36
5.67
4.37
4.26
78.21
GLM-4.1V-9B-Thinking
6.80
4.75
4.72
91.95
5.93
4.83
4.60
93.33
7.38
4.96
4.68
99.09
6.41
4.86
4.59
96.15
InternVL3.5-8B-Instruct
6.99
4.63
4.43
88.51
4.47
3.63
3.10
63.33
7.15
4.76
3.82
92.73
6.14
4.18
3.90
78.21
MiniCPM-V 4.5
7.24
4.72
4.62
94.25
5.37
4.43
4.03
80.00
7.41
4.95
4.33
100.00
6.65
4.87
4.28
93.59
Phi-4-reasoning-vision-15B
2.54
2.18
1.34
10.34
3.13
2.43
1.83
16.67
4.30
3.04
1.94
25.45
2.79
2.24
1.24
5.13
Table 3: Category-wise results on SceneSafetyBench across nine target models. H is reported on a 1–10 scale, A and D on a 1–5 scale, and ASR as a percentage. The Overall column aggregates all benchmark instances rather than averaging the eight category-level values, while the final row reports the arithmetic mean across models. Boldface marks the highest ASR in each category and Overall, including ties.
Attack method
Qwen3-VL-8B-Instruct
GLM-4.1V-9B-Thinking
InternVL3.5-8B-Instruct
MiniCPM-V 4.5
Phi-4-reasoning-vision-15B
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
QR-Related
2.04
2.21
1.90
4.46
3.64
3.34
3.15
36.52
2.97
2.76
2.69
25.82
4.30
3.60
3.56
45.37
2.14
2.52
2.18
5.66
VisCRA
2.48
2.54
1.93
12.45
6.08
4.73
4.67
75.31
4.45
3.59
3.49
47.02
6.76
4.90
4.90
80.86
2.02
2.06
1.05
0.62
FigStep
3.83
2.98
2.47
28.81
4.56
3.44
3.19
45.42
4.18
3.12
3.13
39.81
6.25
4.34
4.33
71.60
2.33
2.16
1.24
4.94
MML-WR
7.21
4.52
4.10
84.05
7.26
4.05
3.39
62.14
7.10
4.63
4.32
86.63
7.14
4.82
4.79
92.28
6.93
4.60
4.02
82.51
MML-M
7.97
4.63
4.43
88.17
8.55
4.63
4.46
87.04
7.25
4.66
4.53
86.11
7.55
4.88
4.91
93.42
7.80
4.75
4.51
89.61
Table 4: End-to-end results for SGA and seven baseline configurations on nine target models. SGA is scored using the final response from the complete pipeline. Each method uses its own attack benchmark, while target-model settings and the judge protocol remain fixed. H is reported on a 1–10 scale, A and D on a 1–5 scale, and ASR as a percentage. Boldface marks the best result for each metric and target model, including ties. The rightmost column group gives the arithmetic mean across models.
Variant
Qwen
GLM
Intern
Claude
Avg.
Full SGA
100.00
98.00
98.00
84.00
95.00
w/o Scene
42.00
28.00
34.00
18.00
30.50
w/o Coverage
94.00
90.00
92.00
82.00
89.50
No Target Evidence
8.00
8.00
8.00
8.00
8.00
Table 5: Ablation study of SGA on four representative target models. We report attack success rate (ASR, %) on the fixed 50-question subset.
Evaluator
H
A
D
ASR
Human 1
5.76
4.12
3.82
74.00
Human 2
5.10
3.84
3.94
80.00
Human 3
5.54
4.12
3.88
80.00
Human aggregate
5.47
4.03
3.88
78.00
GPT-4o-mini judge
6.42
4.10
3.62
72.00
Judge–human agreement
Table 6: Human validation of the GPT-4o-mini-based judge protocol on 50 randomly sampled InternVL3.5-8B-Instruct responses. H , A , and D are mean scores. ASR and agreement values are percentages. Ranges give the minimum and maximum across evaluator comparisons.
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China