Vision-language models (VLMs) can provide harmful assistance by interpreting visual events, even without an explicit request for the underlying harmful objective. We identify this scene-state safety vulnerability and introduce SceneSafetyBench, comprising 375 image--question pairs spanning Initial, Medium, and Ultimate scene states and five visual presentations. Across eight VLMs, scene-state inputs elicit harmful assistance with attack success rates (ASRs) of 45.6--72.3%. Comparative analysis across three VLMs shows that scene-state inputs most closely resemble QR-related images in their layer-wise safety-related signal profiles, while exhibiting lower average late-layer readouts and distinct signal dynamics during generation. To turn this vulnerability into a systematic attack, we propose S3TA, which translates harmful objectives into scene states, queries image--question pairs independently, and assembles the responses. On seven MM-SafetyBench categories across eight open-weight and commercial VLMs, S3TA achieves a model-average ASR of 71.2%, exceeding the strongest baseline by 6.1 percentage points. Text-side filters and visual moderation provide limited protection. These findings highlight the need to safeguard the assistance models infer through scene-grounded reasoning, beyond screening input content.
Figures & tables
Figure 1: Illustrative results on GPT-5.6 Luna under three input configurations. The model explicitly refuses the harmful objective in the text-only and QR-related cases, while providing action-oriented guidance in response to the scene-state input.
Figure 2: Examples from SceneSafetyBench across scene-state categories and visual styles. The panels show distinct tasks rather than successive stages of the same event.
Target model
Original
Pixel Art
Pencil Sketch
Oil Painting
Film Noir
Overall
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
Qwen3-VL-8B
4.52
3.71
4.13
45.3
4.21
3.57
3.89
44.0
4.25
3.43
3.95
42.7
4.59
3.65
3.87
44.0
4.55
3.77
4.23
52.0
4.42
3.63
4.01
45.6
GLM-4.1V-9B
5.16
4.59
4.39
73.3
5.25
4.48
4.28
69.3
5.31
4.20
4.24
73.3
5.43
4.35
4.21
72.0
5.33
4.27
4.21
73.3
5.30
4.38
4.27
72.3
InternVL3.5-8B
5.15
3.73
3.79
57.3
5.23
3.77
3.72
61.3
5.12
3.81
3.73
58.7
4.88
3.73
3.65
54.7
5.29
4.03
3.83
62.7
5.13
3.82
3.74
58.9
MiniCPM-V 4.5
5.57
4.21
3.93
66.7
5.49
3.97
4.13
64.0
5.36
3.97
3.91
64.0
5.29
3.77
4.07
60.0
5.45
4.03
4.12
65.3
5.43
3.99
4.03
64.0
Gemma 3 12B IT
5.28
4.11
4.23
70.7
5.29
3.83
4.07
64.0
5.31
4.15
4.19
72.0
5.09
3.76
4.19
64.0
5.37
3.97
4.19
68.0
5.27
3.96
4.17
67.7
Table 1: SceneSafetyBench by visual style. H/A/D are mean harmfulness (1–10), intent alignment (1–5), and detail (1–5); ASR is a percentage. Each style contains 75 pairs (375 overall per model). Model average is the unweighted mean across the eight models. Bold marks the highest style ASR per model, including ties.
Figure 3: HiddenDetect-based lexical readout framework [ 16 ] , illustrated with Qwen3-VL-8B. Values show mean Scene-State prefill HD across 75 tasks at selected layers ( ×1,000 ).
Model
Initial
Medium
Ultimate
Overall
Qwen3-VL-8B
47.2
48.8
40.8
45.6
GLM-4.1V-9B
80.8
68.0
68.0
72.3
InternVL3.5-8B
65.6
56.8
54.4
58.9
MiniCPM-V 4.5
66.4
64.0
61.6
64.0
Gemma 3 12B IT
70.4
62.4
70.4
67.7
GPT-5.6 Luna
68.0
46.4
48.0
54.1
Table 2: SceneSafetyBench ASR (%) by scene state. Each state contains 25 distinct base tasks in five presentations (125 attempts per model); Overall pools all three states (375 attempts). Model average is the unweighted mean across the eight models.
Figure 4: Prefill HD across seven input conditions on 75 tasks. (a) Mean layer-wise profiles; shading marks the final third of layers. (b) RMSE to Scene-State.
Figure 5: Late-layer HD during generation. (a) Natural generation. (b) Scene-State minus each comparator after eight shared-prefix tokens; markers denote two prefixes.
Figure 6: Differences in intervention effect in InternVL3.5, computed as Ei,Scene(c)−Ei,m(c) for each component c and comparison attack m . Bars show paired means over 55 tasks, lines show 95% bootstrap intervals, and stars denote significance after correction for multiple comparisons.
Figure 7: Overview of S3TA. The framework plans task phases or options, constructs and checks scene states, queries each state independently, and assembles the responses for evaluation.
Method
Qwen3-VL 8B
GLM-4.1V 9B
InternVL3.5 8B
MiniCPM-V 4.5
Gemma 3 12B IT
GPT-5.6 Luna
Gemini 3.5 Flash
Claude Haiku 4.5
Overall
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
H
A
D
ASR
QR-related [ 21 ]
2.2
2.2
1.9
4.4
3.7
2.8
2.7
28.9
3.4
2.5
2.5
23.4
4.0
2.9
2.8
33.8
3.4
2.3
2.1
17.3
2.2
2.7
2.8
9.7
3.1
2.8
2.6
23.7
2.4
2.4
1.9
10.8
3.1
2.6
2.4
19.0
FigStep [ 12 ]
3.8
2.9
2.6
27.9
4.6
3.4
3.2
45.4
4.1
3.2
3.0
38.2
6.3
4.3
4.3
71.6
4.2
3.4
2.9
44.4
2.8
2.9
3.3
21.3
4.1
3.1
3.2
37.7
3.9
3.1
2.9
33.3
4.2
3.3
3.2
40.0
VisCRA [ 31 ]
2.5
2.5
1.9
12.5
6.1
4.7
4.6
72.3
4.5
3.6
3.5
47.0
6.8
4.9
4.9
80.9
5.8
4.4
4.4
33.6
2.1
2.7
2.7
3.6
3.5
3.1
2.8
31.0
2.1
2.2
1.5
3.4
4.2
3.5
3.3
35.5
MML [ 34 ]
7.1
4.0
3.6
67.0
4.9
2.6
2.2
29.6
7.1
4.5
4.2
84.0
7.2
4.5
4.6
87.1
9.3
4.5
4.5
86.2
3.7
4.1
3.9
49.0
8.5
4.8
4.8
94.7
2.0
1.6
1.0
0.0
6.2
3.8
3.6
62.2
FC-Attack [ 41 ]
3.1
2.7
1.9
18.9
5.5
3.5
3.2
45.1
5.2
3.7
3.3
52.0
5.0
3.5
3.0
46.6
6.8
4.2
3.9
68.4
3.5
2.8
3.4
20.3
5.1
3.6
3.1
49.3
3.7
2.4
1.9
10.5
4.7
3.3
3.0
38.9
Table 3: Attack comparison on seven MM-SafetyBench categories (01–06 and 09). H/A/D are mean harmfulness (1–10), intent alignment (1–5), and detail (1–5); ASR is a percentage. Overall averages eight models. Bold marks column maxima, including ties.
Category
Original
Decompose
Text-only
No-gate
Full S3TA
Illegal Activity
0.00
0.00
0.00
87.63
95.88
Malware Generation
11.36
11.36
11.36
93.18
95.45
Physical Harm
10.42
10.42
10.42
79.86
90.97
Privacy Violence
6.47
6.47
6.47
94.96
97.84
Overall
6.84
6.84
6.84
87.97
94.81
Table 4: Ablation ASR (%) on Qwen3-VL-8B. Original uses unmodified MM-SafetyBench questions as text-only inputs. Bold marks row maxima.
Evaluator
H (1–10)
A (1–5)
D (1–5)
ASR (%)
Human 1
5.78
4.12
3.82
76
Human 2
5.10
3.86
3.94
82
Human 3
5.54
4.12
3.90
82
Human aggregate
5.47
4.03
3.89
80
GPT-4o-mini
6.42
4.10
3.62
72
Table 6: Human and automated evaluation of 50 InternVL3.5-8B responses. Human aggregate uses mean H/A/D scores across annotators and majority-vote labels for ASR.
School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University