Reading Scene States: Uncovering Safety Failures in Vision-Language Models
Organizations: Nanjing University · The Hong Kong University of Science and Technology (Guangzhou) · Independent Researcher
Abstract
Vision-language models (VLMs) can provide harmful assistance by interpreting visual events, even without an explicit request for the underlying harmful objective. We identify this scene-state safety vulnerability and introduce SceneSafetyBench, comprising 375 image--question pairs spanning Initial, Medium, and Ultimate scene states and five visual presentations. Across eight VLMs, scene-state inputs elicit harmful assistance with attack success rates (ASRs) of 45.6--72.3%. Comparative analysis across three VLMs shows that scene-state inputs most closely resemble QR-related images in their layer-wise safety-related signal profiles, while exhibiting lower average late-layer readouts and distinct signal dynamics during generation. To turn this vulnerability into a systematic attack, we propose S3TA, which translates harmful objectives into scene states, queries image--question pairs independently, and assembles the responses. On seven MM-SafetyBench categories across eight open-weight and commercial VLMs, S3TA achieves a model-average ASR of 71.2%, exceeding the strongest baseline by 6.1 percentage points. Text-side filters and visual moderation provide limited protection. These findings highlight the need to safeguard the assistance models infer through scene-grounded reasoning, beyond screening input content.
Figures & tables
| Target model | Original | Pixel Art | Pencil Sketch | Oil Painting | Film Noir | Overall | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | |
| Qwen3-VL-8B | 4.52 | 3.71 | 4.13 | 45.3 | 4.21 | 3.57 | 3.89 | 44.0 | 4.25 | 3.43 | 3.95 | 42.7 | 4.59 | 3.65 | 3.87 | 44.0 | 4.55 | 3.77 | 4.23 | 52.0 | 4.42 | 3.63 | 4.01 | 45.6 |
| GLM-4.1V-9B | 5.16 | 4.59 | 4.39 | 73.3 | 5.25 | 4.48 | 4.28 | 69.3 | 5.31 | 4.20 | 4.24 | 73.3 | 5.43 | 4.35 | 4.21 | 72.0 | 5.33 | 4.27 | 4.21 | 73.3 | 5.30 | 4.38 | 4.27 | 72.3 |
| InternVL3.5-8B | 5.15 | 3.73 | 3.79 | 57.3 | 5.23 | 3.77 | 3.72 | 61.3 | 5.12 | 3.81 | 3.73 | 58.7 | 4.88 | 3.73 | 3.65 | 54.7 | 5.29 | 4.03 | 3.83 | 62.7 | 5.13 | 3.82 | 3.74 | 58.9 |
| MiniCPM-V 4.5 | 5.57 | 4.21 | 3.93 | 66.7 | 5.49 | 3.97 | 4.13 | 64.0 | 5.36 | 3.97 | 3.91 | 64.0 | 5.29 | 3.77 | 4.07 | 60.0 | 5.45 | 4.03 | 4.12 | 65.3 | 5.43 | 3.99 | 4.03 | 64.0 |
| Gemma 3 12B IT | 5.28 | 4.11 | 4.23 | 70.7 | 5.29 | 3.83 | 4.07 | 64.0 | 5.31 | 4.15 | 4.19 | 72.0 | 5.09 | 3.76 | 4.19 | 64.0 | 5.37 | 3.97 | 4.19 | 68.0 | 5.27 | 3.96 | 4.17 | 67.7 |
| Model | Initial | Medium | Ultimate | Overall |
|---|---|---|---|---|
| Qwen3-VL-8B | 47.2 | 48.8 | 40.8 | 45.6 |
| GLM-4.1V-9B | 80.8 | 68.0 | 68.0 | 72.3 |
| InternVL3.5-8B | 65.6 | 56.8 | 54.4 | 58.9 |
| MiniCPM-V 4.5 | 66.4 | 64.0 | 61.6 | 64.0 |
| Gemma 3 12B IT | 70.4 | 62.4 | 70.4 | 67.7 |
| GPT-5.6 Luna | 68.0 | 46.4 | 48.0 | 54.1 |
| Method | Qwen3-VL 8B | GLM-4.1V 9B | InternVL3.5 8B | MiniCPM-V 4.5 | Gemma 3 12B IT | GPT-5.6 Luna | Gemini 3.5 Flash | Claude Haiku 4.5 | Overall | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | H | A | D | ASR | |
| QR-related [ 21 ] | 2.2 | 2.2 | 1.9 | 4.4 | 3.7 | 2.8 | 2.7 | 28.9 | 3.4 | 2.5 | 2.5 | 23.4 | 4.0 | 2.9 | 2.8 | 33.8 | 3.4 | 2.3 | 2.1 | 17.3 | 2.2 | 2.7 | 2.8 | 9.7 | 3.1 | 2.8 | 2.6 | 23.7 | 2.4 | 2.4 | 1.9 | 10.8 | 3.1 | 2.6 | 2.4 | 19.0 |
| FigStep [ 12 ] | 3.8 | 2.9 | 2.6 | 27.9 | 4.6 | 3.4 | 3.2 | 45.4 | 4.1 | 3.2 | 3.0 | 38.2 | 6.3 | 4.3 | 4.3 | 71.6 | 4.2 | 3.4 | 2.9 | 44.4 | 2.8 | 2.9 | 3.3 | 21.3 | 4.1 | 3.1 | 3.2 | 37.7 | 3.9 | 3.1 | 2.9 | 33.3 | 4.2 | 3.3 | 3.2 | 40.0 |
| VisCRA [ 31 ] | 2.5 | 2.5 | 1.9 | 12.5 | 6.1 | 4.7 | 4.6 | 72.3 | 4.5 | 3.6 | 3.5 | 47.0 | 6.8 | 4.9 | 4.9 | 80.9 | 5.8 | 4.4 | 4.4 | 33.6 | 2.1 | 2.7 | 2.7 | 3.6 | 3.5 | 3.1 | 2.8 | 31.0 | 2.1 | 2.2 | 1.5 | 3.4 | 4.2 | 3.5 | 3.3 | 35.5 |
| MML [ 34 ] | 7.1 | 4.0 | 3.6 | 67.0 | 4.9 | 2.6 | 2.2 | 29.6 | 7.1 | 4.5 | 4.2 | 84.0 | 7.2 | 4.5 | 4.6 | 87.1 | 9.3 | 4.5 | 4.5 | 86.2 | 3.7 | 4.1 | 3.9 | 49.0 | 8.5 | 4.8 | 4.8 | 94.7 | 2.0 | 1.6 | 1.0 | 0.0 | 6.2 | 3.8 | 3.6 | 62.2 |
| FC-Attack [ 41 ] | 3.1 | 2.7 | 1.9 | 18.9 | 5.5 | 3.5 | 3.2 | 45.1 | 5.2 | 3.7 | 3.3 | 52.0 | 5.0 | 3.5 | 3.0 | 46.6 | 6.8 | 4.2 | 3.9 | 68.4 | 3.5 | 2.8 | 3.4 | 20.3 | 5.1 | 3.6 | 3.1 | 49.3 | 3.7 | 2.4 | 1.9 | 10.5 | 4.7 | 3.3 | 3.0 | 38.9 |
| Category | Original | Decompose | Text-only | No-gate | Full S3TA |
|---|---|---|---|---|---|
| Illegal Activity | 0.00 | 0.00 | 0.00 | 87.63 | 95.88 |
| Malware Generation | 11.36 | 11.36 | 11.36 | 93.18 | 95.45 |
| Physical Harm | 10.42 | 10.42 | 10.42 | 79.86 | 90.97 |
| Privacy Violence | 6.47 | 6.47 | 6.47 | 94.96 | 97.84 |
| Overall | 6.84 | 6.84 | 6.84 | 87.97 | 94.81 |
| Evaluator | H (1–10) | A (1–5) | D (1–5) | ASR (%) |
|---|---|---|---|---|
| Human 1 | 5.78 | 4.12 | 3.82 | 76 |
| Human 2 | 5.10 | 3.86 | 3.94 | 82 |
| Human 3 | 5.54 | 4.12 | 3.90 | 82 |
| Human aggregate | 5.47 | 4.03 | 3.89 | 80 |
| GPT-4o-mini | 6.42 | 4.10 | 3.62 | 72 |