cs.CVNov 17, 2026

Reading Scene States: Uncovering Safety Failures in Vision-Language Models

Authors: Tongxi Wu, Yuyang Cheng, Jiacheng Xian, Menglin Yang, Kai Chen, Jian Zhang, Yang Gao

Organizations: Nanjing University · The Hong Kong University of Science and Technology (Guangzhou) · Independent Researcher

Abstract

Vision-language models (VLMs) can provide harmful assistance by interpreting visual events, even without an explicit request for the underlying harmful objective. We identify this scene-state safety vulnerability and introduce SceneSafetyBench, comprising 375 image--question pairs spanning Initial, Medium, and Ultimate scene states and five visual presentations. Across eight VLMs, scene-state inputs elicit harmful assistance with attack success rates (ASRs) of 45.6--72.3%. Comparative analysis across three VLMs shows that scene-state inputs most closely resemble QR-related images in their layer-wise safety-related signal profiles, while exhibiting lower average late-layer readouts and distinct signal dynamics during generation. To turn this vulnerability into a systematic attack, we propose S3TA, which translates harmful objectives into scene states, queries image--question pairs independently, and assembles the responses. On seven MM-SafetyBench categories across eight open-weight and commercial VLMs, S3TA achieves a model-average ASR of 71.2%, exceeding the strongest baseline by 6.1 percentage points. Text-side filters and visual moderation provide limited protection. These findings highlight the need to safeguard the assistance models infer through scene-grounded reasoning, beyond screening input content.

Figures & tables

Explore similar work

CardsList
  1. Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

    Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak AttacksAdversarial Attacks on VLMs

  2. EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

    Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMsVLM Safety

  3. Scenes as an Attack Surface: Uncovering Scene-Grounded Safety Failures in MLLMs

    Oct 8, 2026Tongxi Wu, Yuyang Cheng, Jiacheng Xian +5MLLM SafetyAdversarial Attacks on VLMs