VLM Safety Alignment

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

15 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Visual Grounding Safety in Vision-Language Models
Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM Evaluation VLM Safety Alignment
University of California, Riverside · Work done during internship at Apple. · Apple
Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
Sep 29, 2026Haotian Deng, Wenbin Xing, Gang Xu +5VLM Robustness Vision-Language Models
Southern University of Science and Technology · Sun Yat-sen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) +5
SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models
Sep 3, 2026Caoyuan Ma, Tian Gu, Wenpu Liu +11Vision-Language Models VLM Safety Alignment
The University of Tokyo · Wuhan University · Shanghai AI Laboratory +4
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models
Sep 3, 2026Nicholas Sansoterra, Zishuo Zheng, Sachin KumarVLM Safety Alignment VLM Robustness
The Ohio State University
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMs VLM Safety
School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
Aug 19, 2026Mehak Gupta, Tanmoy ChakrabortyVLM Safety Alignment Abstention in VLMs
Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India
SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning
Aug 11, 2026Caoyuan Ma, Wenpu Liu, Weichu Xie +12VLM Robustness Adversarial Attacks on VLMs
The University of Tokyo · Peking University · Wuhan University +4
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images
Mar 9, 2026Qishun Yang, Shu Yang, Lijie Hu +1VLM Safety Alignment VLM Safety
King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay +1
SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safety
Mar 3, 2026Zixuan Xu, Tiancheng He, Huahui Yi +7Cultural Safety Evaluation of VLMs VLM Safety
Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Hospital, Sichuan University +1
GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
Feb 27, 2026Xingyu Zhu, Beier Zhu, Junfeng Fang +4VLM Safety Alignment VLM Safety
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1
Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents
Feb 18, 2026Idhant Gulati, Shivam RavalVLM Safety Alignment
University of California, Berkeley Berkeley, CA · Harvard University Cambridge, MA
Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Feb 8, 2026Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu +1VLM Robustness Adversarial Attacks on VLMs
Pennsylvania State University · Bangladesh University of Engineering and Technology

Oct 5, 2026J/K move · Enter open · S save

Visual Grounding Safety in Vision-Language Models

Erfan Shayegani, Kundan Krishna, Yue Dong, Nael Abu-Ghazaleh+2

University of California, Riverside · Work done during internship at Apple. · Apple

VLM Evaluation VLM Safety Alignment Vision-Language Grounding Visual Grounding VLM Safety

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.