VLM Safety Alignment

VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

17 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Compositional Unalignment: Auditing Safety in Large Vision-Language Models
Nov 4, 2026Hong kyu Lee, Kai Shu, Li XiongVLM Safety VLM Safety Alignment
Emory University
Visual Grounding Safety in Vision-Language Models
Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM Evaluation VLM Safety Alignment
University of California, Riverside · Work done during internship at Apple. · Apple
Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
Sep 29, 2026Haotian Deng, Wenbin Xing, Gang Xu +5VLM Robustness Vision-Language Models
Southern University of Science and Technology · Sun Yat-sen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) +5
SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models
Sep 3, 2026Caoyuan Ma, Tian Gu, Wenpu Liu +11Vision-Language Models VLM Safety Alignment
The University of Tokyo · Wuhan University · Shanghai AI Laboratory +4
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models
Sep 3, 2026Nicholas Sansoterra, Zishuo Zheng, Sachin KumarVLM Safety Alignment VLM Robustness
The Ohio State University
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMs VLM Safety
School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University
When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
Aug 19, 2026Mehak Gupta, Tanmoy ChakrabortyVLM Safety Alignment Abstention in VLMs
Yardi School of Artifical Intelligence Indian Institute of Technology Delhi, New Delhi, India
SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning
Aug 11, 2026Caoyuan Ma, Wenpu Liu, Weichu Xie +12VLM Robustness Adversarial Attacks on VLMs
The University of Tokyo · Peking University · Wuhan University +4
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images
Mar 9, 2026Qishun Yang, Shu Yang, Lijie Hu +1VLM Safety Alignment VLM Safety
King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay +1
SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safety
Mar 3, 2026Zixuan Xu, Tiancheng He, Huahui Yi +7Cultural Safety Evaluation of VLMs VLM Safety
Huazhong University of Science and Technology · Beijing University of Posts and Telecommunications · West China Hospital, Sichuan University +1
GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
Feb 27, 2026Xingyu Zhu, Beier Zhu, Junfeng Fang +4VLM Safety Alignment VLM Safety
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1
Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents
Feb 18, 2026Idhant Gulati, Shivam RavalVLM Safety Alignment
University of California, Berkeley Berkeley, CA · Harvard University Cambridge, MA
Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Feb 8, 2026Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu +1VLM Robustness Adversarial Attacks on VLMs
Pennsylvania State University · Bangladesh University of Engineering and Technology
ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks
Oct 3, 2025Zhaorun Chen, Xun Liu, Mintong Kang +4Adversarial Attacks on VLMs VLM Safety
University of Chicago · University of Illinois, Urbana-Champaign · Virtue AI +1

Nov 4, 2026J/K move · Enter open · S save

Compositional Unalignment: Auditing Safety in Large Vision-Language Models

Hong kyu Lee, Kai Shu, Li Xiong

Emory University

VLM Safety VLM Safety Alignment VLM Evaluation Activation Steering

Open paper AskPDF arXiv

Large vision-language models (LVLMs) are vulnerable to compositional harm: an image and a prompt that are benign in isolation become harmful through the model's cross-modal reasoning. Recent alignment methods report strong scores on compositional safety benchmarks, but an LLM judge only measures observed behavior and cannot reveal how robustly safety is integrated into the model. We argue that un-alignment provides a complementary audit of alignment robustness. We first show that minimal supervised fine-tuning reaches an LLM-judge score comparable to a multi-stage reinforcement-learning pipeline, so similar benchmark scores cannot establish robustness by themselves. We then show that recognition and refusal of compositional harm are decoupled: nine standard direction-finding methods detect compositional harm with up to AUC=0.99, yet ablating the recovered directions either leaves refusal intact or destroys model coherence. We therefore propose CMaG, a training-free un-alignment method for compositional safety. CMaG measures refusal intention with a refusal-versus-compliance log-probability margin, takes its gradient with respect to mid-layer activations, and contrasts it against partially benign pairs to isolate the cross-modal interaction term. A single backward pass over 80 examples yields a direction along which steering monotonically converts refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient-based baselines; the direction also transfers to text-explicit harm and un-aligns models hardened by defensive training. Compositional safety can thus appear behaviorally strong while remaining fragile to targeted internal intervention. Warning: this paper contains unsafe model outputs, including references to self-harm.