Directional Embedding Smoothing for Robust Vision Language Models

Ye Wang; Jing Liu; Toshiaki Koike-Akino

cs.LGMar 16, 2026

Directional Embedding Smoothing for Robust Vision Language Models

Authors: Ye Wang, Jing Liu, Toshiaki Koike-Akino

Organizations: Mitsubishi Electric Research Laboratories, Cambridge, MA 02139, USA

Abstract

The safety and reliability of vision-language models (VLMs) are a crucial part of deploying trustworthy agentic AI systems. However, VLMs remain vulnerable to jailbreaking attacks that undermine their safety alignment to yield harmful outputs. In this work, we extend the Randomized Embedding Smoothing and Token Aggregation (RESTA) defense to VLMs and evaluate its performance against the JailBreakV-28K benchmark of multi-modal jailbreaking attacks. We find that RESTA is effective in reducing attack success rate over this diverse corpus of attacks, in particular, when employing directional embedding noise, where the injected noise is aligned with the original token embedding vectors. Our results demonstrate that RESTA can contribute to securing VLMs within agentic systems, as a lightweight, inference-time defense layer of an overall security framework.

Figures & tables

Panel of Figure 1 — Figure 1: Safety-utility tradeoff results for RESTA defense applied on LLaVa and Gemma VLMs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Hard noise		Normal
Noise $\sigma$	SciQA (%)	ASR (%)	SciQA (%)	ASR (%)
0	64.07	50.13	64.07	50.13
0.001	64.21	49.91	64.04	50.20
0.002	63.85	49.26	64.02	50.10
0.003	63.55	47.08	—	—
0.004	61.97	37.64	—	—

Appendix

Table 1: Safety-utility tradeoff results for RESTA defense applied on LLaVa-1.5-7B.

	Hard noise		Normal
Noise $\sigma$	SciQA (%)	ASR (%)	SciQA (%)	ASR (%)
0	71.42	56.31	71.42	56.31
0.05	71.23	55.99	71.16	55.97
0.1	71.09	56.04	71.09	56.06
0.2	71.42	56.14	71.47	56.20
0.3	71.40	56.10	71.73	55.97

Appendix

Table 2: Safety-utility tradeoff results for RESTA defense applied on Gemma-3-4B.

Explore similar work

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
Mar 23, 2026Xingyu Zhu, Beier Zhu, Shuo Wang +4VLM Robustness Activation Steering
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore · Nanyang Technological University
Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift
Mar 18, 2026Zhihua Wei, Qiang Li, Jian Ruan +4Adversarial Attacks on VLMs VLM Robustness
School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3

Mar 23, 2026J/K move · Enter open · S save

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models

Xingyu Zhu, Beier Zhu, Shuo Wang, Junfeng Fang+3

MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore · Nanyang Technological University

VLM Robustness Activation Steering Adversarial Attacks on VLMs

Open paper AskPDF arXiv

As vision-language models (VLMs) are increasingly deployed in open-world scenarios, they can be easily induced by visual jailbreak attacks to generate harmful content, posing serious risks to model safety and trustworthy usage. Recent activation steering methods inject directional vectors into model activations during inference to induce refusal behaviors and have demonstrated effectiveness. However, a steering vector may both enhance refusal ability and cause over-refusal, thereby degrading model performance on benign inputs. Moreover, due to the lack of theoretical interpretability, these methods still suffer from limited robustness and effectiveness. To better balance safety and utility, we propose NullSteer, a null-space projected activation defense framework. Our method constructs refusal directions within model activations through a linear transformation: it maintains zero perturbation within the benign subspace while dynamically inducing refusal along potentially harmful directions, thereby theoretically achieving safety enhancement without impairing the model's general capabilities. Extensive experiments show that NullSteer significantly reduces harmful outputs under various jailbreak attacks (average ASR reduction over 15 percent on MiniGPT-4) while maintaining comparable performance to the original model on general benchmarks.

Abstract

Explore similar work

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models

Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

Jailbreaking Vision-Language Models Through the Visual Modality