cs.CVNov 16, 2026

SmoothShift: Disrupting Visual Adversarial Attacks on LVLMs at Test Time

Authors: Xiaoqing Chen, Jiayi Ouyang, Dongrui Wu

Organizations: Huazhong University of Science and Technology · Zhongguancun Academy

Abstract

Visual adversarial perturbations can substantially impair large vision-language models (LVLMs) despite introducing only subtle input changes. To understand how these small changes cause substantial errors, we investigate their optimization and propagation through the model, revealing two key findings. First, stronger attacks rely on finer iterative optimization, while subtle input perturbations induce representation discrepancies that are amplified layer by layer through the vision encoder and persist in the language model. Second, disrupting the alignment between within-patch input patterns and the model's computation reduces clean-adversarial representation discrepancies and substantially weakens attacks. Together, these findings suggest that adversarial effectiveness depends on both fine-grained perturbation patterns and their coupling with the model's computation. Motivated by these findings, we propose SmoothShift, a lightweight, training-free defense that combines local smoothing and spatial transformations. Smoothing weakens fine-grained perturbation patterns, while spatial transformations disrupt their alignment with the encoder's patch grid. SmoothShift requires no model updates, auxiliary networks, or additional model inference. Experiments cover three LVLM families on four datasets and robustly fine-tuned Contrastive Language-Image Pre-training (CLIP) models on eight classification datasets. Evaluations across eight attack methods and multiple perturbation budgets demonstrate improved robustness while largely preserving clean performance. SmoothShift also provides robustness gains under defense-aware attacks using mixed-gradient BPDA with EOT.

Figures & tables

Explore similar work

CardsList
  1. PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

    Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM RobustnessVision-Language Models

  2. Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

    Dec 11, 2024Hee-Seon Kim, Minbeom Kim, Seokil Ham +1VLM RobustnessAdversarial Attacks on VLMs

  3. VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense

    Feb 23, 2026Nadav Kadvil, Malak Fares, Ayellet TalVLM RobustnessLarge Vision-Language Models