Adversarial attacks against Modern Vision-Language Models

Alejandro Paredes La Torre

cs.CRMar 17, 2026

Adversarial attacks against Modern Vision-Language Models

Authors: Alejandro Paredes La Torre

Organizations: Duke University

Abstract

We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.

Figures & tables

Figure 1: Overview of the adversarial red-teaming pipeline. An adversarially perturbed product image is served through a Flask storefront, captured as a screenshot by a Selenium agent, and passed to the VLM inference server. The VLM returns a structured action that causes the agent to purchase the adversarially targeted product rather than the intended item.

Method	LLaVA-v1.5-7B	Qwen2.5-VL-7B
Clean Baseline	90.2 $\pm$ 2.3	98.3 $\pm$ 1.0
BIM	47.4 $\pm$ 3.9 (ASR 52.6)	93.5 $\pm$ 1.9 (ASR 6.5)
PGD	46.2 $\pm$ 3.9 (ASR 53.8)	92.3 $\pm$ 2.1 (ASR 7.7)
CLIP Spectral	33.1 $\pm$ 3.7 (ASR 66.9)	84.5 $\pm$ 2.8 (ASR 15.5)

Table 1: Correct purchase rate (%) and ASR (%) with 95% CIs across 630 trials per condition. All attacks use

\epsilon=16/255

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Comparison of the original product image and its BIM-perturbed adversarial counterpart. The perturbation is bounded by $\|\delta\|_{\infty}\leq\epsilon$ and is visually imperceptible, yet sufficient to fool the LLaVA-based web agent.

Panel of Figure 3 — Figure 3: LLaVA Shopping Web Agent initialization. The agent receives a natural language command and begins autonomous browser navigation with a configurable iteration budget.

Panel of Figure 4 — Figure 4: Storefront viewed by the LLaVA agent during the attack. The adversarially perturbed sweatshirt (item 0, $7,400) occupies the first product slot. Despite the user intent pointing to a different item type, the perturbation manipulates the agent’s perception.

Panel of Figure 5 — Figure 5: Raw LLaVA model outputs and parsed actions for both attack scenarios. The agent’s reasoning is shown verbatim, demonstrating how the adversarial perturbation causes it to misidentify or mis-navigate to the wrong product.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 179 papers

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM Robustness Adversarial Robustness
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Apr 1, 2026Jingning Xu, Haochen Luo, Chen LiuVLM Robustness Vision-Language Models
Department of Computer Science City University of Hong Kong
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau

Mar 13, 2026J/K move · Enter open · S save

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

Sen Nie, Jie Zhang, Zhongqi Wang, Zhaoyang Wei+2

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China

VLM Robustness Adversarial Robustness Adversarial Attacks on VLMs VLM Adaptation

Open paper AskPDF arXiv

Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.

Abstract

Adversarial Attacks on VLMs

Explore similar work

What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking