Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.
Figures & tables
Figure 1: Targeted adversarial attack using vision encoder embedding alignment.
Figure 2: Targeted adversarial attack forcing a VLM to misinterpret a military tank as an ambulance.
VLM
Qwen
Granite
FastVLM
Phi
VLM Params
3.7B
3B
7.8B
4.1B
VT Params
670M
442M
125M
424M
Ratio
18%
14%
1.6%
10%
Table 1: Comparison of VLM and vision encoder parameter counts.
Figure 3: General architecture of a Vision–Language Model combining a visual encoder and a language model to generate textual outputs from image and text inputs.
epsilon
qwen2.5-3B
granite-3.2-2b
FastVLM
Phi-3.5
0.05
9.2±0.8
41.15±1.9
26.4±1.3
2.0±0.4
0.10
20.9±1.4
47.12±2.1
27.7±1.6
1.6±0.3
0.20
25.0±1.7
45.91±2.4
29.5±1.8
1.6±0.3
Table 2: Targeted attack success rate (%) across VLM models for different perturbation budgets ϵ . Mean ± standard deviation over four runs.
Figure 4: Targeted attack success rate as a function of the perturbation budget ϵ .
epsilon
qwen2.5-3B
granite-3.2-2b
FastVLM
Phi-3.5
0.05
93.7±1.1
99.79±0.08
99.8±0.07
93.0±1.3
0.10
97.7±0.9
99.78±0.07
99.6±0.09
95.7±1.0
0.20
97.5±1.0
99.77±0.06
99.4±0.12
97.1±0.8
Table 3: Untargeted attack success rate (%) across VLM models for different perturbation budgets ϵ . Mean ± standard deviation over four runs.
Figure 5: Untargeted attack success rate as a function of the perturbation budget ϵ .
Figure 6: Prompts used for the LLM-as-a-Judge evaluation to assess semantic similarity or difference between generated image descriptions.
Figure 7: Generated captions on source (left) and adversarial (right) images. We perform a targeted attack on the caption output with ε=0.05 on Granite-Vision-3.2-2B. The perturbations are hardly visible and would not be noticed by a user.
Figure 8: GPU compute and VRAM usage over time for Granite-Vision-3.2-2B on an NVIDIA H100 80 GB. Top: optimization through the full VLM. Bottom: optimization restricted to the vision encoder. Encoder-only optimization substantially reduces both VRAM usage and GPU utilization.
Figure 9: GPU compute and VRAM usage over time for Qwen2.5-VL-3B on an NVIDIA H100 80 GB. Top: optimization through the full VLM. Bottom: optimization restricted to the vision encoder. The full-VLM experiment may require more than 20 minutes without producing a successful adversarial example, whereas the encoder-only approach completes in approximately 100 seconds.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
Large Vision-Language Models (LVLMs) have transformed multi-modal understanding, excelling in tasks like image captioning and visual question answering by integrating visual and textual inputs. However, their robustness against adversarial attacks, particularly those exploiting both modalities, remains underexplored, posing risks to critical applications like autonomous driving and content moderation. Existing attacks focus on single modalities or require impractical white-box access, limiting their real-world relevance. In this paper, we introduce Multi-Modal Adversarial Synergy, a groundbreaking framework that crafts universal, black-box multi-modal attacks against LVLMs. MMAS simultaneously generates a texture scale-constrained universal adversarial perturbation for images and a learnable prompt perturbation for text, optimized jointly using only model queries. The image perturbation leverages wavelet-based texture constraints to ensure imperceptibility and robustness across diverse visual inputs. The text perturbation, constrained by an L-norm in the embedding space, maintains semantic coherence while steering outputs toward a target. A novel cross-modal regularization term aligns the perturbations' gradient directions, enhancing their synergistic impact and transferability across tasks and models. Extensive experiments show the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs.
Xiang Fang, Wanlong Fang, Changshuo Wang
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · University College London
Large Vision-Language Models (LVLMs) have achieved remarkable performance on multimodal tasks but remain highly vulnerable to small adversarial perturbations in input images. Existing attacks typically target the vision encoder's final output embeddings, implicitly treating the encoder as a uniform attack surface, while a systematic analysis of which internal components are most vulnerable has remained largely unexplored. We show such analysis is essential, as adversarial vulnerability in LVLM vision encoders is structurally concentrated rather than uniformly distributed. Building on this, we propose Vision Encoder Vulnerable-Component-Targeted Universal Adversarial Perturbation (VEV-UAP), a task-agnostic and cost-efficient attack framework. Through a component- and layer-wise analysis of attention mechanisms, we identify the value components in middle layers as critical vulnerabilities that strongly influence downstream language model behavior. VEV-UAP selectively targets these components to generate a single universal perturbation shared across images, without involving textual inputs or the language model during optimization. Experiments across multiple LVLMs and tasks show VEV-UAP achieves state-of-the-art attack success rates with reduced computational overhead. Moreover, a single VEV-UAP transfers across LVLMs sharing the same vision encoder, even when paired with different language models, making it a practical framework for scalable robustness evaluation.
Hee-Seon Kim, Minbeom Kim, Seokil Ham +1
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea
While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., Visual supervision branch and Semantic supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.