Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored. As VLMs are increasingly deployed in real environments, this gap becomes critical, since adversarial perturbations must be physically realizable. Despite this practical relevance, physical attacks against VLMs have not been systematically studied. Such attacks may induce recognition failures and further disrupt multimodal reasoning, leading to severe semantic misinterpretation in downstream tasks. Therefore, investigating physical attacks on VLMs is essential for assessing their real-world security risks. To address this gap, we propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs. MSLA uses controllable adversarial lighting to disrupt multimodal semantic understanding in real scenes, attacking semantic alignment rather than only task-specific outputs. Consequently, it degrades zero-shot classification performance of mainstream CLIP variants while inducing severe semantic hallucinations in advanced VLMs such as LLaVA and BLIP across image captioning and visual question answering (VQA). Extensive experiments in both digital and physical domains demonstrate that MSLA is effective, transferable, and practically realizable. Our findings provide the first evidence that VLMs are highly vulnerable to physically deployable semantic attacks, exposing a previously overlooked robustness gap and underscoring the urgent need for physical-world robustness evaluation of VLMs.
Figures & tables
Figure 1. Overall framework of MSLA. Given a clean image, we parameterize the triangular light by its relative position, radius, color, and three polar angles, and optimize these variables with a genetic algorithm. The optimized triangular light can be deployed using simple optical devices such as a flashlight, colored transparent sheets, and shape templates. An overview of the proposed MSLA framework.
Figure 2. Experimental devices. The physical equipment used to deploy the MSLA.
Method
OpenCLIP ViT-B/16
Meta-CLIP ViT-L/14
EVA-CLIP ViT-G/14
OpenAI CLIP ViT-L/14
Clean
97
98
98
93
Natural Light Attack ( Hsiao et al., 2024 )
94 ( ↓3 )
97 ( ↓1 )
97 ( ↓1 )
93(-)
Shadow Attack ( Zhong et al., 2022 )
84 ( ↓13 )
82 ( ↓16 )
95 ( ↓3 )
79 ( ↓14 )
ITA ( Liu et al., 2025b )
46 ( ↓51 )
64 ( ↓34 )
84 ( ↓14 )
51 ( ↓42 )
MSLA (ours)
29 ( ↓68 )
23 ( ↓75 )
36 ( ↓62 )
11 ( ↓82 )
Table 1. Performance Degradation ( ↓ ) of VLMs on Zero-shot Classification Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the accuracy (%) of existing illumination-based attack methods with our proposed method across four CLIP variants. Lower accuracy indicates stronger attack performance. The best result is shown in bold.
Image Encoder
Models
#Params
Clean
Natural Light Attack ( Hsiao et al., 2024 )
Shadow Attack ( Zhong et al., 2022 )
ITA ( Liu et al., 2025b )
MSLA (ours)
OpenAI CLIP ViT-L/14
LLaVA-1.5
7B
78.60
77.00( ↓1.63 )
74.60( ↓4.00 )
63.73( ↓14.87 )
56.23( ↓22.37 )
LLaVA-1.6
7B
72.10
71.70( ↓0.39 )
71.17( ↓0.93 )
61.60( ↓10.51 )
56.33( ↓15.77 )
OpenFlamingo
3B
70.20
69.53( ↓0.67 )
67.80( ↓2.40 )
53.93( ↓16.27 )
41.73( ↓28.47 )
Blip-2 (FlanT5 XL ViT-L)
3.4B
75.10
70.77( ↓4.33 )
68.57( ↓6.53 )
60.93( ↓14.17 )
53.88( ↓21.22 )
EVA-CLIP ViT-G/14
Blip-2 (FlanT5 XL )
4.1B
74.96
71.27( ↓3.69 )
68.80( ↓6.17 )
62.01( ↓11.88 )
49.89( ↓25.07 )
InstructBLIP (FlanT5 XL )
4.1B
76.50
72.07( ↓4.43 )
69.77( ↓6.73 )
63.20( ↓13.30 )
59.43( ↓17.07 )
Table 2. Performance Degradation ( ↓ ) of VLMs on Image Captioning Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the consistency (%) evaluated by GPT-4 of previous methods with our proposed approach. Lower values indicate better attack performance. The best result is shown in bold.
Image Encoder
Models
#Params
Clean
Natural Light Attack ( Hsiao et al., 2024 )
Shadow Attack ( Zhong et al., 2022 )
ITA ( Liu et al., 2025b )
MSLA (ours)
OpenAI CLIP ViT-L/14
LLaVA-1.5
7B
68.00
68.00(-)
67.00( ↓1.00 )
48.00( ↓20.00 )
24.00( ↓44.00 )
LLaVA-1.6
7B
64.00
63.00( ↓1.00 )
64.00(-)
43.00( ↓21.00 )
26.00( ↓38.00 )
OpenFlamingo
3B
45.00
39.00( ↓6.00 )
44.00( ↓1.00 )
19.00( ↓26.00 )
3.00( ↓42.00 )
Blip-2 (FlanT5 XL ViT-L)
3.4B
63.00
58.00( ↓5.00 )
50.00( ↓13.00 )
38.00( ↓25.00 )
6.00( ↓57.00 )
EVA-CLIP ViT-G/14
Blip-2 (FlanT5 XL )
4.1B
54.00
53.00( ↓1.00 )
54.00(-)
33.00( ↓21.00 )
10.00( ↓44.00 )
InstructBLIP (FlanT5 XL )
4.1B
68.00
64.00( ↓4.00 )
62.00( ↓6.00 )
44.00( ↓24.00 )
15.00( ↓53.00 )
Table 3. Performance Degradation ( ↓ ) of VLMs on VQA Task Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the correctness (%) evaluated by GPT-4 of previous methods with our proposed approach. The performance is considered better with lower values. Number in bold indicates the best performance.
Figure 3. Visualization Results of Zero-shot Classification. The first row shows clean samples with their corresponding Top-1 labels, while the second row presents the MSLA samples with misclassified labels. Representative adversarial examples generated by the proposed MSLA in the digital setting.
Figure 4. Visualization of some digital MSLA samples causing LVLMs to give incorrect answers. In each LVLM response, we highlight the correct and error parts using different colors. Representative adversarial examples showing incorrect outputs generated by vision-language models on image captioning and VQA tasks under the proposed MSLA.
Figure 5. Attention analysis of CLIP models before and after our proposed method in the digital setting. Attention heatmaps of different CLIP variants on clean and adversarial samples, showing how the proposed MSLA shifts or diffuses model attention.
Figure 6. Visualization of physical MSLA samples causing LVLMs to give incorrect answers. In each LVLM response, we highlight the correct and error parts using different colors. Representative key frames from adversarial videos in the physical world, showing distorted captions and incorrect VQA answers generated by vision-language models under the proposed MSLA.
Figure 7. Physical MSLA samples for Zero-shot Classification with the corresponding frame-level ASR. Representative frames extracted from adversarial videos in the physical-world zero-shot classification setting, showing successful attacks and corresponding prediction changes.
Figure 8. Attention analysis of CLIP models before and after our proposed method in the physical setting. Attention heatmaps of aligned frames from clean and adversarial videos, showing changes in visual attention under the MSLA in the physical world.
Figure 9. Attack performance under different transparency parameter α with varying radius scaling factors γ on four CLIP variants. Results of the ablation study showing attack performance under different transparency values and radius scaling factors.
CLIP variants
Lprob(Θ)
Llog(Θ)
Lmulti(Θ) (ours)
OpenCLIP ViT-B/16
44( ↓53 )
31( ↓66 )
29( ↓68 )
Meta-CLIP ViT-L/14
40( ↓58 )
28( ↓70 )
23( ↓75 )
EVA-CLIP ViT-G/14
53( ↓45 )
39( ↓59 )
36( ↓62 )
OpenAI CLIP ViT-L/14
16( ↓77 )
13( ↓80 )
11( ↓82 )
Table 4. Performance Degradation ( ↓ ) of CLIP variants on Zero-shot Classification under different fitness functions. Lower accuracy (%) indicates stronger attack performance. The best result in each row is shown in bold.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.
Ilan Zini, Boussad Addad, Katarzyna Kapusta
ESILV student and Thales intern · Thales, Thales cortAIx Labs, France
Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.
Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2
Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan
Vision-language models (VLMs) are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems, where perception or reasoning errors can lead to unsafe decisions or actions. Although many red-teaming methods have been developed to probe VLM vulnerabilities, their evaluation remains fragmented across datasets, metrics, and threat models, making direct comparison difficult and obscuring whether observed differences arise from stronger attacks, more vulnerable models, or incompatible evaluation settings. Existing chatbot-centric red-teaming benchmarks mainly standardize jailbreak and content-safety evaluation, but they do not systematically capture physically grounded functional failures or cover red-teaming methods that target physical-world VLMs. This raises the key challenge of comparing diverse attack methods under a unified protocol while targeting the same scenario-specific failures. We introduce REALM, to our knowledge the first unified red-teaming benchmark for physical-world VLMs. REALM integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model with shared datasets and metrics. To align adversarial objectives across attack families, REALM introduces an agentic target-generation pipeline that constructs shared, scenario-specific, and physically grounded attack objectives for each scene, enabling fair comparison of diverse red-teaming methods under aligned adversarial goals. Our evaluation shows that text and typographic injection attacks induce the most failures, multimodal co-optimization yields the strongest visual-perturbation transfer, single-pass attacks approach iterative methods at much lower cost, and model scale alone does not confer adversarial robustness. Code is available at https://github.com/UCF-ML-Research/REALM.