Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

Yingying Zhao; Chengyin Hu; Qike Zhang; Xin Li; Xin Wang; Yiwei Wei; Jiujiang Guo; Jiahuan Long

cs.CVApr 14, 2026

Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

Authors: Yingying Zhao, Chengyin Hu, Qike Zhang, Xin Li, Xin Wang, Yiwei Wei, Jiujiang Guo, Jiahuan Long,

Abstract

Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored. As VLMs are increasingly deployed in real environments, this gap becomes critical, since adversarial perturbations must be physically realizable. Despite this practical relevance, physical attacks against VLMs have not been systematically studied. Such attacks may induce recognition failures and further disrupt multimodal reasoning, leading to severe semantic misinterpretation in downstream tasks. Therefore, investigating physical attacks on VLMs is essential for assessing their real-world security risks. To address this gap, we propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs. MSLA uses controllable adversarial lighting to disrupt multimodal semantic understanding in real scenes, attacking semantic alignment rather than only task-specific outputs. Consequently, it degrades zero-shot classification performance of mainstream CLIP variants while inducing severe semantic hallucinations in advanced VLMs such as LLaVA and BLIP across image captioning and visual question answering (VQA). Extensive experiments in both digital and physical domains demonstrate that MSLA is effective, transferable, and practically realizable. Our findings provide the first evidence that VLMs are highly vulnerable to physically deployable semantic attacks, exposing a previously overlooked robustness gap and underscoring the urgent need for physical-world robustness evaluation of VLMs.

Figures & tables

Figure 1. Overall framework of MSLA. Given a clean image, we parameterize the triangular light by its relative position, radius, color, and three polar angles, and optimize these variables with a genetic algorithm. The optimized triangular light can be deployed using simple optical devices such as a flashlight, colored transparent sheets, and shape templates. An overview of the proposed MSLA framework.

Figure 2. Experimental devices. The physical equipment used to deploy the MSLA.

Method	OpenCLIP ViT-B/16	Meta-CLIP ViT-L/14	EVA-CLIP ViT-G/14	OpenAI CLIP ViT-L/14
Clean	97	98	98	93
Natural Light Attack ( Hsiao et al., 2024 )	94 ( $\downarrow 3$ )	97 ( $\downarrow 1$ )	97 ( $\downarrow 1$ )	93(-)
Shadow Attack ( Zhong et al., 2022 )	84 ( $\downarrow 13$ )	82 ( $\downarrow 16$ )	95 ( $\downarrow 3$ )	79 ( $\downarrow 14$ )
ITA ( Liu et al., 2025b )	46 ( $\downarrow 51$ )	64 ( $\downarrow 34$ )	84 ( $\downarrow 14$ )	51 ( $\downarrow 42$ )
MSLA (ours)	29 ( $\downarrow 68$ )	23 ( $\downarrow 75$ )	36 ( $\downarrow 62$ )	11 ( $\downarrow 82$ )

Table 1. Performance Degradation (

\downarrow

) of VLMs on Zero-shot Classification Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the accuracy (%) of existing illumination-based attack methods with our proposed method across four CLIP variants. Lower accuracy indicates stronger attack performance. The best result is shown in bold.

Image Encoder	Models	#Params	Clean	Natural Light Attack ( Hsiao et al., 2024 )	Shadow Attack ( Zhong et al., 2022 )	ITA ( Liu et al., 2025b )	MSLA (ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	78.60	77.00( $\downarrow 1.63$ )	74.60( $\downarrow 4.00$ )	63.73( $\downarrow 14.87$ )	56.23( $\downarrow 22.37$ )
	LLaVA-1.6	7B	72.10	71.70( $\downarrow 0.39$ )	71.17( $\downarrow 0.93$ )	61.60( $\downarrow 10.51$ )	56.33( $\downarrow 15.77$ )
	OpenFlamingo	3B	70.20	69.53( $\downarrow 0.67$ )	67.80( $\downarrow 2.40$ )	53.93( $\downarrow 16.27$ )	41.73( $\downarrow 28.47$ )
	Blip-2 (FlanT5 XL ViT-L)	3.4B	75.10	70.77( $\downarrow 4.33$ )	68.57( $\downarrow 6.53$ )	60.93( $\downarrow 14.17$ )	53.88( $\downarrow 21.22$ )
EVA-CLIP ViT-G/14	Blip-2 (FlanT5 XL )	4.1B	74.96	71.27( $\downarrow 3.69$ )	68.80( $\downarrow 6.17$ )	62.01( $\downarrow 11.88$ )	49.89( $\downarrow 25.07$ )
EVA-CLIP ViT-G/14	InstructBLIP (FlanT5 XL )	4.1B	76.50	72.07( $\downarrow 4.43$ )	69.77( $\downarrow 6.73$ )	63.20( $\downarrow 13.30$ )	59.43( $\downarrow 17.07$ )

Table 2. Performance Degradation (

\downarrow

) of VLMs on Image Captioning Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the consistency (%) evaluated by GPT-4 of previous methods with our proposed approach. Lower values indicate better attack performance. The best result is shown in bold.

Image Encoder	Models	#Params	Clean	Natural Light Attack ( Hsiao et al., 2024 )	Shadow Attack ( Zhong et al., 2022 )	ITA ( Liu et al., 2025b )	MSLA (ours)
OpenAI CLIP ViT-L/14	LLaVA-1.5	7B	68.00	68.00(-)	67.00( $\downarrow 1.00$ )	48.00( $\downarrow 20.00$ )	24.00( $\downarrow 44.00$ )
	LLaVA-1.6	7B	64.00	63.00( $\downarrow 1.00$ )	64.00(-)	43.00( $\downarrow 21.00$ )	26.00( $\downarrow 38.00$ )
	OpenFlamingo	3B	45.00	39.00( $\downarrow 6.00$ )	44.00( $\downarrow 1.00$ )	19.00( $\downarrow 26.00$ )	3.00( $\downarrow 42.00$ )
	Blip-2 (FlanT5 XL ViT-L)	3.4B	63.00	58.00( $\downarrow 5.00$ )	50.00( $\downarrow 13.00$ )	38.00( $\downarrow 25.00$ )	6.00( $\downarrow 57.00$ )
EVA-CLIP ViT-G/14	Blip-2 (FlanT5 XL )	4.1B	54.00	53.00( $\downarrow 1.00$ )	54.00(-)	33.00( $\downarrow 21.00$ )	10.00( $\downarrow 44.00$ )
EVA-CLIP ViT-G/14	InstructBLIP (FlanT5 XL )	4.1B	68.00	64.00( $\downarrow 4.00$ )	62.00( $\downarrow 6.00$ )	44.00( $\downarrow 24.00$ )	15.00( $\downarrow 53.00$ )

Table 3. Performance Degradation (

\downarrow

) of VLMs on VQA Task Under Multimodal Semantic Lighting Attacks Examples Generated from the COCO Dataset. We compare the correctness (%) evaluated by GPT-4 of previous methods with our proposed approach. The performance is considered better with lower values. Number in bold indicates the best performance.

Figure 3. Visualization Results of Zero-shot Classification. The first row shows clean samples with their corresponding Top-1 labels, while the second row presents the MSLA samples with misclassified labels. Representative adversarial examples generated by the proposed MSLA in the digital setting.

Figure 4. Visualization of some digital MSLA samples causing LVLMs to give incorrect answers. In each LVLM response, we highlight the correct and error parts using different colors. Representative adversarial examples showing incorrect outputs generated by vision-language models on image captioning and VQA tasks under the proposed MSLA.

Figure 5. Attention analysis of CLIP models before and after our proposed method in the digital setting. Attention heatmaps of different CLIP variants on clean and adversarial samples, showing how the proposed MSLA shifts or diffuses model attention.

Figure 6. Visualization of physical MSLA samples causing LVLMs to give incorrect answers. In each LVLM response, we highlight the correct and error parts using different colors. Representative key frames from adversarial videos in the physical world, showing distorted captions and incorrect VQA answers generated by vision-language models under the proposed MSLA.

Figure 7. Physical MSLA samples for Zero-shot Classification with the corresponding frame-level ASR. Representative frames extracted from adversarial videos in the physical-world zero-shot classification setting, showing successful attacks and corresponding prediction changes.

Figure 8. Attention analysis of CLIP models before and after our proposed method in the physical setting. Attention heatmaps of aligned frames from clean and adversarial videos, showing changes in visual attention under the MSLA in the physical world.

Figure 9. Attack performance under different transparency parameter $\alpha$ with varying radius scaling factors $\gamma$ on four CLIP variants. Results of the ablation study showing attack performance under different transparency values and radius scaling factors.

CLIP variants	$\mathcal{L}_{\mathrm{prob}}(\Theta)$	$\mathcal{L}_{\mathrm{log}}(\Theta)$	$\mathcal{L}_{\mathrm{multi}}(\Theta)$ (ours)
OpenCLIP ViT-B/16	44( $\downarrow 53$ )	31( $\downarrow 66$ )	29( $\downarrow 68$ )
Meta-CLIP ViT-L/14	40( $\downarrow 58$ )	28( $\downarrow 70$ )	23( $\downarrow 75$ )
EVA-CLIP ViT-G/14	53( $\downarrow 45$ )	39( $\downarrow 59$ )	36( $\downarrow 62$ )
OpenAI CLIP ViT-L/14	16( $\downarrow 77$ )	13( $\downarrow 80$ )	11( $\downarrow 82$ )

Table 4. Performance Degradation (

\downarrow

) of CLIP variants on Zero-shot Classification under different fitness functions. Lower accuracy (%) indicates stronger attack performance. The best result in each row is shown in bold.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Breaking the weakest link to evade vision language models
Aug 19, 2026Ilan Zini, Boussad Addad, Katarzyna KapustaVLM Robustness Adversarial Attacks on VLMs
ESILV student and Thales intern · Thales, Thales cortAIx Labs, France
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
Apr 6, 2026Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2VLM Robustness VLM Evaluation
Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan
REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs
Jun 22, 2026Yifei Zhao, Qian Lou, Mengxin ZhengVLM Evaluation VLM Robustness
University of Central Florida

Aug 19, 2026J/K move · Enter open · S save

Breaking the weakest link to evade vision language models

Ilan Zini, Boussad Addad, Katarzyna Kapusta

ESILV student and Thales intern · Thales, Thales cortAIx Labs, France

VLM Robustness Adversarial Attacks on VLMs Adversarial Attacks

Open paper AskPDF arXiv

Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Breaking the weakest link to evade vision language models

Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs