Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

David Fernandez; Pedram MohajerAnsari; Amir Salarpour; Long Cheng; Abolfazl Razi; Mert D. Pesé

cs.CVMar 9, 2026

Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures

Authors: David Fernandez, Pedram MohajerAnsari, Amir Salarpour, Long Cheng, Abolfazl Razi, Mert D. Pesé

Organizations: School of Computing, Clemson University, Clemson, SC, USA

Abstract

Vision-language models are emerging for autonomous driving, yet their robustness to physical adversarial attacks remains unexplored. This paper presents a systematic framework for comparative adversarial evaluation across three VLM architectures: Dolphins, OmniDrive (Omni-L), and LeapVAD. Using black-box optimization with semantic homogenization for fair comparison, we evaluate physically realizable patch attacks in CARLA simulation. Results reveal severe vulnerabilities across all architectures, sustained multi-frame failures, and critical object detection degradation. Our analysis exposes distinct architectural vulnerability patterns, demonstrating that current VLM designs inadequately address adversarial threats in safety-critical autonomous driving applications.

Figures & tables

Fig. 1: Overview of Attack Framework: (1) Black-box NES optimization generates adversarial patches; (2) CARLA scenarios capture frame video sequences; (3) VLM architectures process scenes: Dolphins , OmniDrive (Omni-L) , and LeapVAD ; (4) Semantic homogenization layer projects all VLM outputs into a unified embedding space; (5) Multi-dimensional evaluation.

Panel of Figure 2 — Fig. 2: Scenario 1:Bus Shelter Crosswalk Attack temporal sequence. The attack scenario demonstrates how the adversarial patch suppresses pedestrian detection as the ego vehicle approaches bus ad shelter.

Model	Crosswalk ASR (%)	Highway ASR (%)	Combined ASR (%)	Baseline (%)	Significance
Dolphins	$73.1\pm 2.8$	$79.2\pm 3.1$	$76.0\pm 2.4$	3.8	$p<0.001$
OmniDrive (Omni - L)	$71.8\pm 3.2$	$75.6\pm 2.9$	$73.5\pm 2.6$	5.1	$p<0.001$
LeapVAD	$68.4\pm 2.5$	$81.7\pm 2.7$	$75.0\pm 3.1$	6.3	$p<0.001$

TABLE I: Overall attack success rates across VLM architectures. Attack success is defined as the proportion of frames where the VLM recommends the target unsafe action rather than the appropriate safety-critical action.

Panel of Figure 4 — Fig. 3: Distance-dependent attack success rates across VLM architectures for both scenarios. The shaded red region indicates the critical decision-making range. Omni-L (purple) exhibits highest vulnerability across distances, while LeapVAD (orange) shows superior robustness, particularly at close ranges where its explicit critical object attention provides maximal benefit.

Panel of Figure 5 — Fig. 4: Temporal attack persistence across VLM architectures. Each row shows representative trials for crosswalk and highway scenarios. Models demonstrate similar temporal consistency patterns with LeapVAD showing slightly longer attack persistence (7.8 $\pm$ 1.4 frames highway) compared to OmniDrive (6.9 $\pm$ 1.3 frames).

	Crosswalk (Pedestrian)					Highway (Barrier)
Model	Benign	Adv.	Degrad.	BLEU-4	Sem.	Benign	Adv.	Degrad.	BLEU-4	Sem.
	DR (%)	DR (%)	(pp)		Sim.	DR (%)	DR (%)	(pp)		Sim.
Dolphins	92.3	21.2	$-71.1$	0.18	0.49	88.5	47.9	$-40.6$	0.24	0.59
OmniDrive (Omni - L)	89.7	34.8	$-54.9$	0.22	0.54	91.2	58.3	$-32.9$	0.28	0.63
LeapVAD	94.6	48.2	$-46.4$	0.26	0.58	87.4	52.1	$-35.3$	0.31	0.67

TABLE II: Detection rates and scene description quality metrics across VLM architectures. Detection degradation is reported in percentage points (pp). BLEU-4 and semantic similarity scores compare adversarial vs. benign descriptions.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis
Apr 30, 2026David Fernandez, Pedram MohajerAnsari, Amir Salarpour +1Adversarial Attacks on VLMs Vision-Language Models
School of Computing, Clemson University, USA
Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Driving
Oct 6, 2026Heyam Bin Jahlan Areej Alhothali Abeer AlhothaliAdversarial Attacks on VLMs VLMs for Autonomous Driving
Technical Report on the CVPR 2026@AdvML Workshop Challenge
Jul 13, 2026Tianyuan Zhang, Zonglei Jing, Jiangfan Liu +47VLMs for Autonomous Driving Adversarial Attacks on VLA Models

Apr 30, 2026J/K move · Enter open · S save

Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

David Fernandez, Pedram MohajerAnsari, Amir Salarpour, Mert D. Pese

School of Computing, Clemson University, USA

Adversarial Attacks on VLMs Vision-Language Models VLMs for Autonomous Driving Autonomous Driving Transfer-Based Adversarial Attacks Adversarial Transferability+1

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across different VLM architectures, is not well understood and poses a practical risk when attackers do not know which model a vehicle uses. We address this gap with a systematic cross-architecture study of adversarial transferability in VLM-based driving, evaluating three representative architectures (Dolphins, OmniDrive, and LeapVAD) using physically realizable patches placed on roadside infrastructure in both crosswalk and highway scenarios. Our transfer-matrix evaluation shows high cross-architecture effectiveness, with transfer rates of 73-91% (mean TR = 0.815 for crosswalk and 0.833 for highway) and sustained frame-level manipulation over 64.7-79.4% of the critical decision window even when patches are not optimized for the target model.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Driving

Technical Report on the CVPR 2026@AdvML Workshop Challenge