Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey

Bhavuk Jain; Sercan Ö. Arık; Hardeo K. Thakur

cs.CRMar 30, 2026

Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey

Authors: Bhavuk Jain, Sercan Ö. Arık, Hardeo K. Thakur

Organizations: Google · Bennett University, India

Abstract

Multimodal large language models (MLLMs) integrate information from multiple modalities such as text, images, audio, and video, enabling complex capabilities such as visual question answering and audio translation. While powerful, this increased expressiveness introduces new and amplified vulnerabilities to adversarial manipulation. This survey provides a comprehensive and systematic analysis of adversarial threats to MLLMs, moving beyond enumerating attack techniques to explain the underlying causes of model susceptibility. We introduce a taxonomy that organizes adversarial attacks according to attacker objectives, unifying diverse attack surfaces across modalities and deployment settings. Additionally, we also present a vulnerability-centric analysis that links integrity attacks, safety and jailbreak failures, control and instruction hijacking, and training-time poisoning to shared architectural and representational weaknesses in multimodal systems. Together, this framework provides an explanatory foundation for understanding adversarial behavior in MLLMs and informs the development of more robust and secure multimodal language systems.

Figures & tables

Figure 1: A high-level overview of the adversarial attack landscape for Multimodal LLMs.

Panel of Figure 2 — Figure 2: Hierarchical taxonomy of attacks on multimodal LLMs.

Panel of Figure 3 — Figure 3: Representative attacks: visual perturbations, indirect prompt injection via images, visual safety bypasses, and inaudible audio commands.

Figure 4: A backdoor attack where the model is ’poisoned’ during training.

	Attack Goal
Attacker Knowledge	Integrity Attack	Safety Attack	Control Attack	Training Attack
White box	( Cui et al., 2024 ) , ( Zhang et al., 2025d )	( Qi et al., 2023 ) , ( Wang et al., 2024a ) , ( Hao et al., 2024 )	( Fu et al., 2023 ) , ( Bailey et al., 2024 )	( Lyu et al., 2024 ) , ( Xu et al., 2024 ) , ( Liang et al., 2024 )
Grey box	( Zhao et al., 2023 )	( Geng et al., 2025 ) , ( Shayegani et al., 2023a )	( Wu et al., 2025 )	( Xu et al., 2024 ) , ( Liu and Zhang, 2025 )
Black box	( Zhao et al., 2023 ) , ( Xie et al., 2024 )	( Gong et al., 2025 ) , ( Jeong et al., 2025 ) , ( Wang et al., 2025d )	( Clusmann et al., 2025 ) , ( Kimura et al., 2024 )	( Xu et al., 2024 ) , ( Liang et al., 2025 )

Table 1: Representative techniques for each attack objective based on the attacker’s knowledge level.

Figure 5: Hierarchical Overview of LLM Vulnerabilities

Taxonomy Category	Primary Vulnerabilities Exploited	Representative Works
Signal Perturbations	Specific component weaknesses	Zhang et al. (2025d)
	Visual Input Processing; Embedding Space Weaknesses	Wang et al. (2024b)
Discrete Triggers	Visual Input Processing; Embedding Space Weaknesses; Universal Representation Vulnerability	Liu et al. (2024a)
	Visual Input Processing; Misalignment & Integration Failures	Cao et al. (2025)
Representation & Fusion Exploits	Embedding Space Weaknesses	Bagdasaryan et al. (2024)
	Misalignment & Integration Failures	Shayegani et al. (2023a)

Table 2: Mapping Integrity Attacks to underlying vulnerabilities drivers

Taxonomy Category	Primary Vulnerabilities Exploited	Representative Works
Unimodal Surfaces	Visual Input Processing; Jailbreaking & Safety Bypass	Qi et al. (2023)
	Audio Input Processing; Asymmetric Vulnerability Across Modalities	Yang et al. (2024) Roh et al. (2025)
Multimodal Composite Jailbreaks	Misalignment & Integration Failures; Embedding Space Weaknesses	Gong et al. (2025)
	Context Manipulation; Misalignment & Integration Failures	Wang et al. (2025d)
	Temporal Input Processing; Attention Mechanism Exploitation	Hu et al. (2025b)
	Jailbreaking & Safety Bypass; Misalignment & Integration Failures;	Li et al. (2025b) Wang et al. (2025b)

Table 3: Mapping Safety/Jailbreak Attacks to underlying vulnerability drivers

Taxonomy Category	Primary Vulnerabilities Exploited	Representative Works
Prompt Injection	Context Manipulation; Exploiting Instruction Following Nature; Visual + Textual Input Processing	Clusmann et al. (2025)
	Audio Input Processing; Exploiting Instruction-Following Nature; Context Manipulation	Hou et al. (2025)
System Instruction Manipulation	Exploiting Instruction Following Nature; Context Manipulation; Misalignment & Integration Failures	Wang et al. (2025a)
	Misalignment & Integration Failures; Over-Reliance on Specific Modalities	Zhang et al. (2025c)
Tool, Retrieval and Agentic Injection	Specific Component Weakness; Context Manipulation; Visual Input Processing	Fu et al. (2023)
	Context Manipulation; Exploiting Instruction-Following Nature; Misalignment & Integration Failures	Zhang et al. (2025a)

Table 4: Mapping Control & Injection Attacks to underlying vulnerability drivers.

Taxonomy Category	Primary Vulnerabilities Exploited	Representative Works
Data Poisoning	Inherited Vulnerabilities from pre-trained components;	Xu et al. (2024)
Backdoor Attacks	Data & Training Induced; Textual Input Processing	Yuan et al. (2025)
Fine-tuning, Adapter and Representation Poisoning	Inherited Vulnerabilites from pre-trained components; Specific Component Weaknesses;	Liu and Zhang (2025)

Table 5: Mapping Poisoning & Backdoor Attacks to underlying vulnerability drivers

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Paper	Target Modalities	Attacker Knowledge	Impact on Performance
( Dong et al., 2023 )	Image	Black-box	ASR: Bard 22%, Bing 26%, GPT-4V 45%, ERNIE 86%
( Qraitem et al., 2024 )	Image + Text	Black-box	Accuracy drop up to 60%
( Liu et al., 2024a )	Image + Text	Black-box	Semantic similarity score up to 0.879
( Cheng et al., 2024 )	Image + Text	Black-box	Performance drop up to 42%
( Qraitem et al., 2025 )	Image + Text	Black-box	ASR up to 100% (artifact ensembles)
( Cao et al., 2025 )	Image + Text	Black-box	ASR: 44% (MCQ), 62% (open-ended)

Appendix

Table 6: Overview of adversarial attacks on multimodal large language models

Adversarial Attacks on Unified Multimodal Models

See all 23 papers

Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
Jul 27, 2026Xi Li, Shu Zhao, Xiaohan Zou +6MLLM Safety Adversarial Attacks on VLMs
University of Alabama at Birmingham · NVIDIA · Penn State University +4
Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models
Mar 5, 2026Yuanbo Li, Tianyang Xu, Cong Hu +3Adversarial Attacks on Unified Multimodal Models Transfer-Based Adversarial Attacks
School of Artificial Intelligence and Computer Science, Jiangnan University · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey
Investigating Adversarial Robustness of Multi-modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanAdversarial Training VLM Robustness
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia

Jul 27, 2026J/K move · Enter open · S save

Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

Xi Li, Shu Zhao, Xiaohan Zou, Fei Zhao+5

University of Alabama at Birmingham · NVIDIA · Penn State University+4

MLLM Safety Adversarial Attacks on VLMs Multimodal Large Language Models LLM Safety

Open paper AskPDF arXiv

Multi-modal large language models (MLLMs) integrate heterogeneous modalities through modality alignment and fusion, enabling stronger understanding and reasoning. However, this architectural shift reshapes the safety landscape of machine learning. Increased model complexity and cross-modal interactions give rise to novel threats, including compromised modality integration, modality misalignment, and fused safety risks, reflecting shifts in threat modeling beyond uni-modal assumptions. These shifts, in turn, impose new constraints on safety solutions not captured by existing frameworks rooted in uni-modal learning. Motivated by these challenges, this survey provides a systematic analysis of the evolving safety landscape of MLLMs. We first propose a multimodal grounded taxonomy of safety threats and analyze shifts in threat models, covering adversarial attacks, data poisoning, jailbreaks, and hallucinations. We then summarize updated safety assumptions and organize recent advances in MLLM safety strategies accordingly. Finally, we discuss open challenges and future directions to inform the development of more principled and scalable safety mechanisms for multimodal systems.

Abstract

Adversarial Attacks on Unified Multimodal Models

Explore similar work

Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models

Investigating Adversarial Robustness of Multi-modal Large Language Models