AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective

Zhenyi Wang; Siyu Luan

cs.CRMar 25, 2026

AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective

Authors: Zhenyi Wang, Siyu Luan

Organizations: Department of Computer Science, Institute of Artificial Intelligence, University of Central Florida · University of Copenhagen

Abstract

As machine learning (ML) systems expand in both scale and functionality, the security landscape has become increasingly complex, with a proliferation of attacks and defenses. However, existing studies largely treat these threats in isolation, lacking a coherent framework to expose their shared principles and interdependencies. This fragmented view hinders systematic understanding and limits the design of comprehensive defenses. Crucially, the two foundational assets of ML -- data and models -- are no longer independent; vulnerabilities in one directly compromise the other. The absence of a holistic framework leaves open questions about how these bidirectional risks propagate across the ML pipeline. To address this critical gap, we propose a unified closed-loop threat taxonomy that explicitly frames model-data interactions along four directional axes. Our framework offers a principled lens for analyzing and defending foundation models. The resulting four classes of security threats represent distinct but interrelated categories of attacks: (1) Data $\rightarrow$ Data (D $\rightarrow$ D): including data decryption attacks and watermark removal attacks; (2) Data $\rightarrow$ Model (D $\rightarrow$ M): including poisoning, harmful fine-tuning attacks, and jailbreak attacks; (3) Model $\rightarrow$ Data (M $\rightarrow$ D): including model inversion, membership inference attacks, and training data extraction attacks; (4) Model $\rightarrow$ Model (M $\rightarrow$ M): including model extraction attacks. Our unified framework elucidates the underlying connections among these security threats and establishes a foundation for developing scalable, transferable, and cross-modal security strategies, particularly within the landscape of foundation models.

Figures & tables

Survey	$\mathbf{D\!\rightarrow\!D}$	$\mathbf{D\!\rightarrow\!M}$	$\mathbf{M\!\rightarrow\!D}$	$\mathbf{M\!\rightarrow\!M}$	Loop	FM	Scope and limitations
Training data poisoning survey ( Tian et al., 2022 )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	data poisoning attack
Model inversion attack survey ( Fang et al., 2024 )	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	model inversion attack
Membership inference attack survey ( Hu et al., 2022a )	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	membership inference attack
Harmful fine-tuning attack ( Huang et al., 2024a )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	large language model harmful fine-tuning attack
Jailbreak attack ( Yi et al., 2024b )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	large language model jailbreak attack
Model extraction (stealing) survey ( Oliynyk et al., 2023 )	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	Deep coverage of model extraction and Intellectual Property (IP) theft

Table 1: Comparison with prior surveys and taxonomies. Coverage across closed-loop attack directions and emphasis on foundation-model (FM), closed-loop analysis.

Attack class	Threat-model assumptions	Attacker capabilities required	Typical success metrics	Computational cost
D $\rightarrow$ D	Access to protected data or prompts; goal is to bypass ownership or safety constraints while preserving content or utility.	Ability to apply transformations to inputs (regeneration, rewriting, adversarial prompting, etc); usually no need for victim model weights.	Bypass success rate (e.g., decrypt success; watermark detection failure; jailbreak success), plus fidelity or utility preservation and downstream task performance.	Low to moderate computation cost: input transformation or rewriting; jailbreak often needs iterative prompt search; watermark removal may involve generative regeneration.
D $\rightarrow$ M	Adversary can influence training or fine-tuning data to implant targeted misbehavior or erode alignment.	Inject or modify a subset of training samples to create malicious training data.	Attack success (attack success rate, targeted misclassification, backdoor trigger success); clean utility (clean accuracy, helpfulness); safety metrics (refusal rate drop, unsafe prompt completion rate).	Moderate to high computation cost: requires training or fine-tuning data modification by optimization.
M $\rightarrow$ D	Adversary queries a trained model to recover memorized training information (samples, attributes or membership).	Black-box (query outputs) or white-box (gradients or representations) access; optimization over inputs or prompts.	Data reconstruction quality; membership information accuracy/AUC.	Often high computation cost: many queries and/or iterative optimization; generative extraction can require generative model training.
M $\rightarrow$ M	Adversary aims to replicate a deployed proprietary model via queries (data-free/data-based/architecture cloning).	Query access to victim; ability to train surrogate; optionally synthesize queries through optimization or generative models; sometimes partial architecture knowledge.	Fidelity and utility of stolen model (accuracy, KL divergence of logit matching); query efficiency (e.g., number of queries).	High computation cost: dominated by query budget and surrogate training compute; adaptive query synthesis can reduce queries but adds generation cost.

Table 2: Comparison of attack categories in the unified closed-loop taxonomy. We summarize threat-model assumptions, attacker capabilities, typical evaluation metrics, and computation cost across the four directional attack classes (D

\rightarrow

D, D

\rightarrow

M, M

\rightarrow

D, M

\rightarrow

M).

Defense Method	Attack(s) Addressed	Effectiveness	Limitations
Data Sanitization	data poisoning attack, backdoor attack, harmful fine-tuning attack	Partially effective	Ineffective once a model is trained; vulnerable to adaptive or clean-label poisoning; may remove useful data and hurt utility
Defensive training	Adversarial attack, data poisoning attack, model inversion attack, model extraction attack and membership inference attack	Partially effective	High computational cost; limited scalability; may reduce accuracy on clean data
Watermarking	model extraction attack (ownership verification)	Detection-only (passive)	passive and cannot prevent model extraction from happening
Output perturbation	model extraction attack, membership inference attack	Partially effective	Degrades utility; vulnerable to adaptive attacks; costly at inference
Alignment fine-tuning	harmful fine-tuning attack, jailbreak attack	Partially effective	Vulnerable to prompt-based bypasses and adversarial fine-tuning; expensive and not robust under distribution shift
Detection	Adversarial attack, harmful fine-tuning attack, jailbreak attack, model extraction attack	not preventive	Reactive only; requires reliable detection signals

Table 3: Mapping of defense methods to the attacks they address, with effectiveness and limitations.

Attack	Privacy	Integrity	Availability
Adversarial examples	$\times$	$\checkmark$	$\times$
Data decryption attack	$\checkmark$	$\times$	$\times$
Watermark removal attack	$\times$	$\checkmark$	$\times$
Data poisoning	$\times$	$\checkmark$	$\times$
Harmful fine-tuning	$\times$	$\checkmark$	$\times$
Jailbreak	$\times$	$\checkmark$	$\times$

Table 4: Mapping representative attacks in our taxonomy to security objectives (privacy, integrity, availability). A checkmark indicates that the attack commonly targets the corresponding security objective.

Attack Category	Training-time	Inference-time	Deployment-time
Adversarial examples	$\times$	✓	$\times$
Data decryption	✓	$\times$	$\times$
Watermark Removal	✓	✓	$\times$
Data poisoning	✓	$\times$	$\times$
Jailbreak	$\times$	✓	✓
Harmful fine-tuning	✓	$\times$	$\times$

Table 5: Mapping representative attack categories in our taxonomy to attack timing (training / inference / deployment). A checkmark indicates that the attack can be instantiated at the corresponding phase.

Dataset	Methods	PSNR	SSIM	RMSE	RMSE w
LVW	U-Net ( Ronneberger et al., 2015 )	30.33	0.9517	7.11	42.18
	Attentive recurrent network ( Qian et al., 2018 )	39.92	0.9902	3.31	21.40
	DHAN ( Cun et al., 2020 )	40.68	0.9949	2.62	17.29
	cGAN-based ( Li et al., 2019 )	33.57	0.9690	5.84	34.71
	VWGAN ( Cao et al., 2019 )	34.16	0.9714	5.51	33.42
	WDNet ( Liu et al., 2021 )	42.45	0.9954	2.39	12.75

Table 6: Comparison of different Watermark Removal attack methods on LVW and CLWD datasets.

Methods	Harmful Score $\downarrow$						Fine-tune Accuracy $\uparrow$
(n=1000)	clean	p=0.01	p=0.05	p=0.1	p=0.2	Avg.	clean	p=0.01	p=0.05	p=0.1	p=0.2	Avg.
Non-Aligned	34.20	65.60	81.00	77.60	79.20	67.52	95.60	94.60	94.00	94.60	94.40	94.64
SFT ( Li et al., 2023 )	48.60	49.80	52.60	55.20	60.00	53.24	94.20	94.40	94.80	94.40	94.20	94.40
EWC ( Kirkpatrick et al., 2017 )	50.60	50.60	50.60	50.60	50.60	50.60	88.60	88.20	87.40	86.80	80.60	86.32
VIguard ( Zong et al., )	49.40	50.00	54.00	54.40	53.60	60.20	94.80	94.80	94.60	94.60	94.60	94.68
KL ( Huang et al., 2024c )	54.40	53.60	55.20	54.00	56.60	54.76	85.80	85.80	85.00	85.40	84.60	59.08

Table 7: Performance under different harmful ratio.

LoRA Fine-tuning	RoleBench $\uparrow$			Safety $\uparrow$				Jailbreak $\uparrow$
LoRA Fine-tuning	RAW	SPE	AVG.	AdvBench	BeaverTails	HEx-PHI	AVG.	AVG.
LLaMA-3-8B-Instruct
LLaMA-3-8B-Instruct baseline	23.86	19.14	21.50	98.46	91.40	95.33	95.06	78.80
SFT ( Ouyang et al., 2022 )	30.42	24.82	27.62	87.75	78.77	83.43	83.32	73.26
SPPFT ( Li et al., 2025b )	29.48	24.49	26.99	86.23	78.95	81.47	82.22	58.18
SafeLoRA ( Hsu et al., 2024 )	29.64	23.84	26.74	88.75	80.88	85.17	84.93	72.28

Table 8: The overall results on the role-play and safety benchmarks with LLaMA-3-8B-Instruct, and Gemma-2-9b-it under the LoRA fine-tuning settings. The results are the average performance across 10 roles.

Method	Pythia	Falcon	LLaMA-2
PPL ( Yeom et al., 2018 )	0.69	0.62	0.61
Min-K% ( Shi et al., )	0.74	0.64	0.64
Min-K%++ ( Zhang et al., 2025b )	0.66	0.74	0.57
Zlib ( Carlini et al., 2021 )	0.72	0.64	0.63
Lowercase ( Carlini et al., 2021 )	0.69	0.63	0.61
Neighbor ( Mattern et al., 2023 )	0.66	0.59	0.61

Table 9: Membership inference attack (MIA) comparison between methods for LLMs on WIKIMIA dataset, "—" indicates that the data are unavailable.

Query Method	Attack Method	None	RevSig	MAD	AM	Top-1	Rounding	MGW	MGS	QUEEN
KnockoffNet	Naive Attack ( Truong et al., 2021 )	87.32%	84.86%	83.61%	82.84%	83.51%	86.91%	75.06%	84.11%	10.00%
	Top-1 Attack ( Sanyal et al., 2022 )	83.51%	83.51%	83.51%	80.30%	83.51%	83.51%	83.51%	83.51%	81.17%
	S4L Attack ( Jagielski et al., 2020 )	85.99%	82.04%	80.72%	81.86%	83.72%	85.49%	71.23%	82.80%	10.00%
	Smoothing Attack ( Lukas et al., 2022 )	87.86%	85.27%	84.07%	84.81%	86.16%	87.37%	76.26%	85.36%	10.00%
	D-DAE ( Chen et al., 2023b )	87.32%	85.62%	84.82%	77.30%	84.81%	86.96%	63.79%	84.97%	78.21%
	D-DAE+ ( Chen et al., 2023b )	87.32%	86.58%	86.84%	84.17%	84.26%	86.93%	58.23%	84.27%	50.24%

Table 11: Defensive performance of different defense methods against different attacks on the target model trained on CIFAR-10.

Watermark	PSNR $\uparrow$	SSIM $\uparrow$	MS-SSIM $\downarrow$	WM Succ. $\uparrow$
DwtDct	39.51	0.972	0.993	0.7815
DwtDctSvd	39.44	0.974	0.993	1.0
RivaGAN	41.22	0.963	0.992	0.6675

Table 12: Stage 1 (D

\rightarrow

D) Evaluation results on the MedMNIST dataset. We report PSNR, SSIM, MS-SSIM and watermark detection success rate.

Method	Attack	Bit Acc.	WM Succ.
DwtDct	cheng2020	0.4825	0.0000
	bmshj2018	0.4820	0.0000
	jpeg	0.5222	0.006
DwtDctSvd	cheng2020	0.4871	0.0005
	bmshj2018	0.4924	0.0000
	jpeg	0.6803	0.2475

Table 13: Stage 1 (D

\rightarrow

D) Watermark removal performance under different attacks. We report bit accuracy (Bit Acc.) and watermark success rate (WM Succ.).

Test accuracy (Original training data)	Test accuracy (Poisoned training data)
93.64%	85.72%

Table 14: Stage 2 (D

\rightarrow

M) data poisoning impact on the model performance.

Category	Victim model test accuracy (Teacher)	Clone model test accuracy (Student)
Soft label	85.72%	34.69%
Hard label	85.72%	36.32%

Table 15: Stage 3 (M

\rightarrow

M) model extraction on the victim model.

Teacher model test accuracy	Student model test accuracy
36.32%	20.18%

Table 16: Stage 4 (M

\rightarrow

D) model inversion on the victim model.

Jul 27, 2026

Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

Multi-modal large language models (MLLMs) integrate heterogeneous modalities through modality alignment and fusion, enabling stronger understanding and reasoning. However, this architectural shift reshapes the safety landscape of machine learning. Increased model complexity and cross-modal interactions give rise to novel threats, including compromised modality integration, modality misalignment, and fused safety risks, reflecting shifts in threat modeling beyond uni-modal assumptions. These shifts, in turn, impose new constraints on safety solutions not captured by existing frameworks rooted in uni-modal learning. Motivated by these challenges, this survey provides a systematic analysis of the evolving safety landscape of MLLMs. We first propose a multimodal grounded taxonomy of safety threats and analyze shifts in threat models, covering adversarial attacks, data poisoning, jailbreaks, and hallucinations. We then summarize updated safety assumptions and organize recent advances in MLLM safety strategies accordingly. Finally, we discuss open challenges and future directions to inform the development of more principled and scalable safety mechanisms for multimodal systems.

Xi Li, Shu Zhao, Xiaohan Zou +6

University of Alabama at Birmingham · NVIDIA · Penn State University +4

MLLM Safety Adversarial Attacks on VLMs Multimodal Large Language Models LLM Safety

Ask PDF

Aug 17, 2026

Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation

Foundation models are increasingly used for perception, reasoning, planning, and action generation in embodied agents, creating security risks that can propagate from digital inputs to physical behavior. Existing surveys often organize threats by mechanisms such as jailbreaks, prompt injection, backdoors, poisoning, or adversarial examples, but these categories do not consistently identify where an adversary first enters the embodied control loop. We present a trust-boundary-centric survey of foundation-model-powered embodied-agent security. Using a first-compromised-trust-boundary principle, we separate attack surface from attack mechanism and organize the system into five layers and twelve attack surfaces spanning the model supply chain, user instructions, context and memory, physical semantic environments, multimodal perception, world state, internal reasoning, task planning, action interfaces, middleware, multi-agent communication, and execution control. Based on 58 attack records and 61 defense records collected through August 15, 2026, we analyze representative attacks, cross-layer propagation, defense placement, and evaluation practices. Our quantitative analysis shows that attack research is concentrated on multimodal perception and action interfaces, while defenses are especially concentrated on action-level and runtime protection. Context and long-term memory, middleware and networking, world-state integrity, and multi-agent trust remain comparatively underexplored. We conclude with open challenges in state provenance, compositional defenses, long-horizon attack propagation, physical realizability, Byzantine multi-robot behavior, and unified closed-loop evaluation.

Jiawei Liu, Jiacheng Guo, Tian Zhang +4

Wuhan University

AI Agent Security Multi-Agent System Security

Ask PDF

Apr 7, 2026

A Formal Security Framework for MCP-Based AI Agents: Threat Taxonomy, Verification Models, and Defense Mechanisms

The Model Context Protocol (MCP), introduced by Anthropic in November 2024 and now governed by the Linux Foundation's Agentic AI Foundation, has rapidly become the de facto standard for connecting large language model (LLM)-based agents to external tools and data sources, with over 97 million monthly SDK downloads and more than 177000 registered tools. However, this explosive adoption has exposed a critical gap: the absence of a unified, formal security framework capable of systematically characterizing, analyzing, and mitigating the diverse threats facing MCP-based agent ecosystems. Existing security research remains fragmented across individual attack papers, isolated benchmarks, and point defense mechanisms. This paper presents MCPSHIELD, a comprehensive formal security framework for MCP-based AI agents. We make four principal contributions: (1) a hierarchical threat taxonomy comprising 7 threat categories and 23 distinct attack vectors organized across four attack surfaces, grounded in the analysis of over 177000 MCP tools; (2) a formal verification model based on labeled transition systems with trust boundary annotations that enables static and runtime analysis of MCP tool interaction chains; (3) a systematic comparative evaluation of 12 existing defense mechanisms, identifying coverage gaps across our threat taxonomy; and (4) a defense in depth reference architecture integrating capability based access control, cryptographic tool attestation, information flow tracking, and runtime policy enforcement. Our analysis reveals that no existing single defense covers more than 34 percent of the identified threat landscape, whereas MCPSHIELD's integrated architecture achieves theoretical coverage of 91 percent. We further identify seven open research challenges that must be addressed to secure the next generation of agentic AI systems.

Nirajan Acharya, Gaurav Kumar Gupta

University of the Cumberlands · Youngstown State University

MCP Security AI Agent Security Tool Access Control for LLM Agents Formal Verification

Ask PDF

Abstract

Explore similar work

Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation

A Formal Security Framework for MCP-Based AI Agents: Threat Taxonomy, Verification Models, and Defense Mechanisms