Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges

Seyedakbar Mostafavi

cs.AISep 12, 2026

Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges

Authors: Seyedakbar Mostafavi

Organizations: Department of Computer Engineering, Yazd University, Yazd 89195-741, Iran

Abstract

The transition from passive foundation models to autonomous, goal-directed agentic AI systems has introduced unprecedented capabilities by coupling recursive cognitive reasoning loops, persistent memory architectures, live tool execution planes, and multi-agent collaboration topologies. However, granting probabilistic neural cores execution authority across filesystems, networks, and cloud infrastructure dissolves classical security perimeters: natural language simultaneously serves as input data, internal control code, and communication protocols, exposing a Turing-complete blast radius where untrusted data represents executable instructions. This survey delivers a comprehensive systems-security reference framework for trustworthy agentic AI, synthesizing 206 foundational studies and regulatory standards. We formalize the general agent architecture as a stateful 5-tuple and establish a 6-dimensional trustworthiness taxonomy covering security, safety, privacy, explainability, fairness, and accountability. We systematically analyze threat surfaces across intra-execution loops and interaction planes, formulate a multi-layered zero-trust defense-in-depth architecture integrating Dual-LLM isolation, Capability-Based Access Control, kernel eBPF probes, and sandboxed runtimes, review standardized evaluation benchmarks, and map technical controls to international AI governance frameworks.

Figures & tables

Panel of Figure 1 — Fig. 1: Historical Evolution of Artificial Intelligence Paradigms.

Panel of Figure 2 — Fig. 2: The Four Critical Knowledge Gaps in Trustworthy Agentic AI Security.

Survey	Year	Primary Focus / Scope	Key Capabilities	Identified Limitations / Gaps
Xi et al. [ 1 ]	2023	General LLM agents	Cognitive architecture	Overlooks adversarial threat vectors and systems sandboxing.
Wang et al. [ 2 ]	2024	Agent development	Planning & evaluation	Lacks systems security and memory poisoning analysis.
Greshake et al. [ 20 ]	2023	Indirect prompt injection	App vulnerability demo	Limited to initial IPI discovery; lacks defense blueprint.
Zhan et al. [ 21 ]	2024	Tool-integrated agents	Benchmarking across tools	Empirical attack suite; omits multi-agent and memory planes.
Deng et al. [ 15 ]	2024	Web interactive agents	Web task execution	Focuses on task success, not operational security.
Yao et al. [ 5 ]	2023	Deliberate planning	ToT reasoning search	Evaluates reasoning without adversarial threat models.

TABLE I: Comprehensive Comparison of This Survey with Existing Surveys in AI and Agent Security

Abbreviation	Definition
ACL	Agent Communications Language
ANN	Approximate Nearest Neighbor
ASR	Attack Success Rate
BFT / PBFT	Byzantine Fault Tolerance / Practical BFT
BRI	Blast Radius Index
CapBAC	Capability-Based Access Control

TABLE II: Summary of Key Abbreviations in Alphabetical Order

Panel of Figure 5 — Fig. 3: General Architecture and Closed-Loop Execution Workflow of Agentic AI Systems.

Framework	Primary Domain	Reasoning	Memory Model	Key Vulnerability Point
AutoGPT [ 1 ]	General automation	ReAct	Working + Vector DB	Unsanitized tool execution; loops.
AutoGen [ 12 ]	Multi-agent chat	Conversable	Context passing	Byzantine peer injection; no signing.
MetaGPT [ 13 ]	Software engineering	SOP role play	Pub/Sub shared	Cascading hallucination propagation.
ChatDev [ 26 ]	Collaborative dev	Waterfall	Role buffers	Cross-role prompt injection via code.
SWE-agent [ 14 ]	Issue resolution	ACI + ReAct	Working + diffs	Shell parameter injection in bash.
MemGPT [ 8 ]	Conversational OS	Context paging	Multi-tier vector	Vector poisoning & inversion.

TABLE III: Comparison of Leading Agent Frameworks and Architectures

Panel of Figure 7 — Fig. 4: The 6-Dimensional Trustworthiness Taxonomy for Agentic AI Systems.

Dimension	Core System Invariant	Verification Method	Enforced Systems Security Control
1. Security	$\mathbf{Adv}_{\mathcal{A}}^{\text{leak}}\leq\operatorname{negl}(\lambda)\land\text{Integ}(\Pi)$	Red-teaming [ 40 ] , SMT [ 113 ]	CapBAC tokens & micro-VM sandboxes.
2. Safety	$\forall a_{t}\in\mathcal{T}_{\text{irr}},\text{Auth}_{\text{h}}(a_{t})=1$	Invariant check, R-Judge [ 100 ]	Bounded action envelopes & rollbacks.
3. Privacy	$\Pr[\mathcal{M}(D)\in S]\leq e^{\epsilon}\Pr[\mathcal{M}(D^{\prime})\in S]+\delta$	Membership audits [ 103 ]	Differential Privacy noise injection.
4. Explainability	$\text{CIE}(th_{t}\to a_{t})\geq\tau$	Mediation analysis [ 111 ]	Immutable action-thought chains & SMT logs.
5. Fairness	$\mathbb{E}[U\mid G_{A}]=\mathbb{E}[U\mid G_{B}]$	Parity testing [ 115 ]	Balanced priority scheduling & auction rules.
6. Accountability	$\text{VerifySig}(a_{t})\land\text{MerklePath}(a_{t},\text{Root}_{t})=1$	Audit validation [ 117 ]	W3C DIDs [ 118 ] & signed Merkle trees.

TABLE IV: Cross-Dimensional Trustworthiness Matrix: Formal Dimensions, Verification Methods, and Systems Controls

Panel of Figure 9 — Fig. 5: Taxonomy of Intra-Execution Security Threats in Agentic Systems.

Panel of Figure 10 — Fig. 6: Planning Topologies and Error Amplification Pathways in Cognitive Cores: (a) Linear error propagation without recovery in sequential execution loops; (b) Adversarial heuristic score manipulation directing search trees toward poisoned execution branches.

Layer	Attack Vector	Primary Mechanism	Evaluated Impact	Literature
Perception	Direct Prompt Injection	Ingesting adversarial text	Alters goal; bypasses rules	[ 22 , 122 ]
Perception	Adversarial Suffixes	Token search (GCG)	Jailbreaks aligned models	[ 38 , 42 ]
Perception	Multimodal Injection	Steganographic noise	Jailbreaks vision agents	[ 126 , 125 ]
Brain	Backdoors	Poisoned datasets	Trigger-activated actions	[ 50 , 51 ]
Brain	Misalignment	Data bias & sycophancy	Generates toxic outputs	[ 141 , 142 ]
Brain	Planning Errors	Error amplification	Cascading task failure	[ 6 , 49 ]

TABLE V: Master Overview of Security Threats to Agentic AI Systems

Interaction Surface	Threat Vector	Adversary Model	Systems Impact	Key References
Environment ( $\mathcal{E}$ )	Indirect Prompt Injection	Black-box data untrusted injection	Goal hijacking & unauthorized tool dispatch	Greshake et al. [ 20 ] , Zhan et al. [ 21 ]
Environment ( $\mathcal{E}$ )	Markdown Data Exfiltration	Passive untrusted content poisoning	Zero-click credential exfiltration via GET	Greshake et al. [ 20 ] , Zhan et al. [ 21 ]
Environment ( $\mathcal{E}$ )	Cloud Metadata SSRF	Injected URL tool dispatch	Cluster node IAM credential theft (IMDS)	Fang et al. [ 35 ]
Environment ( $\mathcal{E}$ )	Sponge DoS / Context Bomb	Malicious prompt perturbation	GPU memory exhaustion & latency spike	Shumailov et al. [ 57 ] , Guastalla et al. [ 58 ]
Memory ( $\mathcal{M}$ )	PoisonedRAG Vector Attack	Black-box document injection ( $<5$ docs)	$>90\%$ top-1 retrieval hijacking	Zou et al. [ 59 ] , Chen et al. [ 60 ]
Memory ( $\mathcal{M}$ )	Dense Embedding Inversion	White/gray-box vector access	$>70\%$ raw private document reconstruction	Morris et al. [ 110 ] , Li et al. [ 109 ]

TABLE VI: Taxonomy of Interaction Threat Vectors, Adversary Models, Systems Impact, and Citations

Panel of Figure 13 — Fig. 7: The 4-Layer Zero-Trust Defense-in-Depth Architecture for Agentic AI Systems.

Defensive Control	Target Attack Surface	Latency Overhead	Token Cost	Deployment Model	Key Operational Limitation
Dual-LLM Isolation [ 158 ]	IPI / Data-as-Code	High ( $+1.5\text{--}3.0\text{s}$ )	$2.0\times$	API & Self-Hosted	Doubles model inference costs.
Attention Masking [ 159 ]	Attention-level IPI	Negligible ( $<5\text{ms}$ )	$1.0\times$	White-Box Only	Incompatible with closed-weight APIs.
Signed Vectors [ 117 ]	RAG DB Poisoning	Low ( $<10\text{ms}$ )	$1.0\times$	Universal Vector DB	Requires trusted signing PKI infrastructure.
DP Noise Injection [ 161 ]	Embedding Inversion	Zero ( $0\text{ms}$ )	$1.0\times$	Embedding Pipeline	Degrades top- $k$ semantic retrieval precision.
CapBAC Tokens [ 36 ]	Confused Deputy / RCE	Negligible ( $<2\text{ms}$ )	$1.0\times$	Cluster Gateway	Requires granular tool permission schema.
Micro-VM Sandboxing [ 54 ]	Host Escape / RCE	Moderate ( $35\text{--}50\text{ms}$ )	$1.0\times$	Kubernetes / Cloud	Ephemeral state loss; storage volume overhead.

TABLE VII: Comprehensive Comparative Evaluation of Zero-Trust Defensive Mechanisms

Benchmark	Target Layer	Environment	Primary Metric	Empirical Baseline Result
InjecAgent [ 21 ]	Tool Plane	17+ APIs	Attack Success (ASR)	GPT-4: 24.1%–45.3% ASR under IPI.
BIPIA [ 144 ]	Cognitive Input	Email, Web, Docs	Injection Rate	$>50\%$ ASR across standard LLMs.
AgentBench [ 181 ]	Full Loop	OS, DB, Web	Success / Safety	GPT-4: 4.01/5; open-source: ¡2.0.
R-Judge [ 100 ]	Cognitive Plan	1,624 Records	Safety Risk F1	GPT-4: 70% F1 on dangerous actions.
ToolBench [ 182 ]	Tool Execution	16,000+ APIs	Pass Rate	Function selection scales with tuning.
CyberSecEval [ 28 ]	Cyber Action	Exploits & Repos	Exploit Likelihood	LLMs generate weaponizable exploits.

TABLE VIII: Comprehensive Benchmark Taxonomy and Evaluation Analysis

Framework	Mandate	Target Vulnerability	Implemented Technical Defense
NIST AI RMF [ 119 ]	GOVERN 1.2, MANAGE 2.4	Unauthorized tool dispatch	CapBAC tokens & Firecracker micro-VMs.
EU AI Act Art. 12 [ 121 ]	High-Risk Event Logs	Reasoning tampering	Merkle Audit Trees with hash chaining ( $\text{Root}_{t}$ ).
EU AI Act Art. 14 [ 121 ]	Human Oversight	Autonomous drift	Three-tier HITL/HOTL/HIC approval gates.
ISO/IEC 42001 [ 120 ]	AI Data Traceability	Vector DB poisoning	Signed vector embeddings & DP noise.
OWASP Top 10 [ 192 ]	Excessive Agency (LLM06)	Arbitrary RCE	Kernel eBPF syscall probes & Dual-LLM isolation.

TABLE IX: Cross-Regulatory Mapping Between International Mandates and Zero-Trust Technical Controls

Panel of Figure 17 — Fig. 8: Taxonomy of Open Research Horizons in Trustworthy Agentic AI.

Research Horizon	Timeframe	Foundational Scientific Bottleneck	Key Breakthrough Milestone
1. Formal Action Verification	1–3 Years	Semantic-to-symbolic fidelity gap; state-space explosion.	Real-time grammar-constrained SMT invariant solver.
2. Lifelong Dynamic Alignment	2–4 Years	In-context alignment decay; non-invertible side effects.	Non-monotonic memory scrubbers with shadow execution.
3. Autonomic Swarm Immunity	3–5 Years	$O(n^{2})$ PBFT token/latency cost in LLM clusters.	Optimistic consensus guided by GNN message telemetry.
4. Machine-Speed Cyber Defense	2–5 Years	Asymmetric exploit generation vs. regression-free patching.	Autonomous SOC agents with formal patch verification.
5. Pareto Cluster Scheduling	1–3 Years	Multi-LLM inspection latency vs. throughput.	BRI-throttled scheduling across GPU/TEE clusters.
6. Autonomous Value Networks	3–6 Years	Lack of verifiable agency and private provenance.	Standardized $ZK\text{-ML}$ reasoning proofs and W3C DIDs.

TABLE X: Strategic Roadmap for Open Research Horizons in Trustworthy Agentic AI

Explore similar work

May 17, 2026

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Jinhu Qi, Muzhi Li, Jiahong Liu +9

Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.

AI Agent Reliability AI Agent Evaluation AI Agent Security AI Agent Safety

Ask PDF

Sep 19, 2026

Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems

Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.

Fayeq Jeelani Syed, Rehan Ahmad, Ali Al Bataineh +1

Department of Human-Centered Computing, Indiana University Indianapolis, 755 W. Michigan Street, Indianapolis, Indiana, 46202, USA · Department of Electrical and Computer Engineering, Purdue University Northwest, 2200 169th Street, Hammond, Indiana, 46323, USA · Computer Information Science, Higher Colleges of Technology, Abu Dhabi, United Arab Emirates +1

LLM Agent Security LLM Agents LLM Agent Safety AI Agent Governance AI Agent Auditing

Ask PDF

Sep 20, 2026

Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges

Agentic AI extends LLM security beyond generated content to persistent state, autonomous actions, tool use, and interactions with humans and other agents. Existing threat classifications often emphasize individual dimensions, obscuring connections among entry points, affected components, and security consequences. The known threat landscape also differs from the coverage demonstrated by empirical research. Through a structured review of 66 studies published from 2022 to 2026, we introduce T={S, B, P, A}, a cross-dimensional representation linking affected functional or system surfaces {S}, interaction or trust boundaries {B}, violated security properties {P}, and empirically examined architectures {A}. We analyze 22 artifact-backed red-teaming studies and 11 representative security benchmarks to characterize empirical coverage and evaluation maturity. Within the selected studies, evidence concentrates on prompt/reasoning, memory, and tool-mediated attacks, predominantly in single-agent settings. Persistent, Human--Agent, complex multi-agent, systemic, and long-horizon threats receive less coverage. These findings describe the selected corpus rather than establish gaps across all empirical research. Heterogeneous metrics, limited adaptive defense evaluation, architectural imbalance, and incomplete execution-state capture further constrain comparison and reproducibility. We derive 13 open research questions to guide more systematic, architecture-aware, and reproducible security evaluation of agentic AI.

Heewon Baek, Alsharif Abuadbba, Kristen Moore +2

Sungkyunkwan University, Republic of Korea · CSIRO, Australia · Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia

AI Agent Security AI Agent Evaluation AI Agent Security Benchmarks Multi-Agent System Security

Ask PDF