Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges

Seyedakbar Mostafavi

cs.AISep 12, 2026

Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges

Authors: Seyedakbar Mostafavi

Organizations: Department of Computer Engineering, Yazd University, Yazd 89195-741, Iran

Abstract

The transition from passive foundation models to autonomous, goal-directed agentic AI systems has introduced unprecedented capabilities by coupling recursive cognitive reasoning loops, persistent memory architectures, live tool execution planes, and multi-agent collaboration topologies. However, granting probabilistic neural cores execution authority across filesystems, networks, and cloud infrastructure dissolves classical security perimeters: natural language simultaneously serves as input data, internal control code, and communication protocols, exposing a Turing-complete blast radius where untrusted data represents executable instructions. This survey delivers a comprehensive systems-security reference framework for trustworthy agentic AI, synthesizing 206 foundational studies and regulatory standards. We formalize the general agent architecture as a stateful 5-tuple and establish a 6-dimensional trustworthiness taxonomy covering security, safety, privacy, explainability, fairness, and accountability. We systematically analyze threat surfaces across intra-execution loops and interaction planes, formulate a multi-layered zero-trust defense-in-depth architecture integrating Dual-LLM isolation, Capability-Based Access Control, kernel eBPF probes, and sandboxed runtimes, review standardized evaluation benchmarks, and map technical controls to international AI governance frameworks.

Figures & tables

Panel of Figure 1 — Fig. 1: Historical Evolution of Artificial Intelligence Paradigms.

Panel of Figure 2 — Fig. 2: The Four Critical Knowledge Gaps in Trustworthy Agentic AI Security.

Survey	Year	Primary Focus / Scope	Key Capabilities	Identified Limitations / Gaps
Xi et al. [ 1 ]	2023	General LLM agents	Cognitive architecture	Overlooks adversarial threat vectors and systems sandboxing.
Wang et al. [ 2 ]	2024	Agent development	Planning & evaluation	Lacks systems security and memory poisoning analysis.
Greshake et al. [ 20 ]	2023	Indirect prompt injection	App vulnerability demo	Limited to initial IPI discovery; lacks defense blueprint.
Zhan et al. [ 21 ]	2024	Tool-integrated agents	Benchmarking across tools	Empirical attack suite; omits multi-agent and memory planes.
Deng et al. [ 15 ]	2024	Web interactive agents	Web task execution	Focuses on task success, not operational security.
Yao et al. [ 5 ]	2023	Deliberate planning	ToT reasoning search	Evaluates reasoning without adversarial threat models.

TABLE I: Comprehensive Comparison of This Survey with Existing Surveys in AI and Agent Security

Abbreviation	Definition
ACL	Agent Communications Language
ANN	Approximate Nearest Neighbor
ASR	Attack Success Rate
BFT / PBFT	Byzantine Fault Tolerance / Practical BFT
BRI	Blast Radius Index
CapBAC	Capability-Based Access Control

TABLE II: Summary of Key Abbreviations in Alphabetical Order

Panel of Figure 5 — Fig. 3: General Architecture and Closed-Loop Execution Workflow of Agentic AI Systems.

Framework	Primary Domain	Reasoning	Memory Model	Key Vulnerability Point
AutoGPT [ 1 ]	General automation	ReAct	Working + Vector DB	Unsanitized tool execution; loops.
AutoGen [ 12 ]	Multi-agent chat	Conversable	Context passing	Byzantine peer injection; no signing.
MetaGPT [ 13 ]	Software engineering	SOP role play	Pub/Sub shared	Cascading hallucination propagation.
ChatDev [ 26 ]	Collaborative dev	Waterfall	Role buffers	Cross-role prompt injection via code.
SWE-agent [ 14 ]	Issue resolution	ACI + ReAct	Working + diffs	Shell parameter injection in bash.
MemGPT [ 8 ]	Conversational OS	Context paging	Multi-tier vector	Vector poisoning & inversion.

TABLE III: Comparison of Leading Agent Frameworks and Architectures

Panel of Figure 7 — Fig. 4: The 6-Dimensional Trustworthiness Taxonomy for Agentic AI Systems.

Dimension	Core System Invariant	Verification Method	Enforced Systems Security Control
1. Security	$\mathbf{Adv}_{\mathcal{A}}^{\text{leak}}\leq\operatorname{negl}(\lambda)\land\text{Integ}(\Pi)$	Red-teaming [ 40 ] , SMT [ 113 ]	CapBAC tokens & micro-VM sandboxes.
2. Safety	$\forall a_{t}\in\mathcal{T}_{\text{irr}},\text{Auth}_{\text{h}}(a_{t})=1$	Invariant check, R-Judge [ 100 ]	Bounded action envelopes & rollbacks.
3. Privacy	$\Pr[\mathcal{M}(D)\in S]\leq e^{\epsilon}\Pr[\mathcal{M}(D^{\prime})\in S]+\delta$	Membership audits [ 103 ]	Differential Privacy noise injection.
4. Explainability	$\text{CIE}(th_{t}\to a_{t})\geq\tau$	Mediation analysis [ 111 ]	Immutable action-thought chains & SMT logs.
5. Fairness	$\mathbb{E}[U\mid G_{A}]=\mathbb{E}[U\mid G_{B}]$	Parity testing [ 115 ]	Balanced priority scheduling & auction rules.
6. Accountability	$\text{VerifySig}(a_{t})\land\text{MerklePath}(a_{t},\text{Root}_{t})=1$	Audit validation [ 117 ]	W3C DIDs [ 118 ] & signed Merkle trees.

TABLE IV: Cross-Dimensional Trustworthiness Matrix: Formal Dimensions, Verification Methods, and Systems Controls

Panel of Figure 9 — Fig. 5: Taxonomy of Intra-Execution Security Threats in Agentic Systems.

Panel of Figure 10 — Fig. 6: Planning Topologies and Error Amplification Pathways in Cognitive Cores: (a) Linear error propagation without recovery in sequential execution loops; (b) Adversarial heuristic score manipulation directing search trees toward poisoned execution branches.

Layer	Attack Vector	Primary Mechanism	Evaluated Impact	Literature
Perception	Direct Prompt Injection	Ingesting adversarial text	Alters goal; bypasses rules	[ 22 , 122 ]
Perception	Adversarial Suffixes	Token search (GCG)	Jailbreaks aligned models	[ 38 , 42 ]
Perception	Multimodal Injection	Steganographic noise	Jailbreaks vision agents	[ 126 , 125 ]
Brain	Backdoors	Poisoned datasets	Trigger-activated actions	[ 50 , 51 ]
Brain	Misalignment	Data bias & sycophancy	Generates toxic outputs	[ 141 , 142 ]
Brain	Planning Errors	Error amplification	Cascading task failure	[ 6 , 49 ]

TABLE V: Master Overview of Security Threats to Agentic AI Systems

Interaction Surface	Threat Vector	Adversary Model	Systems Impact	Key References
Environment ( $\mathcal{E}$ )	Indirect Prompt Injection	Black-box data untrusted injection	Goal hijacking & unauthorized tool dispatch	Greshake et al. [ 20 ] , Zhan et al. [ 21 ]
Environment ( $\mathcal{E}$ )	Markdown Data Exfiltration	Passive untrusted content poisoning	Zero-click credential exfiltration via GET	Greshake et al. [ 20 ] , Zhan et al. [ 21 ]
Environment ( $\mathcal{E}$ )	Cloud Metadata SSRF	Injected URL tool dispatch	Cluster node IAM credential theft (IMDS)	Fang et al. [ 35 ]
Environment ( $\mathcal{E}$ )	Sponge DoS / Context Bomb	Malicious prompt perturbation	GPU memory exhaustion & latency spike	Shumailov et al. [ 57 ] , Guastalla et al. [ 58 ]
Memory ( $\mathcal{M}$ )	PoisonedRAG Vector Attack	Black-box document injection ( $<5$ docs)	$>90\%$ top-1 retrieval hijacking	Zou et al. [ 59 ] , Chen et al. [ 60 ]
Memory ( $\mathcal{M}$ )	Dense Embedding Inversion	White/gray-box vector access	$>70\%$ raw private document reconstruction	Morris et al. [ 110 ] , Li et al. [ 109 ]

TABLE VI: Taxonomy of Interaction Threat Vectors, Adversary Models, Systems Impact, and Citations

Panel of Figure 13 — Fig. 7: The 4-Layer Zero-Trust Defense-in-Depth Architecture for Agentic AI Systems.

Defensive Control	Target Attack Surface	Latency Overhead	Token Cost	Deployment Model	Key Operational Limitation
Dual-LLM Isolation [ 158 ]	IPI / Data-as-Code	High ( $+1.5\text{--}3.0\text{s}$ )	$2.0\times$	API & Self-Hosted	Doubles model inference costs.
Attention Masking [ 159 ]	Attention-level IPI	Negligible ( $<5\text{ms}$ )	$1.0\times$	White-Box Only	Incompatible with closed-weight APIs.
Signed Vectors [ 117 ]	RAG DB Poisoning	Low ( $<10\text{ms}$ )	$1.0\times$	Universal Vector DB	Requires trusted signing PKI infrastructure.
DP Noise Injection [ 161 ]	Embedding Inversion	Zero ( $0\text{ms}$ )	$1.0\times$	Embedding Pipeline	Degrades top- $k$ semantic retrieval precision.
CapBAC Tokens [ 36 ]	Confused Deputy / RCE	Negligible ( $<2\text{ms}$ )	$1.0\times$	Cluster Gateway	Requires granular tool permission schema.
Micro-VM Sandboxing [ 54 ]	Host Escape / RCE	Moderate ( $35\text{--}50\text{ms}$ )	$1.0\times$	Kubernetes / Cloud	Ephemeral state loss; storage volume overhead.

TABLE VII: Comprehensive Comparative Evaluation of Zero-Trust Defensive Mechanisms

Benchmark	Target Layer	Environment	Primary Metric	Empirical Baseline Result
InjecAgent [ 21 ]	Tool Plane	17+ APIs	Attack Success (ASR)	GPT-4: 24.1%–45.3% ASR under IPI.
BIPIA [ 144 ]	Cognitive Input	Email, Web, Docs	Injection Rate	$>50\%$ ASR across standard LLMs.
AgentBench [ 181 ]	Full Loop	OS, DB, Web	Success / Safety	GPT-4: 4.01/5; open-source: ¡2.0.
R-Judge [ 100 ]	Cognitive Plan	1,624 Records	Safety Risk F1	GPT-4: 70% F1 on dangerous actions.
ToolBench [ 182 ]	Tool Execution	16,000+ APIs	Pass Rate	Function selection scales with tuning.
CyberSecEval [ 28 ]	Cyber Action	Exploits & Repos	Exploit Likelihood	LLMs generate weaponizable exploits.

TABLE VIII: Comprehensive Benchmark Taxonomy and Evaluation Analysis

Framework	Mandate	Target Vulnerability	Implemented Technical Defense
NIST AI RMF [ 119 ]	GOVERN 1.2, MANAGE 2.4	Unauthorized tool dispatch	CapBAC tokens & Firecracker micro-VMs.
EU AI Act Art. 12 [ 121 ]	High-Risk Event Logs	Reasoning tampering	Merkle Audit Trees with hash chaining ( $\text{Root}_{t}$ ).
EU AI Act Art. 14 [ 121 ]	Human Oversight	Autonomous drift	Three-tier HITL/HOTL/HIC approval gates.
ISO/IEC 42001 [ 120 ]	AI Data Traceability	Vector DB poisoning	Signed vector embeddings & DP noise.
OWASP Top 10 [ 192 ]	Excessive Agency (LLM06)	Arbitrary RCE	Kernel eBPF syscall probes & Dual-LLM isolation.

TABLE IX: Cross-Regulatory Mapping Between International Mandates and Zero-Trust Technical Controls

Panel of Figure 17 — Fig. 8: Taxonomy of Open Research Horizons in Trustworthy Agentic AI.

Research Horizon	Timeframe	Foundational Scientific Bottleneck	Key Breakthrough Milestone
1. Formal Action Verification	1–3 Years	Semantic-to-symbolic fidelity gap; state-space explosion.	Real-time grammar-constrained SMT invariant solver.
2. Lifelong Dynamic Alignment	2–4 Years	In-context alignment decay; non-invertible side effects.	Non-monotonic memory scrubbers with shadow execution.
3. Autonomic Swarm Immunity	3–5 Years	$O(n^{2})$ PBFT token/latency cost in LLM clusters.	Optimistic consensus guided by GNN message telemetry.
4. Machine-Speed Cyber Defense	2–5 Years	Asymmetric exploit generation vs. regression-free patching.	Autonomous SOC agents with formal patch verification.
5. Pareto Cluster Scheduling	1–3 Years	Multi-LLM inspection latency vs. throughput.	BRI-throttled scheduling across GPU/TEE clusters.
6. Autonomous Value Networks	3–6 Years	Lack of verifiable agency and private provenance.	Standardized $ZK\text{-ML}$ reasoning proofs and W3C DIDs.

TABLE X: Strategic Roadmap for Open Research Horizons in Trustworthy Agentic AI

Explore similar work

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
May 17, 2026Jinhu Qi, Muzhi Li, Jiahong Liu +9AI Agent Reliability AI Agent Evaluation
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
Sep 19, 2026Fayeq Jeelani Syed, Rehan Ahmad, Ali Al Bataineh +1LLM Agent Security LLM Agents
Department of Human-Centered Computing, Indiana University Indianapolis, 755 W. Michigan Street, Indianapolis, Indiana, 46202, USA · Department of Electrical and Computer Engineering, Purdue University Northwest, 2200 169th Street, Hammond, Indiana, 46323, USA · Computer Information Science, Higher Colleges of Technology, Abu Dhabi, United Arab Emirates +1
Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges
Sep 20, 2026Heewon Baek, Alsharif Abuadbba, Kristen Moore +2AI Agent Security AI Agent Evaluation
Sungkyunkwan University, Republic of Korea · CSIRO, Australia · Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia

May 17, 2026J/K move · Enter open · S save

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Jinhu Qi, Muzhi Li, Jiahong Liu, Yuqin Shu+8

Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.

AI Agent Reliability AI Agent Evaluation AI Agent Security AI Agent Safety

Open paper AskPDF arXiv

Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Abstract

Explore similar work

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems

Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges