Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
Authors: Fayeq Jeelani Syed, Rehan Ahmad, Ali Al Bataineh, Aakriti Adhikari
Organizations: Department of Human-Centered Computing, Indiana University Indianapolis, 755 W. Michigan Street, Indianapolis, Indiana, 46202, USA · Department of Electrical and Computer Engineering, Purdue University Northwest, 2200 169th Street, Hammond, Indiana, 46323, USA · Computer Information Science, Higher Colleges of Technology, Abu Dhabi, United Arab Emirates · Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, 333 Cedar Street, New Haven, Connecticut, 06510, USA
Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.
Figures & tables
Survey
Arch.
Safety/ Security
Alignment/ Oversight
Transparency
Privacy
Governance
Life cycle
Agent Evaluation
Yu et al. (2025) [ 76 ]
P
S
P
P
S
L
–
P
Raza et al. (2026) [ 58 ]
P
S
P
P
P
S
P
P
Abou Ali et al. (2026) [ 1 ]
S
P
L
L
L
P
–
P
Gan et al. (2024) [ 28 ]
P
S
L
P
S
L
–
L
This survey
P
S
S
S
S
S
S
P
Table 1: Comparison of the present survey with closely related prior surveys on key trustworthiness dimensions. Coverage levels: S = Substantial (primary focus with analysis and mitigations); P = Partial (discussed but not the primary focus); L = Limited (mentioned or briefly noted); – = Not a primary focus. For all surveys including this one, ratings are author assessments of the depth of treatment a dimension actually receives, not an independent quality assessment. The same rubric was applied to the present survey; where a dimension is surveyed to identify gaps rather than analysed in depth, it is rated P rather than S. Reasonable disagreement is possible; readers are encouraged to consult the original works.
Architecture
Trust Challenges
Examples
Single-Agent ReAct
Goal drift; tool misuse; context-length limits
ReAct [ 74 ]
Hierarchical Multi-Agent
Cascading failures; authority confusion
AutoGen [ 72 ] ; MetaGPT [ 32 ]
Collaborative Peer Agents
Convergence on an incorrect view; judge bias
LLM debate [ 44 ]
Human-in-the-Loop (HITL)
Approval fatigue; automation bias
HITL agent deployment patterns
Fully Autonomous Pipeline
All failure modes amplified
Automated software-engineering agents
Table 2: Agentic AI architecture classes and distinctive trustworthiness challenges.
Dimension
Sub-type
Example
Illus. Sev.
Safety
Unsafe tool use
Agent executes attacker-injected code
C
Safety
Cascading failure
Cloud misconfiguration triggers data loss
C
Safety
Resource exhaustion
Loop causes 10,000 unintended API calls
H
Alignment
Goal misgeneralization
Agent spams to maximise “email responses”
C
Alignment
Spec. gaming
Agent marks task complete without solving it
H
Alignment
Instruction misinterp.
Agent deletes active files matching “old”
H
Table 3: Taxonomy of agentic failure modes with illustrative examples and author-assigned illustrative severity ratings (C = Critical; H = High). Ratings are based on scope of harm, reversibility, agent autonomy, and detectability, and reflect typical high-stakes scenarios; actual severity is context-dependent.
Juris.
Instrument / Framework
Legal Status
Binding?
Applicability to Agentic AI
Key Effective Date
EU
AI Act [ 21 ] (Reg. (EU) 2024/1689)
Binding law (in force), as amended by Reg. (EU) 2026/1744
Yes
Agentic systems classified by purpose/domain/role, not architecture
General application 2 Aug 2026
EU
Digital Omnibus on AI [ 22 ] (Reg. (EU) 2026/1744)
Binding law (in force)
Yes
Amends the AI Act; defers high-risk obligations and adds new Art. 5 prohibitions
In force 27 Jul 2026; Annex III 2 Dec 2027; Annex I 2 Aug 2028
EU
Product Liability Directive 2024/2853 [ 20 ]
Adopted; transposition pending
Yes (on transposition)
Covers software and AI-enabled products; defect liability
9 Dec 2026 (transposition deadline)
EU
AI Liability Directive (COM(2022) 496) [ 17 ]
Proposal withdrawn (Oct 2025)
N/A
No longer advancing; coverage remains via PLD and national law
Withdrawn 6 Oct 2025
USA
EO 14179 [ 24 ]
Active EO following the separate revocation of EO 14110
EO only
Deregulatory emphasis; AI Action Plan mandated; sector agency guidance persists
Signed 23 Jan 2025
USA
America’s AI Action Plan [ 26 ]
Federal policy plan (issued under EO 14179)
No
Three pillars: innovation, infrastructure, international diplomacy and security; no direct obligations on deployers
Released 23 Jul 2025
Table 4: International AI governance landscape and relevance to agentic AI systems (status: August 2026). The table distinguishes binding law, adopted-but-not-fully-applicable requirements, revoked executive actions, proposed or archived legislation, technical standards, voluntary frameworks, and government guidance.
Phase
Key Trust Activities
Required Artefacts
Gate Criteria
1. Specification
Define task scope, principal hierarchy, risk tier, and regulatory requirements; model agent-specific threats
No unresolved critical failures; thresholds met; residual risks documented and accepted by the responsible party
5. Deployment
Use staged rollout and risk-proportionate capability restrictions; configure monitoring and incident response
Deployment specification; capability-restriction specification; monitoring configuration; incident-response plan
Staging tests passed; monitoring active; incident-response and rollback procedures established
6. Monitoring
Monitor behaviour and anomalies; respond to incidents; re-evaluate after model, environment, policy, or tool changes
Monitoring dashboards; incident logs; periodic evaluation reports; change-triggered evaluation records
Service metrics within approved bounds; incident rate below the defined threshold; evaluations completed on schedule and after triggering changes
Table 5: Trustworthy Agent Development Lifecycle (TADL): phases, trust activities, required artefacts, and gate criteria.
Figure 1: Conceptual structure of the Trustworthy Agent Development Lifecycle (TADL). The six phases proceed left to right, separated by trust gates (orange diamonds) whose criteria are set out in Table 5 . A gate that is not passed returns the system to the preceding phase for revision (dashed arcs below). The dashed arc above represents continuous re-evaluation, in which monitoring feeds back into specification. The figure depicts the proposed structure of the framework only; it is not evidence that the arrangement shown improves any trust outcome.
Feature
NIST AI RMF [ 52 ]
ISO/IEC 42001 [ 35 ]
TADL
Scope
General AI systems
Organisational AI management systems
LLM-based agentic systems
Agentic specificity
Low
Low
High
Agent threat modelling
Not explicit
Not explicit
Explicit: injection, memory, and cascading failures
Tool-permission design
Not explicit
Not explicit
Required design artefact
Trust gates
Not prescribed
Management review and audit processes
Explicit phase-specific gates
Alignment coverage
General risk treatment
General management controls
Five explicit trust dimensions
Table 6: Comparison of TADL with established AI risk and management frameworks. TADL is an author-proposed framework that has not yet been empirically validated.
Dimension
Gap
Safety
No standardised prompt-injection benchmark across architectures; AgentDojo [ 12 ] and InjecAgent [ 77 ] cover limited threat models.
Alignment
No benchmark evaluates goal drift during sustained operation. SWE-bench and WebArena use bounded benchmark episodes rather than deployments lasting hours or days.
Transparency
No standardised CoT-faithfulness metric; current measures are task- and model-specific.
Privacy
Benchmarks for attacks on agent memory remain nascent [ 69 ] .
Governance
No compliance-testing framework evaluates agentic systems across regulatory regimes.
Cross-cutting
AgentBench spans heterogeneous environments, so a single aggregate score may obscure task-specific behaviour [ 46 ] .
Table 7: Key evaluation and benchmarking gaps in trustworthy agentic AI.
Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.
Jinhu Qi, Muzhi Li, Jiahong Liu +9
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
The transition from passive foundation models to autonomous, goal-directed agentic AI systems has introduced unprecedented capabilities by coupling recursive cognitive reasoning loops, persistent memory architectures, live tool execution planes, and multi-agent collaboration topologies. However, granting probabilistic neural cores execution authority across filesystems, networks, and cloud infrastructure dissolves classical security perimeters: natural language simultaneously serves as input data, internal control code, and communication protocols, exposing a Turing-complete blast radius where untrusted data represents executable instructions. This survey delivers a comprehensive systems-security reference framework for trustworthy agentic AI, synthesizing 206 foundational studies and regulatory standards. We formalize the general agent architecture as a stateful 5-tuple and establish a 6-dimensional trustworthiness taxonomy covering security, safety, privacy, explainability, fairness, and accountability. We systematically analyze threat surfaces across intra-execution loops and interaction planes, formulate a multi-layered zero-trust defense-in-depth architecture integrating Dual-LLM isolation, Capability-Based Access Control, kernel eBPF probes, and sandboxed runtimes, review standardized evaluation benchmarks, and map technical controls to international AI governance frameworks.
Seyedakbar Mostafavi
Department of Computer Engineering, Yazd University, Yazd 89195-741, Iran
Agentic large language model systems that autonomously invoke tools, maintain persistent memory, and execute multi-step plans are increasingly deployed in public-facing domains, including government services, healthcare triage, and financial advising. We ask whether the frameworks used to build these systems provide architectural-level structural safety guarantees. Applying six containment principles derived from a compositional model of agentic architectures, we audit three dominant frameworks (LangChain, AutoGPT, and OpenAI Agents SDK) and find no native compliance in any of them. Memory integrity, a defense against one of the most prevalent vulnerability classes, is not observed in any of the three evaluated frameworks. We validate these findings empirically: in a simulated government benefits agent built on LangChain, a single memory-poisoning write induces persistent targeted corruption across all tested seeds and backends, increasing the wrongful denial rate for targeted applicants to 88.9%. Under a complex five-factor policy, the same attack preserves aggregate accuracy while increasing targeted wrongful denials by 3.5x, rendering the corruption difficult to detect through standard monitoring. We then introduce two lightweight containment mechanisms: a memory integrity validator and a policy gate, which eliminate both attack vectors with sub-millisecond overhead (<0.2ms per call). We conclude that the current agentic framework ecosystem may not yet meet secure-by-default expectations for public-facing deployments and outline priority architectural interventions to enable trustworthy deployment in high-stakes, socially impactful applications.
Md Jafrin Hossain, Mohammad Arif Hossain, Weiqi Liu +1
Department of Electrical and Computer Engineering, Florida International University, Miami, FL, USA · Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ, USA