Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
Organizations: Department of Human-Centered Computing, Indiana University Indianapolis, 755 W. Michigan Street, Indianapolis, Indiana, 46202, USA · Department of Electrical and Computer Engineering, Purdue University Northwest, 2200 169th Street, Hammond, Indiana, 46323, USA · Computer Information Science, Higher Colleges of Technology, Abu Dhabi, United Arab Emirates · Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, 333 Cedar Street, New Haven, Connecticut, 06510, USA
Abstract
Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.
Figures & tables
| Survey | Arch. | Safety/ Security | Alignment/ Oversight | Transparency | Privacy | Governance | Life cycle | Agent Evaluation |
|---|---|---|---|---|---|---|---|---|
| Yu et al. (2025) [ 76 ] | P | S | P | P | S | L | – | P |
| Raza et al. (2026) [ 58 ] | P | S | P | P | P | S | P | P |
| Abou Ali et al. (2026) [ 1 ] | S | P | L | L | L | P | – | P |
| Gan et al. (2024) [ 28 ] | P | S | L | P | S | L | – | L |
| This survey | P | S | S | S | S | S | S | P |
| Architecture | Trust Challenges | Examples |
|---|---|---|
| Single-Agent ReAct | Goal drift; tool misuse; context-length limits | ReAct [ 74 ] |
| Hierarchical Multi-Agent | Cascading failures; authority confusion | AutoGen [ 72 ] ; MetaGPT [ 32 ] |
| Collaborative Peer Agents | Convergence on an incorrect view; judge bias | LLM debate [ 44 ] |
| Human-in-the-Loop (HITL) | Approval fatigue; automation bias | HITL agent deployment patterns |
| Fully Autonomous Pipeline | All failure modes amplified | Automated software-engineering agents |
| Dimension | Sub-type | Example | Illus. Sev. |
|---|---|---|---|
| Safety | Unsafe tool use | Agent executes attacker-injected code | C |
| Safety | Cascading failure | Cloud misconfiguration triggers data loss | C |
| Safety | Resource exhaustion | Loop causes 10,000 unintended API calls | H |
| Alignment | Goal misgeneralization | Agent spams to maximise “email responses” | C |
| Alignment | Spec. gaming | Agent marks task complete without solving it | H |
| Alignment | Instruction misinterp. | Agent deletes active files matching “old” | H |
| Juris. | Instrument / Framework | Legal Status | Binding? | Applicability to Agentic AI | Key Effective Date |
|---|---|---|---|---|---|
| EU | AI Act [ 21 ] (Reg. (EU) 2024/1689) | Binding law (in force), as amended by Reg. (EU) 2026/1744 | Yes | Agentic systems classified by purpose/domain/role, not architecture | General application 2 Aug 2026 |
| EU | Digital Omnibus on AI [ 22 ] (Reg. (EU) 2026/1744) | Binding law (in force) | Yes | Amends the AI Act; defers high-risk obligations and adds new Art. 5 prohibitions | In force 27 Jul 2026; Annex III 2 Dec 2027; Annex I 2 Aug 2028 |
| EU | Product Liability Directive 2024/2853 [ 20 ] | Adopted; transposition pending | Yes (on transposition) | Covers software and AI-enabled products; defect liability | 9 Dec 2026 (transposition deadline) |
| EU | AI Liability Directive (COM(2022) 496) [ 17 ] | Proposal withdrawn (Oct 2025) | N/A | No longer advancing; coverage remains via PLD and national law | Withdrawn 6 Oct 2025 |
| USA | EO 14179 [ 24 ] | Active EO following the separate revocation of EO 14110 | EO only | Deregulatory emphasis; AI Action Plan mandated; sector agency guidance persists | Signed 23 Jan 2025 |
| USA | America’s AI Action Plan [ 26 ] | Federal policy plan (issued under EO 14179) | No | Three pillars: innovation, infrastructure, international diplomacy and security; no direct obligations on deployers | Released 23 Jul 2025 |
| Phase | Key Trust Activities | Required Artefacts | Gate Criteria |
|---|---|---|---|
| 1. Specification | Define task scope, principal hierarchy, risk tier, and regulatory requirements; model agent-specific threats | Risk assessment; principal-hierarchy document; threat model; compliance checklist | Risk tier confirmed; applicable regulations identified; threat model reviewed; oversight requirements established |
| 2. Design | Select architecture; design tool permissions, memory governance, human oversight, and privacy protections | Architecture specification; tool-permission matrix; memory-governance policy; oversight plan; privacy-impact assessment | Safety and privacy reviews passed; tool permissions and oversight plan approved by relevant stakeholders |
| 3. Training & Fine-Tuning | Conduct alignment and safety training; improve robustness to distribution shift; document data provenance | Training-data provenance record; alignment-procedure documentation; safety-evaluation results | Alignment evaluation passed on held-out scenarios; robustness thresholds met; provenance documented |
| 4. Evaluation | Conduct red-teaming, prompt-injection testing, goal-misgeneralization probing, privacy-attack simulation, and multi-agent testing | Red-team report; benchmark results; failure-mode analysis; privacy-audit report; residual-risk register | No unresolved critical failures; thresholds met; residual risks documented and accepted by the responsible party |
| 5. Deployment | Use staged rollout and risk-proportionate capability restrictions; configure monitoring and incident response | Deployment specification; capability-restriction specification; monitoring configuration; incident-response plan | Staging tests passed; monitoring active; incident-response and rollback procedures established |
| 6. Monitoring | Monitor behaviour and anomalies; respond to incidents; re-evaluate after model, environment, policy, or tool changes | Monitoring dashboards; incident logs; periodic evaluation reports; change-triggered evaluation records | Service metrics within approved bounds; incident rate below the defined threshold; evaluations completed on schedule and after triggering changes |
| Feature | NIST AI RMF [ 52 ] | ISO/IEC 42001 [ 35 ] | TADL |
|---|---|---|---|
| Scope | General AI systems | Organisational AI management systems | LLM-based agentic systems |
| Agentic specificity | Low | Low | High |
| Agent threat modelling | Not explicit | Not explicit | Explicit: injection, memory, and cascading failures |
| Tool-permission design | Not explicit | Not explicit | Required design artefact |
| Trust gates | Not prescribed | Management review and audit processes | Explicit phase-specific gates |
| Alignment coverage | General risk treatment | General management controls | Five explicit trust dimensions |
| Dimension | Gap |
|---|---|
| Safety | No standardised prompt-injection benchmark across architectures; AgentDojo [ 12 ] and InjecAgent [ 77 ] cover limited threat models. |
| Alignment | No benchmark evaluates goal drift during sustained operation. SWE-bench and WebArena use bounded benchmark episodes rather than deployments lasting hours or days. |
| Transparency | No standardised CoT-faithfulness metric; current measures are task- and model-specific. |
| Privacy | Benchmarks for attacks on agent memory remain nascent [ 69 ] . |
| Governance | No compliance-testing framework evaluates agentic systems across regulatory regimes. |
| Cross-cutting | AgentBench spans heterogeneous environments, so a single aggregate score may obscure task-specific behaviour [ 46 ] . |