Agent Failure Analysis

Follow this topic Ask about this topic

Agent failure analysis examines how errors arise and spread in tool-using, coding, GUI, and multi-agent systems, including failures hidden by apparently successful final outcomes. Common approaches annotate execution trajectories, classify failure modes, monitor runtime behavior, and use dependency graphs or counterfactual replay to distinguish root causes from downstream symptoms. Benchmarks use real or human-labeled traces, controlled fault injection, and repeated runs to test diagnosis and recovery. A central challenge is making attributions genuinely causal and robust when logs are incomplete, task conditions vary, and benchmark shortcuts can make performance look stronger than it is.

Latest papers

154 papers27 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
Feb 23, 2026Jin Jia, Zhiling Deng, Zhuangbin Chen +2Multi-Agent LLM Systems LLM Agent Reliability
Sun Yat-sen University Zhuhai, Guangdong, China
Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks
Feb 22, 2026Wilson Y. LeeAgent Failure Analysis LLM Agent Reliability
Independent Researcher United States
Wink: Recovering from Misbehaviors in Coding Agents
Feb 19, 2026Rahul Nanda, Chandra Maddila, Smriti Jha +3LLM Agent Reliability Agent Reliability
Meta Platforms, Inc. New York, NY, USA · Meta Platforms, Inc. Bellevue, WA, USA · Meta Platforms, Inc. Menlo Park, CA, USA
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
Feb 2, 2026Shraddha Barke, Arnav Goyal, Alind Khare +3Fault Diagnosis AI Agent Reliability
1Microsoft Research · 2Microsoft · 3UIUC

Feb 23, 2026J/K move · Enter open · S save

MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems

Jin Jia, Zhiling Deng, Zhuangbin Chen, Yingqi Wang+1

Sun Yat-sen University Zhuhai, Guangdong, China

Multi-Agent LLM Systems LLM Agent Reliability AI Agent Evaluation Agent Failure Analysis

Open paper AskPDF arXiv

As LLM-based Multi-Agent Systems (MAS) are increasingly deployed for complex tasks, ensuring their reliability has become a pressing challenge. Since MAS coordinate through unstructured natural language rather than rigid protocols, they are prone to semantic failures (e.g., hallucinations, misinterpreted instructions, and reasoning drift) that propagate silently without raising runtime exceptions. Prevailing evaluation approaches, which measure only end-to-end task success, offer limited insight into how these failures arise or how effectively agents recover from them. To bridge this gap, we propose MAS-FIRE, a systematic framework for fault injection and reliability evaluation of MAS. We define a taxonomy of 15 fault types covering intra-agent cognitive errors and inter-agent coordination failures, and inject them via three non-invasive mechanisms: prompt modification, response rewriting, and message routing manipulation. Applying MAS-FIRE to three representative MAS architectures, we uncover a rich set of fault-tolerant behaviors that we organize into four tiers: mechanism, rule, prompt, and reasoning. This tiered view enables fine-grained diagnosis of where and why systems succeed or fail. Our findings reveal that stronger foundation models do not uniformly improve robustness. We further show that architectural topology plays an equally decisive role, with iterative, closed-loop designs neutralizing over 40% of faults that cause catastrophic collapse in linear workflows. MAS-FIRE provides the process-level observability and actionable guidance needed to systematically improve multi-agent systems.

Latest papers

MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems

Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks

Wink: Recovering from Misbehaviors in Coding Agents

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories