Agent Failure Analysis

Follow this topic Ask about this topic

Agent failure analysis examines how errors arise and spread in tool-using, coding, GUI, and multi-agent systems, including failures hidden by apparently successful final outcomes. Common approaches annotate execution trajectories, classify failure modes, monitor runtime behavior, and use dependency graphs or counterfactual replay to distinguish root causes from downstream symptoms. Benchmarks use real or human-labeled traces, controlled fault injection, and repeated runs to test diagnosis and recovery. A central challenge is making attributions genuinely causal and robust when logs are incomplete, task conditions vary, and benchmark shortcuts can make performance look stronger than it is.

Latest papers

154 papers27 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriLLM Agent Reliability Computer-Use Agent Benchmarks
University of California, Santa Cruz · Massachusetts Institute of Technology
Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents
May 18, 2026Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya +1Agent Failure Analysis AI Agent Reliability
Department of Computer Science Cornell University
VerifyMAS: Hypothesis Verification for Failure Attribution in LLM-based Multi-Agent Systems
May 17, 2026Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2Agent Failure Analysis Multi-Agent LLM Systems
Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago
DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents
May 17, 2026Sirui Hong, Zhijie Liu, Tengfei Li +3Agent Failure Analysis GUI Agents
1DeepWisdom · 2Independent Researcher · 3The Hong Kong University of Science and Technology (Guangzhou)
STAR: A Stage-attributed Triage and Repair framework for RCA Agents in Microservices
May 15, 2026Junle Wang, Xingchuang Liao, Wenjun WuAgent Failure Analysis Root Cause Analysis
School of Artificial Intelligence, Beihang University
Holistic Evaluation and Failure Diagnosis of AI Agents
May 14, 2026Netta Madvil, Gilad Dym, Alon Mecilati +12Agent Failure Analysis AI Agent Evaluation
Deepchecks, Ramat Gan, Israel
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
May 13, 2026Priyam Sahoo, Gaurav Mittal, Xiaomin Li +4Agent Failure Analysis Process-Based Agent Evaluation
University of Illinois, Urbana-Champaign, IL, USA · Microsoft, Redmond, WA, USA
The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents
May 10, 2026Baoyuan Wu, Qingshan Liu, Adel Bibi +2Agent Failure Analysis AI Agent Security
Chinese University of Hong Kong, Shenzhen, China · Nanjing University of Posts and Telecommunications, China · University of Oxford, UK +2
Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents
May 9, 2026Chenyu Zhao, Shenglin Zhang, Yihang Lin +7Agent Failure Analysis Software Engineering Agents
Nankai University Tianjin, China · Tsinghua University Beijing, China · Microsoft Redmond, USA
Log analysis is necessary for credible evaluation of AI agents
May 8, 2026Peter Kirgis, Sayash Kapoor, Stephan Rabanser +8Agent Failure Analysis AI Agent Evaluation
Princeton University · Independent · UK AISI +4
Conformal Agent Error Attribution
May 7, 2026Naihe Feng, Yi Sui, Shiyi Hou +2Agent Failure Analysis AI Agent Reliability
Dalhousie University · Halifax, NS, Canada · Layer 6 AI +1
Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems
May 6, 2026Krti TallamAgent Failure Analysis Evidence-Grounded Reasoning
KamiwazaAI
Ambient Persuasion in a Deployed AI Agent: Unauthorized Escalation Following Routine Non-Adversarial Content Exposure
Apr 29, 2026Diego F. Cuadros, Abdoul-Aziz MaigaAgent Failure Analysis AI Agent Security
Digital Epidemiology Laboratory, Digital Futures, University of Cincinnati, Cincinnati, OH, USA · Center for Humanities and Technology (CHaT), University of Cincinnati, Cincinnati, OH, USA · Norwegian University of Science and Technology, Trondheim, Norway
Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows
Apr 28, 2026Shivam Rawat, Lucie FlekAgent Failure Analysis AI for Science
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.
Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents
Apr 28, 2026Jianming Chen, Yawen Wang, Junjie Wang +4Agent Failure Analysis AI Agent Evaluation
Institute of Software, Chinese Academy of Sciences, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China +2
Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents
Apr 27, 2026M. MengAgent Failure Analysis Automated Software Testing
Shenzhen The Nine’s Light Technology Co., Ltd., Shenzhen, Guangdong, China
AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuAgent Failure Analysis AI Agent Evaluation
The University of Hong Kong · Stellaris AI Limited
Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Apr 24, 2026Mengzhuo Chen, Junjie Wang, Fangwen Mu +4Agent Failure Analysis Multi-Agent LLM Systems
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Failure Identification in Imitation Learning Via Statistical and Semantic Filtering
Apr 15, 2026Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste MouretRobot Failure Detection Agent Failure Analysis
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Inria, CNRS, Université de Lorraine, LORIA, F-54000 Nancy, France · Bleu Robotics, Paris, France
The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
Apr 13, 2026Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun +7Long-Horizon Agent Tasks Agent Failure Analysis
University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology
CodeTracer: Towards Traceable Agent States
Apr 13, 2026Han Li, Yifan Yao, Letian Zhu +13AI Coding Agents Agent Trajectory Analysis
Nanjing University · Alibaba Group · Renmin University of China +3
Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
Mar 26, 2026Yeonjun In, Mehrab Tanjim, Jayakumar Subramanian +6Agent Failure Analysis Multi-Agent Systems
KAIST Daejeon, Republic of Korea · Adobe Research San Jose, CA, USA
Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
Mar 25, 2026Myeongsoo Kim, Dingmin Wang, Siwei Cui +6Agent Failure Analysis AI Coding Agents
AWS AI Labs · Monash University
Efficient Failure Management for Multi-Agent Systems with Reasoning Trace Representation
Mar 23, 2026Lingzhe Zhang, Tong Jia, Mingyu Wang +9Agent Failure Analysis Multi-Agent LLM Systems
Peking University Beijing, China · Huawei Technologies Co., Ltd. Beijing, China · Huawei Technologies Co., Ltd. Hong Kong SAR, China +1
CRASH: Cognitive Reasoning Agent for Safety Hazards in Autonomous Driving
Mar 16, 2026Erick Silva, Rehana Yasmin, Ali ShokerAutonomous Driving Safety Evaluation Agent Failure Analysis
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
Mar 16, 2026Zhaohui Geoffrey WangAgent Failure Analysis AI Agent Reliability
USC Viterbi School of Engineering
XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights
Mar 6, 2026Arun JoshiAgent Failure Analysis Explainable Artificial Intelligence
Independent Researcher Islington College (Alumini) Nepal
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
Feb 28, 2026Shu-Xun Yang, Cunxiang Wang, Haoke Zhang +12Agent Failure Analysis Reasoning Trace Analysis
Beijing Institute of Technology, Beijing, China · Zhipu AI, Beijing, China · Tsinghua University, Beijing, China +1
From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems
Feb 27, 2026Yawen Wang, Wenjie Wu, Junjie Wang +1Agent Failure Analysis Multi-Agent LLM Systems
Institute of Software, Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China +2
MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics
Feb 26, 2026Yutong Wang, Siyuan Xiong, Xuebo Liu +4Agent Failure Analysis Multi-Agent Systems
Harbin Institute of Technology, Shenzhen · Alibaba Group

May 19, 2026J/K move · Enter open · S save

AgentAtlas: Beyond Outcome Leaderboards for LLM Agents

Parsa Mazaheri, Kasra Mazaheri

University of California, Santa Cruz · Massachusetts Institute of Technology

LLM Agent Reliability Computer-Use Agent Benchmarks Agent Failure Analysis Benchmark Auditing AI Agent Auditing LLM Agent Evaluation+3

Open paper AskPDF arXiv

Large language model agents now act on codebases, browsers, operating systems, calendars, files, and tool ecosystems, but their evaluations often collapse behavior into final task success. AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol for separating outcome success from control-decision quality and trajectory quality. The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; (iii) a 0/1/2 benchmark-coverage audit over fifteen agent benchmarks; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models under taxonomy-aware and taxonomy-blind prompt formats. The synthetic demonstration is not a public benchmark release and should not be read as a definitive model comparison. Instead, it illustrates two measurement risks: mapped label agreement can change substantially when the explicit label menu is removed, and axis choice can change apparent rankings. AgentAtlas is intended to help benchmark designers state what behavior they cover, and to help evaluators diagnose failures that outcome-only leaderboards hide.