Agent Failure Analysis

Follow this topic Ask about this topic

Agent failure analysis examines how errors arise and spread in tool-using, coding, GUI, and multi-agent systems, including failures hidden by apparently successful final outcomes. Common approaches annotate execution trajectories, classify failure modes, monitor runtime behavior, and use dependency graphs or counterfactual replay to distinguish root causes from downstream symptoms. Benchmarks use real or human-labeled traces, controlled fault injection, and repeated runs to test diagnosis and recovery. A central challenge is making attributions genuinely causal and robust when logs are incomplete, task conditions vary, and benchmark shortcuts can make performance look stronger than it is.

Latest papers

154 papers27 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution
Aug 19, 2026Ting-Wei Li, Yuanchen Bei, Xiao Lin +1Failure Attribution for LLM Agents Agent Failure Analysis
University of Illinois Urbana-Champaign, IL USA
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
Aug 18, 2026Tianchen Guan, Xinlei Lin, Royce Cheng-Yue +2Computer-Use Agent Benchmarks Computer-Use Agents
Duke University · Amazon AGI SF Lab
No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage
Aug 15, 2026Shiven KhurdiAI Agent Reliability Agent Failure Analysis
Northeastern University
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
Aug 14, 2026Yanlin Fei, Nazhou Liu, Xinmiao Yu +7Scientific Agent Benchmarks Autonomous Research Agents
Prentis AI · Stanford University · Titan Holdings
Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence
Aug 13, 2026Varun Pratap Bhardwaj, Garima Singh, Arun Pratap BhardwajAgent Failure Analysis AI Agent Reliability
Qualixar / Independent Researcher, India · Independent Researcher, India
Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction
Aug 12, 2026Pan Wang, Yihao Hu, Hang Wang +6Agent Failure Analysis LLM Self-Correction
Ant International
ASCon: A Direction-Aware Reciprocal Agent--Step Contextualization Model for Failure Attribution in Multi-Agent Systems
Aug 11, 2026Shuyu Jiang, Yue Ran, Kaiyu Xu +5Agent Failure Analysis Multi-Agent LLM Systems
1Sichuan University, China · 2Nanyang Technological University, Singapore
TRACE: TRajectory Attribution for Automated Context Engineering
Aug 10, 2026Yikai Zhao, Pradeep Kumar Misra, Saurabh PandeyAgent Failure Analysis AI Agent Benchmarks
Amazon Dallas, TX, USA · Amazon Seattle, WA, USA
TRACE: A Multi-Layer Benchmark for Human AI Controller Coordination Under Drift and Failure
Aug 7, 2026Joshua Zuniga, Srinivasan Subramanian, Ramya Madhuri Narapureddy +1Agent Failure Analysis AI Agent Benchmarks
College of Computing and Software Engineering Kennesaw State University Marietta, GA, USA
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Aug 6, 2026Yunjia Qi, Zehua Yin, Xintong Shi +10Agent Failure Analysis Long-Horizon Agent Evaluation
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University
OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality
Aug 5, 2026Yidian Chen, Yingzi Gu, Natan Vidra +2Agent Failure Analysis AI Agent Reliability
Anote
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
Aug 5, 2026Zhixiang Liang, Yifei Liu, Yidan Huang +5Agent Failure Analysis AI Agent Reliability
University of Illinois Urbana-Champaign · 2Joy Future Academy, JD
The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents
Aug 4, 2026Mohsen ArjmandiAgent Failure Analysis AI Agent Reliability
Independent researcher
CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits
Aug 3, 2026Wei Wang, Shuanghe Liu, Zhu Zhuo +4Agent Failure Analysis Multi-Agent LLM Systems
ByteDance
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
Jul 31, 2026Weijia Zhang, Kunlun Zhu, Zeyi Liu +8Agent Failure Analysis Computer-Use Agents
University of Illinois Urbana-Champaign · 2Yale University
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
Jul 30, 2026Harsh Raj, Vipul Gupta, Anas Mahmoud +4Agent Failure Analysis LLM Agent Evaluation
Scale AI
How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
SkillMentor: LLM Agent Self-Evolution via Learning Blind-Spot Diagnosis
Jul 29, 2026Xiaoyi Bao, Yuanzhen Xie, Yunzhi Tan +5Agent Failure Analysis LLM Agent Self-Improvement
The Hong Kong Polytechnic University · Work done during an internship at Tencent. · Platform and Content Group, Tencent +1
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
Jul 27, 2026Xueping Gao, Jianwei Yang, Qiang YangAgent Failure Analysis LLM Self-Correction
Alibaba Cloud Computing Hangzhou, China
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
Jul 21, 2026Kunlun Zhu, Xuyan Ye, Zhiguang Han +9Agent Failure Analysis AI Agent Reliability
University of Illinois Urbana-Champaign · University of Toronto · Google +1
Importance Sampling and PCA for Finding Failures in Commercial Autonomous Vehicles
Jul 20, 2026Hailey Warner, Duncan Eddy, Shreya Parjan +6Agent Failure Analysis Autonomous Driving Safety Evaluation
Department of Aeronautics and Astronautics, Stanford University, Palo Alto, CA · Torc Robotics, Blacksburg, VA
Fantastic Adaptive Taxonomies and How to Use Them
Jul 17, 2026Mert Cemri, Andrei Cojocaru, Melissa Pan +9Agent Failure Analysis LLM Agent Evaluation
University of California, Berkeley · Apple · Bespoke Labs
How Far Can Root Cause Analysis Go on Real-World Telemetry Data?
Jul 15, 2026Athira Gopal, Ashwanth KrishnanAgent Failure Analysis Root Cause Analysis
QPIAI India
Tracing Agentic Failure from the Flow of Success
Jul 14, 2026Samuel Yeh, Yiwen Zhu, Shaleen Deep +1Agent Failure Analysis Failure Attribution for LLM Agents
Department of Computer Science, University of Wisconsin-Madison · Microsoft Research
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jul 10, 2026Jiale Liu, Huajun Xi, Shaokun Zhang +6Agent Failure Analysis LLM Agent Evaluation
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Jul 10, 2026Xiangxin Zhao, Han Li, Shuaiting Li +4Agent Failure Analysis Coding Agents
University College London, London, United Kingdom · Nanjing University, Nanjing, China
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
Jul 8, 2026Yufei Xia, Anjun Gao, Yueyang Quan +2Agent Failure Analysis Multi-Agent LLM Systems
University of Louisville · University of North Texas
From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization
Jul 8, 2026Ying Chang, Jiahang Xu, Xuan Feng +3Agent Failure Analysis Large Language Model-Guided Optimization
University of Chinese Academy of Sciences, Beijing, China · Microsoft Research.
Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents
Jul 8, 2026Vikas Reddy, Sumanth Reddy Challaram, Abhishek BasuAgent Failure Analysis AI Agent Reliability
Independent Researcher · Indian Institute of Technology Kharagpur Kharagpur, India · Massachusetts Institute of Technology Cambridge, USA
Verification of Adaptive Agentic Controllers through Finite Rule Revision
Jul 7, 2026Roberto GarroneAgent Failure Analysis Formal Verification
Open University of Cyprus

Aug 19, 2026J/K move · Enter open · S save

Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution

Ting-Wei Li, Yuanchen Bei, Xiao Lin, Hanghang Tong

University of Illinois Urbana-Champaign, IL USA

Failure Attribution for LLM Agents Agent Failure Analysis AI Agent Reliability Multi-Agent LLM Systems Graph Neural Networks

Open paper AskPDF arXiv

Large language model (LLM)-based multi-agent systems (MAS) often exhibit complex failure modes, which frequently cause agents to produce incorrect outcomes. This motivates the task of Agent Failure Attribution: given a failed multi-agent trajectory, identify the faulty agents and their corresponding error types. Existing approaches predominantly rely on LLMs to perform failure attribution, either through direct prompting, fine-tuning on synthetic data or complex agentic pipelines. While effective, these methods incur substantial computational overhead due to long-context processing, expensive post-training and handcrafted workflows. Moreover, empirical evidence shows that even state-of-the-art models achieve limited accuracy on existing benchmarks, suggesting that scaling model size alone is insufficient. In this work, we revisit this task and question the necessity of such expensive generative solutions. We introduce AFANet, a lightweight graph-based framework that models interaction trajectories through step-level semantic signals and agent-level relationships. We show that with significantly fewer parameters and near-zero inference cost, AFANet (i) matches or outperforms LLM-based baselines, including fine-tuned models on in-domain benchmarks, (ii) maintains robust performance across different GNN architectures and (iii) can be further improved with inexpensive test-time adaptation on the OOD benchmark. Our results suggest that effective agent failure attribution does not require heavy LLM reasoning and a lightweight, structured approach can achieve strong performance.