Failure Attribution for LLM Agents

LLM: Large Language Model

Follow this topic Ask about this topic

Latest papers

34 papers12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Oct 19, 2026Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu +2LLM Tool Use LLM Agent Reliability
University of California, Riverside · University of California, Irvine
From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs
Oct 19, 2026Hengyu Ye, Songhua HuCounterfactual Evaluation Failure Attribution for LLM Agents
ShanghaiTech University · City University of Hong Kong
From Trace to Fix: Residual Task State in Long-Horizon Agents
Oct 19, 2026Zile Huang, Ser-nam LimLLM Agent Reliability LLM Agent Planning
University of Central Florida
Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing
Oct 18, 2026Junhao Liu, Tianxiao Li, Yongxian Wu +4LLM Auditing AI Agent Auditing
University of California, Irvine · NEC Labs America
AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation
Oct 17, 2026Congjing Zhang, Siyuan Xu, Yunfei Bai +3Multi-Agent LLM Systems Multi-Agent Systems
Department of Industrial & Systems Engineering, University of Washington · Department of Electrical & Computer Engineering, Texas A&M University · Amazon
Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures
Oct 15, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeLLM Agents Failure Attribution for LLM Agents
Dipartimento di Informatica, Università di Pisa, Pisa, Italy
BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems
Oct 13, 2026Xiaoguang Guo, Yaohao Long, Chuxu ZhangFailure Attribution for LLM Agents Agent Failure Analysis
University of Connecticut
Auditing Failure Attribution: Label Predictability and Judge Behavior
Oct 13, 2026Tianyang Xu, Jing GaoFailure Attribution for LLM Agents AI Agent Evaluation
Purdue University
Structural Baselines Matter for Agent Failure Detection and Attribution
Oct 12, 2026Ramesh Arvind Naagarajan, Stefan StreifAgent Failure Analysis Failure Attribution for LLM Agents
Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany
Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels
Oct 9, 2026Do Nguyen-Thanh, Junlin Fang, Bing Li +2Failure Attribution for LLM Agents Agent Failure Analysis
College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, University of Technology Sydney
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Failure Attribution for LLM Agents Agent Failure Analysis
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China
ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems
Oct 8, 2026Weilin Jin, Mingyu Wang, Taiyu Zhu +7Failure Attribution for LLM Agents Agent Failure Analysis
Peking University, Beijing, China · Joy Future Academy, Beijing, China · Tsinghua University, Beijing, China
DeFA: Dependency-Guided Failure Attribution for LLM Agents
Oct 1, 2026Bo Deng, Xinlei Zheng, Yi Wei +6Agent Failure Analysis LLM Agent Evaluation
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University
What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents
Sep 30, 2026Shuyang Zhang, Jianshuo ChangLLM Agent Evaluation Failure Attribution for LLM Agents
The Hong Kong Polytechnic University
DAAF: From Failure Localization to Editable System Assets in LLM Agents
Sep 26, 2026Xiaoyang Yuan, Qi Liu, Yubin Ruan +9Failure Attribution for LLM Agents Agent Failure Analysis
Tencent
DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
Sep 4, 2026Zehao Wang, Lanjun Wang, Shilong Jin +2Agent Failure Analysis Multi-Agent LLM Systems
College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China +1
EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems
Sep 1, 2026Jun Hou, Priya Pitre, Yi Fang +1Agent Failure Analysis Multi-Agent LLM Systems
Virginia Tech
Cascade Failure Attribution for Multi-Agent Systems
Aug 30, 2026Jiayi Zhang, Zexin Wang, Degang Sun +4Agent Failure Analysis Multi-Agent LLM Systems
Computer Network Information Center, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution
Aug 19, 2026Ting-Wei Li, Yuanchen Bei, Xiao Lin +1Failure Attribution for LLM Agents Agent Failure Analysis
University of Illinois Urbana-Champaign, IL USA
ASCon: A Direction-Aware Reciprocal Agent--Step Contextualization Model for Failure Attribution in Multi-Agent Systems
Aug 11, 2026Shuyu Jiang, Yue Ran, Kaiyu Xu +5Agent Failure Analysis Multi-Agent LLM Systems
1Sichuan University, China · 2Nanyang Technological University, Singapore
HPFA: Hypergraph-Based Paired Failure Attribution for LLM Reasoning
Aug 3, 2026Runchuan Zhu, Hongbin Lai, Bowen Jiang +4LLM Reasoning Reasoning Trace Analysis
National University of Singapore · Peking University · University of Texas at Austin +1
Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines
Jul 24, 2026Santhiya RajanLLM Auditing LLM Agent Reliability
Multiverse Computing
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
Jul 21, 2026Kunlun Zhu, Xuyan Ye, Zhiguang Han +9Agent Failure Analysis AI Agent Reliability
University of Illinois Urbana-Champaign · University of Toronto · Google +1
Tracing Agentic Failure from the Flow of Success
Jul 14, 2026Samuel Yeh, Yiwen Zhu, Shaleen Deep +1Agent Failure Analysis Failure Attribution for LLM Agents
Department of Computer Science, University of Wisconsin-Madison · Microsoft Research
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jul 10, 2026Jiale Liu, Huajun Xi, Shaokun Zhang +6Agent Failure Analysis LLM Agent Evaluation
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
Jun 23, 2026Chenyang Zhu, Jiayu Yao, Kushal Chawla +10Agent Failure Analysis Long-Horizon Agent Evaluation
1AI Foundations, Capital One · 2General Intelligence Com · Department of Engineering Sciences and Ap
REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces
Jun 8, 2026Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok +4LLM Agent Evaluation LLM Agents
University of California, Los Angeles, USA · Cisco, USA · University of Mississippi, USA
Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures
Jun 6, 2026Jaineet ShahAgent Failure Analysis AI Agent Reliability
SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
May 31, 2026Zhuoyun Yu, Xin Xie, Wuguannan Yao +4LLM Agent Skill Learning Failure Attribution for LLM Agents
♠Zhejiang University · ♣Ant Digital Technologies, Ant Group · ♢Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph +1

Oct 19, 2026J/K move · Enter open · S save

Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi+1

University of California, Riverside · University of California, Irvine

LLM Tool Use LLM Agent Reliability LLM Agent Harnesses Agentic RAG Failure Attribution for LLM Agents

Open paper AskPDF arXiv

LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced. Agents use it to answer 158 execution-verified multiple-choice items. When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running all 11,007 logged simulator calls shows that both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not. Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.