Failure Attribution for LLM Agents

LLM: Large Language Model

Follow this topic Ask about this topic

Latest papers

52 papers25 papers in the last four weeks, up 733% on the four weeks before. 0.2% of all new papers.

Divide, Diagnose, and Compose: Optimizing Agentic Pipelines and Harnesses
Nov 5, 2026Divij Handa, Christopher Tensmeyer, Jihyung Kil +1LLM Agent Workflow Optimization Agent Harness Optimization
Arizona State University · Adobe
Signals: Continuous Diagnostics of AI Agents
Nov 3, 2026Amey Rambatla, Vatsal Goel, Sagar Gaur +3AI Agent Monitoring AI Agent Evaluation
Cisco
Treating the Symptom, Missing the Cause: Anatomy of a Scientific Reasoning Failure
Nov 3, 2026Anna Gogleva, Thomas Charman, Martin Balla +1Scientific Reasoning in Language Models Causal Reasoning in Language Models
Relation Therapeutics
Interpretable Early Termination of Web Navigation Agents via Closed Sequential Pattern Mining
Nov 3, 2026Sergio Talavera, Magdalini EirinakiWeb Agents Failure Attribution for LLM Agents
Department of Computer Engineering San Jose State University San Jose, CA 95192
Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction
Nov 2, 2026Yiwei Zha, Qingyu Meng, Herbert Bos +1AI Agent Safety Benchmarks LLM Agent Safety
VUSec, Department of Computer Science, Vrije Universiteit Amsterdam
From Tool State to Scientific Action: Auditing Decision Provenance in Biomedical Research Agents
Nov 1, 2026Xingyu Yi, Xinya Zhang, Yunshen Zhou +1AI Agent Auditing Data Provenance
INTOSYN AI Labs · Columbia University · Microsoft
Hacking or Just Failing? A More Capable LLM Monitor Better Separates Reward Hacking from Honest Failure
Oct 30, 2026Kaushik KezendiraneAI Agent Monitoring Reward Hacking
Independent researcher
ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
Oct 29, 2026Jeremy Canale, Fabien SoulisAI Agent Evaluation Failure Attribution for LLM Agents
Characterizing LLM Agent Behaviors in Module Shadowing Environments
Oct 29, 2026Shuai Yang, Enmeng Lu, Jinyu Fan +7LLM Agent Security AI Agent Auditing
Gaoling School of AI, Renmin University of China, Beijing, China · Beijing Institute of AI Safety and Governance, Beijing, China · School of Humanities, University of Chinese Academy of Sciences, Beijing, China +1
Beyond fabrication: Misattributed numbers in papers written by AI research agents
Oct 28, 2026Mosab Alfaqeeh, Omar Mohammed Horani, Mou'th Hourani +1Claim Verification AI Agent Auditing
Al-Ahliyya Amman University, Amman, Jordan · Binghamton University, State University of New York, Binghamton, NY, USA
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Oct 26, 2026Jialiang Liu, Shusheng Li, Wenjun TanLong-Horizon Agent Evaluation Automated Agent Evaluation
College of Information Science and Engineering, Northeastern University, China · Key Laboratory of Intelligent Computing of Medical Images, Ministry of Education, Northeastern University, China. · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China +1
When Agents Fail: Model or Harness--Who's to Blame?
Oct 23, 2026Shao Zhang, Hangfan Zhang, Yuanyuan Shi +9LLM Agent Evaluation LLM Agent Harnesses
Shanghai Artificial Intelligence Laboratory
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Oct 23, 2026Zhiyong Cao, Hengjie Zhu, Yongxin Wang +20LLM Agent Harnesses LLM Agent Reliability
Institute of Automation, CAS · Zhongguancun Academy · Institute of Information Engineering, CAS +2
Understanding Communication Failures in Multi-Agent LLM Systems
Oct 23, 2026Priya Pitre, Jun Hou, Yi Fang +2Multi-Agent LLM Systems Multi-Agent Communication
Virginia Tech, Blacksburg, VA 24061, USA
Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents
Oct 21, 2026Zixuan Yu, Zhenheng Tang, Xiaowen Chu +1LLM Agent Reliability Process-Based Agent Evaluation
School of Computer science and engineering, Sun Yat-sen University · CSE, The Hong Kong University of Science and Technology · DSA Thrust, The Hong Kong University of Science and Technology (GuangZhou) +1
Alpha: Systematic Fault Injection and Robustness Evaluation for LLM-Driven Embodied Manipulation
Oct 21, 2026Jona JoyLarge Language Model-Based Robot Planning Failure Attribution for LLM Agents
independent robotics and artificial intelligence research engineer
A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems
Oct 20, 2026Wendy Zheng, Liang Wu, Yinhan He +3Failure Attribution for LLM Agents
University of Virginia · Nokia
OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants
Oct 20, 2026Ruoyu Zhang, Zhaocen Liu, Weiyi Tian +2LLM Agent Evaluation LLM Agents
University of British Columbia · University of Chicago · Independent Researcher +1
Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Oct 19, 2026Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu +2LLM Tool Use LLM Agent Reliability
University of California, Riverside · University of California, Irvine
From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs
Oct 19, 2026Hengyu Ye, Songhua HuCounterfactual Evaluation Failure Attribution for LLM Agents
ShanghaiTech University · City University of Hong Kong
From Trace to Fix: Residual Task State in Long-Horizon Agents
Oct 19, 2026Zile Huang, Ser-nam LimLLM Agent Reliability LLM Agent Planning
University of Central Florida
Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing
Oct 18, 2026Junhao Liu, Tianxiao Li, Yongxian Wu +4LLM Auditing AI Agent Auditing
University of California, Irvine · NEC Labs America
AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation
Oct 17, 2026Congjing Zhang, Siyuan Xu, Yunfei Bai +3Multi-Agent LLM Systems Multi-Agent Systems
Department of Industrial & Systems Engineering, University of Washington · Department of Electrical & Computer Engineering, Texas A&M University · Amazon
Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures
Oct 15, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeLLM Agents Failure Attribution for LLM Agents
Dipartimento di Informatica, Università di Pisa, Pisa, Italy
BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems
Oct 13, 2026Xiaoguang Guo, Yaohao Long, Chuxu ZhangFailure Attribution for LLM Agents Agent Failure Analysis
University of Connecticut
Auditing Failure Attribution: Label Predictability and Judge Behavior
Oct 13, 2026Tianyang Xu, Jing GaoFailure Attribution for LLM Agents AI Agent Evaluation
Purdue University
Structural Baselines Matter for Agent Failure Detection and Attribution
Oct 12, 2026Ramesh Arvind Naagarajan, Stefan StreifAgent Failure Analysis Failure Attribution for LLM Agents
Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany
Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels
Oct 9, 2026Do Nguyen-Thanh, Junlin Fang, Bing Li +2Failure Attribution for LLM Agents Agent Failure Analysis
College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, University of Technology Sydney
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Failure Attribution for LLM Agents Agent Failure Analysis
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China
ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems
Oct 8, 2026Weilin Jin, Mingyu Wang, Taiyu Zhu +7Failure Attribution for LLM Agents Agent Failure Analysis
Peking University, Beijing, China · Joy Future Academy, Beijing, China · Tsinghua University, Beijing, China

Nov 5, 2026J/K move · Enter open · S save

Divide, Diagnose, and Compose: Optimizing Agentic Pipelines and Harnesses

Divij Handa, Christopher Tensmeyer, Jihyung Kil, Vlad I. Morariu

Arizona State University · Adobe

LLM Agent Workflow Optimization Agent Harness Optimization LLM Agent Efficiency Failure Attribution for LLM Agents

Open paper AskPDF arXiv

Building agentic pipelines that achieve high task performance while keeping cost and latency low requires jointly optimizing heterogeneous and coupled design choices, including prompts, topology, model selection, and tool policies. Manually searching this combinatorial design space is expensive and time-consuming, while existing automated optimizers typically operate within a predefined subset of this space, modify the pipeline monolithically, or provide little structured evidence for pipeline changes. To mitigate these challenges, we introduce the Decomposed Credit-Assignment Framework (DeCAF), which adapts the design space to the given benchmark and pipeline, optimizes each dimension separately, and generates reasoning traces for every decision. Given a task specification, benchmark examples, and a seed pipeline, DeCAF identifies relevant optimization dimensions and generates diverse experimental hypotheses for each. Independent coding-agent sessions then optimize each dimension under a restricted edit scope, jointly evaluating performance, inference cost, and latency while recording all hypotheses, edits, and outcomes of each experiment. A final synthesis session combines compatible findings across all dimensions and uses a component-level trajectory evaluator, which derives role-specific rubrics from each agent's instructions, to localize underperforming components and guide cross-dimension revisions. Across a diverse suite of agentic benchmarks, DeCAF improves average task performance by 2.58 percentage points, while reducing inference cost by 4.30% and end-to-end latency by 42.45% compared to the best overall baseline.