cs.AIAug 28, 2026

Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration

Authors: Xiaoqing Wang, Keman Huang, Bin Liang, Hongyu Li, Xiaoyong Du, Wuqiong Pan

Organizations: Renmin University of China · Ant Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent systems (MAS) powered by large language models have shown promise for complex tasks but suffer from high failure rates. Current self-reflection methods for MAS require all agents to reflect upon failure, overlooking a critical reality: failures typically stem from a specific agent leading the task astray, namely the decisive error agent, while others merely fulfill their regular duties. Forcing regular-behaving agents to reflect contaminates their memory with wrong insights. Hence, we propose DoCtOR (Diagnose-then-Correct PPO-enhanced Reflection), a novel reflection framework that enhances multi-agent collaboration. DoCtOR first identifies the decisive error step and decisive error agent through automated failure attribution, then employs counterfactual reasoning to generate a corrected decisive error step, and finally engages only the decisive error agent to produce targeted reflections. Experimental results show DoCtOR achieves 22%, 26%, and 27% improvements over initial success rates on HotPotQA, ChartQAPro, and Mind2Web datasets, outperforming Reflexion, Retroformer, and COPPER. We further establish the generalizability of our diagnose-then-correct paradigm and demonstrate that in low-resource settings, focusing reflection on reasoning steps after the decisive error step achieves comparable quality to reflecting on the complete failure trajectory.

Figures & tables

Figure 1: A multi-agent collaboration failure process replay in a data analysis task and the detrimental effects of forced reflection on regular-behaving agents.
Figure 2: Overview of DoCtOR framework. Our framework comprises two components: an action module for generating reasoning trajectories, and a reflection module that provides feedback to action module agents and enriches their memory. The reflection module follows a three-stage workflow: (1) diagnosis submodule identifies the decisive error step and agent; (2) correction submodule generates a corrected decisive error step; and (3) reflector model leverages the previous two stages’ outputs to produce targeted reflections for the decisive error agent.
Figure 3: Performance comparison of DoCtOR with baselines for self-reflection .
BaselineMetricHeld-in DatasetHeld-out Dataset
HotPotQAChartQAProMind2WebAlgorithm- GeneratedHand- Crafted
RandomAgent-Level Accuracy0.370.290.540.280.20
Step-Level Accuracy0.060.090.020.180.04
All-at-OnceAgent-Level Accuracy0.540.550.190.490.51
Step-Level Accuracy0.280.450.230.100.05
Step-by-StepAgent-Level Accuracy0.450.420.510.280.42
Table 1: Performance comparison of ProFA with baselines for automated fault attribution .
Figure 4: Generalizability of the diagnose-then-correct reflection paradigm. We evaluate the paradigm on the prompt-based Reflexion method to verify its generalizability beyond our fine-tuning-based implementation.
Figure 5: The absolute distance between the predicted and actual decisive error step .
Figure 6: Effect of failure trajectory scope on reflection quality. We investigate the impact of different scopes of failed reasoning trajectories on reflection quality by comparing four strategies: the full failure trajectory, steps after the decisive error step, steps before the decisive error step, and only the decisive error step.
Figure 7: Ablation study. Performance comparison of DoCtOR with variants that remove the correction submodule (w/o correct), both diagnosis and correction submodules (w/o diagnose & correct), or proximal policy optimization for the reflector (w/o PPO).
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The final answer prompt for Mind2Web.
Figure 9: The final answer prompt for HotPotQA & ChartQAPro .
Figure 10: Performance on HotPotQA under different Test Set Sizes.
Figure 11: Robustness Across Model Families and Sizes.
Figure 12: Effect of Correctness Threshold γ\gamma .
Figure 13: The action module prompt for HotPotQA when deploying DoCtOR .
Figure 14: The action module prompt for Mind2Web when deploying DoCtOR .
Figure 15: The action module prompt for ChartQAPro when deploying DoCtOR .
Figure 16: The counterfactual reasoning prompt when deploying DoCtOR .
Figure 17: The reflector prompt when deploying DoCtOR .
Figure 18: The action module prompt for HotPotQA when deploying Reflexion, Retroformer, and COPPER.
Figure 19: The action module prompt for ChartQAPro when deploying Reflexion, Retroformer, and COPPER.
Figure 20: The action module prompt for Mind2Web when deploying Reflexion, Retroformer, and COPPER.
Figure 21: The reflector prompt when deploying Reflexion, Retroformer, and COPPER.
More in this topic

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,050 papers

Explore similar work

CardsList
Oct 8, 2026

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose Error-Propagation Modeling for Failure Attribution (EMFA). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5

College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Failure Attribution for LLM AgentsAgent Failure AnalysisMulti-Agent LLM SystemsMulti-Agent Systems
AskPDF
Sep 4, 2026

DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems

Large language model (LLM)-based multi-agent systems have experienced rapid growth in recent years. Despite their promise, such systems remain fragile, frequently exhibiting reasoning and coordination errors that can lead to system-level failures. Failure attribution in such systems relies on tracing natural language interactions among agents to identify the decisive error, which refers to the earliest action whose correction can reverse system failure. There are two key challenges: 1) Shallow attribution: Existing methods often capture only minor deviations, such as incomplete retrievals or formatting errors, which verification mechanisms can correct, while missing the decisive cause of system failure. 2) Contextual degradation: As the length of the system traces increases, the model's reasoning ability rapidly deteriorates. To address these challenges, we propose DCFA, a training-free framework for failure attribution. DCFA integrates a global module that constructs structured causal-inspired dependency graphs from system traces to identify the initial decisive error, and a local module that applies local counterfactual-inspired reasoning to refine causal-inspired attribution. Experiments on the Who&When benchmark across six LLMs show that DCFA improves step-level accuracy by up to 8.27% over state-of-the-art baselines.

Zehao Wang, Lanjun Wang, Shilong Jin +2

College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China +1

Agent Failure AnalysisMulti-Agent LLM SystemsMulti-Agent SystemsFailure Attribution for LLM Agents
AskPDF
Apr 19, 2026

Towards Self-Improving Error Diagnosis in Multi-Agent Systems

Large Language Model (LLM)-based Multi-Agent Systems (MAS) enable complex problem-solving but introduce significant debugging challenges, characterized by long interaction traces, inter-agent dependencies, and delayed error manifestation. Existing diagnostic approaches often rely on expensive expert annotation or ''LLM-as-a-judge'' paradigms, which struggle to pinpoint decisive error steps within extended contexts. In this paper, we introduce ErrorProbe, a self-improving framework for semantic failure attribution that identifies responsible agents and the originating error step. The framework operates via a three-stage pipeline: (1) operationalizing the MAS failure taxonomy to detect local anomalies, (2) performing symptom-driven backward tracing to prune irrelevant context, and (3) employing a specialized multi-agent team (Strategist, Investigator, Arbiter) to validate error hypotheses through tool-grounded execution. Crucially, ErrorProbe maintains a verified episodic memory that updates only when error patterns are confirmed by executable evidence, without the need for annotation. Experiments across the TracerTraj and Who&When benchmarks demonstrate that ErrorProbe significantly outperforms baselines, particularly in step-level localization, while the verified memory enables robust cross-domain transfer without retraining.

Jiazheng Li, Emine Yilmaz, Bei Chen +1

King’s College London · Amazon Alexa AI · University College London

Multi-Agent LLM SystemsMulti-Agent SystemsSelf-Improving AgentsLLM Agent Reliability
AskPDF