cs.AIOct 13, 2026

BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems

Authors: Xiaoguang Guo, Yaohao Long, Chuxu Zhang

Organizations: University of Connecticut

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model multi-agent systems extend individual agents' capabilities through collaboration across specialized roles, tools, and workflows. Yet collaboration makes failures harder to diagnose: one agent may misinterpret a correct result while others carry the mistake forward. Failure attribution therefore seeks to identify the responsible agent and decisive error step. Recent methods trace dependencies, but knowing where information came from is not the same as knowing where it went wrong. We introduce BLADE (Blame Localization via Audited Dependency Evidence), a training-free framework that moves beyond tracing information to auditing its use. Using Jev for structured probabilistic judgments, BLADE uses dependency probabilities to select relevant source-decision handoffs for auditing. At these handoffs, it examines whether receiving decisions use their sources faithfully and within task requirements, distinguishing error introduction, propagation, and repair. These findings inform a global diagnosis that considers all execution steps. Without reference answers, BLADE achieves the best or tied-best step accuracy among eight strategies under a shared Jev decision interface on 1,228 failed runs across five Who&When and Who&When Pro subsets, outperforming All-at-once by 2.2-11.1 percentage points. Making responsibility evidence explicit offers a path toward more interpretable and reliable agentic systems.

Figures & tables

Figure 1: Failure attribution requires distinguishing where an error is first introduced from where it is merely propagated downstream.
Figure 2: Overview of BLADE. (1) Jev reconstructs dependencies between events. (2) The dependencies select the handoffs to audit, and Jev checks how each agent decision uses its source against the task’s requirement spans RR . (3) The diagnosis reads the full trajectory with the blame graph G=(V,E,R,F)G=(V,E,R,F) ; every event remains a candidate.
 Who&WhenWho&When ProMacro Avg.
 HCAGMagentic-OneDyLANMetaGPT 
MethodAgentStepAgentStepAgentStepAgentStepAgentStepAgentStep
All-at-once ( Zhang et al., 2025 )50.024.164.350.080.478.759.952.646.954.360.352.0
Step-by-step ( Zhang et al., 2025 )55.227.657.939.780.576.050.746.153.153.159.548.5
Binary search ( Zhang et al., 2025 )56.931.059.545.274.668.758.656.637.037.057.347.7
TRIAGE ( Wang, 2026 )56.922.460.344.483.579.952.648.046.946.960.048.3
Table 1: Agent and step accuracy (%) without the reference answer. All methods are instantiated under the same Jev decision interface (Appendix B.2 ). HC: Hand-Crafted; AG: Algorithm-Generated. Macro Avg. equally weights the five subsets. Bold marks the best and underline the second best.
Figure 3: Step accuracy under increasing localization tolerance kk , where predictions within kk events of the annotated step are considered correct. k=0k=0 corresponds to the exact step accuracy reported in Table 1 .
 Who&WhenWho&When Pro
EvidenceHCAGM-OneDyLAN
No evidence24.150.078.752.6
Dependencies only27.651.678.453.9
Shuffled evidence25.952.476.752.0
Without breach findings29.352.478.353.3
Full BLADE31.054.080.956.6
Table 2: Step attribution accuracy (%) under different evidence configurations. The diagnosis question and candidate events are held fixed. No evidence corresponds to All-at-once; dependencies only includes dependency edges and task requirement spans; shuffled evidence randomly reassigns edges and audit findings to event pairs. M-One: Magentic-One.
Figure 4: Dependency-guided versus exhaustive auditing. Top: step accuracy gain over exhaustive auditing; middle: handoff audits per run (log scale); bottom: share of findings on the annotated decisive step. AG: Algorithm-Generated; M-One: Magentic-One.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Share of audited handoffs labeled breach by the position of the receiving decision, in agent decisions from the annotated decisive step (0). M-One: the Magentic-One runs of Who&When Pro.
BaselineGain (pp)95% CI
All-at-once+5.6[2.8, 8.6]
Step-by-step+9.1[5.2, 13.2]
Binary search+9.9[5.8, 14.0]
TRIAGE+9.2[6.4, 12.2]
FALAT+17.9[13.5, 22.4]
ErrorProbe+16.2[12.6, 20.1]
Appendix
Table 3: Macro-average step accuracy gain of BLADE over each baseline, weighting the subsets equally, with paired task-cluster bootstrap 95% confidence intervals.
Figure 6: Question templates for dependency reconstruction, contract auditing, and diagnosis.
More in this topic

Failure Attribution for LLM Agents

See all 34 papers

Explore similar work

CardsList
Sep 4, 2026

DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems

Large language model (LLM)-based multi-agent systems have experienced rapid growth in recent years. Despite their promise, such systems remain fragile, frequently exhibiting reasoning and coordination errors that can lead to system-level failures. Failure attribution in such systems relies on tracing natural language interactions among agents to identify the decisive error, which refers to the earliest action whose correction can reverse system failure. There are two key challenges: 1) Shallow attribution: Existing methods often capture only minor deviations, such as incomplete retrievals or formatting errors, which verification mechanisms can correct, while missing the decisive cause of system failure. 2) Contextual degradation: As the length of the system traces increases, the model's reasoning ability rapidly deteriorates. To address these challenges, we propose DCFA, a training-free framework for failure attribution. DCFA integrates a global module that constructs structured causal-inspired dependency graphs from system traces to identify the initial decisive error, and a local module that applies local counterfactual-inspired reasoning to refine causal-inspired attribution. Experiments on the Who&When benchmark across six LLMs show that DCFA improves step-level accuracy by up to 8.27% over state-of-the-art baselines.

Zehao Wang, Lanjun Wang, Shilong Jin +2

College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China +1

Agent Failure AnalysisMulti-Agent LLM SystemsMulti-Agent SystemsFailure Attribution for LLM Agents
AskPDF
Oct 8, 2026

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose Error-Propagation Modeling for Failure Attribution (EMFA). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5

College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Failure Attribution for LLM AgentsAgent Failure AnalysisMulti-Agent LLM SystemsMulti-Agent Systems
AskPDF
Jul 8, 2026

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Yufei Xia, Anjun Gao, Yueyang Quan +2

University of Louisville · University of North Texas

Agent Failure AnalysisMulti-Agent LLM Systems
AskPDF