Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation

Yeonjun In; Mehrab Tanjim; Jayakumar Subramanian; Sungchul Kim; Uttaran Bhattacharya; Wonjoong Kim; Sangwu Park; Somdeb Sarkhel

cs.AIMar 26, 2026

Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation

Authors: Yeonjun In, Mehrab Tanjim, Jayakumar Subramanian, Sungchul Kim, Uttaran Bhattacharya, Wonjoong Kim, Sangwu Park, Somdeb Sarkhel,

Organizations: KAIST Daejeon, Republic of Korea · Adobe Research San Jose, CA, USA

Abstract

Failure attribution is essential for diagnosing and improving multi-agent systems (MAS), yet existing benchmarks and methods largely assume a single deterministic root cause for each failure. In practice, MAS failures often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories. We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution, a practical paradigm that explicitly accounts for attribution ambiguity. To support this setting, we introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm. Through extensive experiments, we find that prior conclusions suggesting LLMs struggle with failure attribution are largely driven by limitations in existing benchmark designs. Our results highlight the necessity of multi-perspective benchmarks and evaluation protocols for realistic and reliable MAS debugging.

Figures & tables

Figure 1. Motivating examples illustrating the multi-perspective nature of MAS failure attribution. (a) presents a simplified execution log of a MAS. (b) presents the example of deterministic failure attribution existing approaches assume (c) presents the example of multi-perspective failure attribution.

	Attribution Granularity	Annotation Method	Attribution Reasoning Eval	Perspective	Size
MAST ( Cemri et al., 2025 )	Execution	Hybrid	✗	Single	1,600
Who&When ( Zhang et al., )	Step	Expert	✗	Single	184
TracerTraj ( Zhang et al., 2025a )	Step	Automated	✗	Single	2,500
Aegis ( Kong et al., 2025 )	Agent	Automated	✗	Single	9,500
MP-Bench	Step	Expert	✓	Multi	289

Table 1. Comparison between our benchmark and others.

Figure 2. Overall framework of the (a) annotation process and (b) evaluation protocol of MP-Bench .

Figure 3. Analysis of MP-Bench highlighting the multi-perspective nature of MAS failure attribution. (a) Distribution of steps grouped by annotator consensus on failure annotations. (b) Inter-annotator disagreement rates for failure annotations.

	Hand-Crafted MAS					Automatic MAS
	Failure Attribution Eval				Attr. Reason. Eval	Failure Attribution Eval				Attr. Reason. Eval
	nDCG@5 (↑)		nDCG@full (↑)		LLM-Judge	nDCG@5 (↑)		nDCG@full (↑)		LLM-Judge
Model	Exp	Linear	Exp	Linear	Avg. Score (↑)	Exp	Linear	Exp	Linear	Avg. Score (↑)
Random	0.1275	0.2368	0.1476	0.2228	-	0.3147	0.4153	0.3207	0.415	-
GPT-4.1	0.4313	0.5386	0.4717	0.4961	7.63	0.6755	0.695	0.6761	0.6851	7.78
GPT-5.1	0.3747	0.5164	0.4526	0.553	7.62	0.7844	0.8134	0.7856	0.8051	8.25

Table 2. Failure attribution and attribution reasoning performance comparison.

Figure 4. Disagreement analysis of the LLM-based failure attribution system. (a) Pairwise disagreement rates across different samplings of GPT-5.1. (b) Pairwise disagreement rates across different LLMs.

	Total (Hand-Crafted + Automatic)
	$\tau=1$	$\tau=0.7$	$\tau=0.5$	$\tau=0.3$	$\tau=0$
GPT-4.1	0 .5349	0.5227	0.5314	0.5219	0.5162
GPT-5.1	0.552	0.562	0.5561	0.5628	0.552
Claude-Sonnet-4.5	0.5916	0.5863	0.588	0.5894	0.5874
Qwen3-8B	0.4553	0.4619	0.4692	0.4803	0.3886
GPT-oss-120B	0.5449	0.5439	0.5458	0.5554	0.5432

Table 3. Performance comparison (nDCG@5 with exponential gain) across different sampling temperatures

\tau

. Bold values indicate the best performance within each model.

	Hand-Crafted			Automatic
	$N=3$	$N=5$	$N=10$	$N=3$	$N=5$	$N=10$
GPT-4.1	0.4359	0.4384	0.4622	0.6874	0.6945	0.7201
GPT-5.1	0.3806	0.3993	0.4198	0.7819	0.7801	0.7769
O3-mini	0.4323	0.4725	0.4862	0.4125	0.4254	0.4604
Claude-Sonnet-4.5	0.4472	0.4672	0.4753	0.766	0.7761	0.7861
Qwen3-8B	0.2951	0.3241	0.3442	0.662	0.69	0.7227

Table 4. Performance comparison (nDCG@5 with exponential gain) across different number of LLM runs

N

. Bold values indicate the best performance within each model.

Model Combination				nDCG@5 (Exp)
G-4.1	G-5.1	OSS	Sonnet	Hand-Crafted	Automatic
✓	✗	✗	✗	0.4313	0.6755
✗	✓	✗	✗	0.3747	0.7844
✗	✗	✓	✗	0.4245	0.703
✗	✗	✗	✓	0.4397	0.7894
✓	✓	✓	✗	0.479	0.8

Table 5. Performance comparison between single-LLM and multi-LLM failure attribution systems. For brevity, we denote GPT-4.1, GPT-5.1, GPT-OSS-120B, and Claude-Sonnet-4.5 as G-4.1, G-5.1, OSS, and Sonnet, respectively.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Consolidation: GPT-4.1			Consolidation: GPT-5.1			Consolidation: Sonnet
	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet
GPT-4.1	8.49 (3)	7.52 (3)	7.07 (2)	8.60 (3)	7.63 (2)	7.20 (2)	8.48 (3)	7.44 (3)	7.14 (2)
GPT-5.1	8.57 (2)	7.53 (2)	7.04 (3)	8.68 (2)	7.62 (3)	7.14 (4)	8.56 (2)	7.49 (2)	7.03 (3)
O3-mini	8.22 (5)	7.31 (5)	6.93 (5)	8.32 (5)	7.48 (5)	7.06 (5)	8.13 (5)	7.23 (5)	6.92 (5)
Claude-Sonnet-4.5	8.70 (1)	7.82 (1)	7.27 (1)	8.83 (1)	7.95 (1)	7.50 (1)	8.64 (1)	7.71 (1)	7.32 (1)
Qwen3-8B	7.45 (6)	5.97 (6)	5.82 (6)	7.54 (6)	6.11 (6)	5.96 (6)	7.37 (6)	5.92 (6)	5.77 (6)

Appendix

Table 6. Performance comparison across consolidation and judge model combinations for hand-crafted MAS executions. Sonnet denotes Claude-Sonnet-4.5. Values in parentheses indicate model rankings.

	Consolidation: GPT-4.1			Consolidation: GPT-5.1			Consolidation: Sonnet
	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet	Judge: GPT-4.1	Judge: GPT-5.1	Judge: Sonnet
GPT-4.1	8.5 (4)	7.66 (4)	7.15 (4)	8.69 (3)	7.78 (4)	7.38 (4)	8.43 (4)	7.49 (4)	7.13 (4)
GPT-5.1	8.84 (1)	8.06 (1)	7.45 (2)	9.01 (2)	8.25 (1)	7.68 (2)	8.81 (1)	8.03 (1)	7.48 (1)
O3-mini	8.13 (5)	7.21 (5)	6.86 (5)	8.28 (5)	7.37 (5)	7.06 (5)	8.03 (5)	7.16 (5)	6.75 (5)
Claude-Sonnet-4.5	8.8 (2)	8.01 (2)	7.5 (1)	9.02 (1)	8.23 (2)	7.73 (1)	8.77 (2)	7.94 (2)	7.47 (2)
Qwen3-8B	7.54 (6)	6.38 (6)	6.03 (6)	7.74 (6)	6.54 (6)	6.24 (6)	7.43 (6)	6.17 (6)	5.93 (6)

Appendix

Table 7. Performance comparison across consolidation and judge model combinations for automatic MAS executions. Sonnet denotes Claude-Sonnet-4.5. Values in parentheses indicate model rankings.

Figure 5. Failure attribution performance across varying numbers of LLM runs (

N

) on MP-Bench . OSS denotes GPT-OSS-120B, and Sonnet denotes the Claude-Sonnet-4.5 model.

Figure 6. Annotation description provided to annotators.

Panel of Figure 7 — Figure 7. Prompt for failure attribution method (All At Once ( Zhang et al., ) ).

Panel of Figure 8 — Figure 8. LLM-as-a-Judge prompt for attribution reasoning evaluation.

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 153 papers

Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Apr 24, 2026Mengzhuo Chen, Junjie Wang, Fangwen Mu +4Agent Failure Analysis Multi-Agent LLM Systems
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
VerifyMAS: Hypothesis Verification for Failure Attribution in LLM-based Multi-Agent Systems
May 17, 2026Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2Agent Failure Analysis Multi-Agent LLM Systems
Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Failure Attribution for LLM Agents Agent Failure Analysis
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Apr 24, 2026J/K move · Enter open · S save

Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems

Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang+3

State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Agent Failure Analysis Multi-Agent LLM Systems LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Failure attribution, i.e., identifying the responsible agent and decisive step of a failure, is particularly challenging in LLM-based multi-agent systems (MAS) due to their natural-language reasoning, nondeterministic outputs, and intricate interaction dynamics. A reliable benchmark is therefore essential to guide and evaluate attribution techniques. Yet existing benchmarks rely on partially observable traces that capture only agent outputs, omitting the inputs and context that developers actually use when debugging. We argue that failure attribution should be studied under full execution observability, aligning with real-world developer-facing scenarios where complete traces, rather than only outputs, are accessible for diagnosis. To this end, we introduce TraceElephant, a benchmark designed for failure attribution with full execution traces and reproducible environments. We then systematically evaluate failure attribution techniques across various configurations. Specifically, full traces improve attribution accuracy by up to 76% over a partial-observation counterpart, confirming that missing inputs obscure many failure causes. TraceElephant provides a foundation for follow-up failure attribution research, promoting evaluation practices that reflect real-world debugging and supporting the development of more transparent MASs.

Abstract

Agent Failure Analysis

Explore similar work

Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM-based Multi-Agent Systems

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems