From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems

Yawen Wang; Wenjie Wu; Junjie Wang; Qing Wang

cs.AIFeb 27, 2026

From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems

Authors: Yawen Wang, Wenjie Wu, Junjie Wang, Qing Wang

Organizations: Institute of Software, Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Wuhan University of Technology, Wuhan, China

Abstract

LLM-powered Multi-Agent Systems (MAS) have demonstrated remarkable capabilities in complex domains but suffer from inherent fragility and opaque failure mechanisms. Existing failure attribution methods, whether relying on direct prompting, costly replays, or supervised fine-tuning, typically treat execution logs as flat sequences. This linear perspective fails to disentangle the intricate causal links inherent to MAS, leading to weak observability and ambiguous responsibility boundaries. To address these challenges, we propose CHIEF, a novel framework that transforms chaotic trajectories into a structured hierarchical causal graph. It then employs hierarchical oracle-guided backtracking to efficiently prune the search space via sybthesized virtual oracles. Finally, it implements counterfactual attribution via a progressive causal screening strategy to rigorously distinguish true root causes from propagated symptoms. Experiments on Who&When benchmark show that CHIEF outperforms eight strong and state-of-the-art baselines on both agent- and step-level accuracy. Ablation studies further confirm the critical role of each proposed module.

Figures & tables

Figure 1: A failure log induced by an unsatisfied constraint. Although the Orchestrator marking constraints (red, Step 14), the Websurfer overlooks these constraints during execution (Step 16).

Type	Method	Hand-Crafted Dataset		Algorithm-Generated Dataset
Type	Method	Agent-level $\uparrow$	Step-level $\uparrow$	Agent-level $\uparrow$	Step-level $\uparrow$
Heuristic	Random	12.00 / 12.00	4.20 / 4.20	29.10 / 29.10	19.10 / 19.10
LLM-based Prompting	All-at-Once	50.00 / 48.28	5.17 / 5.17	61.11 / 59.52	13.49 / 15.87
	Step-by-Step	36.00 / 34.30	6.60 / 6.90	39.70 / 28.30	27.40 / 17.80
	Binary Search	51.70 / 36.20	6.90 / 6.90	44.10 / 30.10	24.00 / 16.60
	ECHO	68.40 / 67.90	28.10 / 26.80	68.80 / 67.20	28.80 / 27.20

Table 1: Main results on the Who&When benchmark. We report agent- and step-level accuracy (%) across both subsets. Each cell reports two values: the left corresponds to the setting w/

\mathcal{G}

, and the right corresponds to w/o

\mathcal{G}

, where

\mathcal{G}

indicates access to the task ground truth (i.e., correct outcome). Best results are in bold .

Method	Token Cost Per Case $\downarrow$
Method	Hand-Crafted	Alg.-Generated
All-at-Once	21,581	5,833
Step-by-Step	87,720	6,533
Binary Search	34,659	5,226
FAMAS	431,620	116,660
ECHO	53,701	25,642

Table 2: Average token cost per case on Who&When across two subsets in w/

\mathcal{G}

setting. Lower is better.

Type	Base LLM	Hand-Crafted Dataset		Algorithm-Generated Dataset
Type	Base LLM	Agent-level $\uparrow$	Step-level $\uparrow$	Agent-level $\uparrow$	Step-level $\uparrow$
Open Source	Qwen3-235B-A22B-Instruct-2507	63.79	22.41	69.04	32.53
	Kimi-k2-0905-preview	67.24	24.13	68.25	42.06
	Deepseek-V3.2(thinking)	77.58	29.31	76.80	52.00
Closed Source	GPT-5.2(medium)	68.96	24.13	66.67	43.65
	Claude-4.5-Sonnet(standard thinking)	68.96	27.58	63.49	51.58

Table 3: Performance of CHIEF on the Who&When benchmark ( w/

\mathcal{G}

) using various base LLMs. We report agent- and step-level accuracy (%) across both subsets, noting the model’s thinking level in parentheses.

CHIEF Variants	Hand-Crafted		Alg.-Generated
CHIEF Variants	Agent $\uparrow$	Step $\uparrow$	Agent $\uparrow$	Step $\uparrow$
All-at-Once	50.00	5.17	61.11	13.49
Only M1	37.93	18.96	66.66	24.60
M1+M2	51.72	22.41	61.11	34.12
M1+M3	50.00	22.41	65.07	26.98
CHIEF	77.59	29.31	76.80	52.00

Table 4: Ablation study of CHIEF on Who&When benchmark in w/

\mathcal{G}

setting. We evaluate the incremental contribution of modules (M1–M3), reporting agent- and step-level accuracy (%) across both subsets.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: An example of hierarchical causal graph.

Panel of Figure 12 — Figure 12: Examples of Planning-Control Attribution .

Explore similar work

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Agent Failure Analysis Multi-Agent LLM Systems
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China
DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
Sep 4, 2026Zehao Wang, Lanjun Wang, Shilong Jin +2Agent Failure Analysis Multi-Agent LLM Systems
College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China +1
VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
May 17, 2026Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2Agent Failure Analysis Multi-Agent LLM Systems
Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago

Oct 8, 2026J/K move · Enter open · S save

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu, Xu Zhang+4

College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Agent Failure Analysis Multi-Agent LLM Systems Multi-Agent Systems

Open paper AskPDF arXiv

LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose Error-Propagation Modeling for Failure Attribution (EMFA). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

Abstract

Explore similar work

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems

VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems