Multi-agent LLM systems are increasingly deployed in real-world applications, where failures can be costly and difficult to localize. Despite growing efforts to automate failure attribution, diagnosing failed runs still largely relies on human engineers. Yet engineers rarely debug complex systems by reading raw logs end to end. Instead, observability tools organize traces around components, actions, and dependencies to support targeted navigation. We hypothesize that modern LLMs can benefit from the same paradigm. To test this hypothesis, we introduce Adaptive Influence Graphs (AIGs), a two-stage agentic framework that first transforms a failed trace into a structured graph and then navigates it to identify the critical error. Across multiple models, we show that richer trace representations consistently improve failure attribution, with adaptive graph construction and agent-directed traversal yielding the strongest results. AIGs establish a new state of the art on Who&When, the standard benchmark for multi-agent failure attribution. This affirms our hypothesis that attribution depends not only on the diagnosing model, but also on how the trace is represented and explored.
Figures & tables
Figure 1: The same trace yields different attributions depending on its representation. The raw log highlights the visible symptom (step 3); the Adaptive Influence Graph traces the error back to its origin (step 2).
Figure 2: Step accuracy improves with richer trace representations, and improves further when the adaptive graph is traversed by an agentic reader (Algorithm-Generated Who&When, Opus-5).
Figure 3: Build-read pipeline for Adaptive Influence Graphs (AIGs). The builder uses log-inspection and graph-construction tools to produce G ; a critic–refiner audits its structural and semantic validity; and a reader uses graph and log-inspection tools to produce the final attribution.
Method
Model
Algorithm-Generated Dataset
Hand-Crafted Dataset
Step Acc. ↑
Agent Acc. ↑
Step Acc. ↑
Agent Acc. ↑
All-at-Once
Opus-5
46.40
67.20
22.41
50.00
Step-by-Step
Opus-5
42.40
54.40
25.86
58.62
CDC-MAS ( Ma et al., 2025 )
GPT-4o
36.00
48.50
18.20
56.80
AgenTracer-8B ( Zhang et al., 2025a )
Qwen3-8B
37.30
63.73
20.68
63.82
CORRECT ( Yu et al., 2025 )
GPT-5
38.10
–
17.20
–
Table 1: Who&When benchmark. Rows use no ground-truth access at inference time. One Algorithm-Generated log is content-filtered by Opus-5; therefore, for this model we use n=125 . † Run using the authors’ released code. Our methods are shaded.
Representation
Step ↑
Agent ↑
Raw log
46.40
67.20
Structured log
48.80
63.20
Influence Graph
52.00
70.40
Adaptive Influence Graph LLM reader
50.40
64.80
Adaptive Influence Graph
55.20
71.20
Table 2: Opus-5 representation ablations on the Algorithm-Generated Who&When partition (no ground-truth access). Our AIGs are in bold.
Representation
Step Acc.
± 1
± 2
± 3
RAFFLES (Llama-3.3-70B)
43.65
58.73
73.81
82.54
Structured log
48.80
60.00
72.00
80.00
Influence Graph
52.00
61.60
73.60
80.00
Adaptive Influence Graph
55.20
64.00
78.40
84.00
Table 3: Step accuracy under relaxed tolerance windows ( ±n steps) on the Algorithm-Generated Who&When partition (Opus-5). RAFFLES results as published by the authors, who report tolerance windows only for Llama-3.3-70B.
Property
AIGs
IGs
Nodes per graph
9.5 ±
2.0
7.5 ±
2.2
Edges per graph
13.5 ±
3.6
9.0 ±
3.2
Edge density (edges / node)
1.4 ±
0.2
1.2 ±
0.2
Max hub out-degree
3.6 ±
1.1
1.8 ±
0.6
Max steps merged into one node
2.8 ±
1.2
1.6 ±
0.5
Mean cross-range distance
3.6 ±
1.0
3.1 ±
1.0
Table 4: Properties of the constructed graphs averaged across the Algorithm-Generated partition ( ± standard deviation across graphs).
Figure 4: The same trace encoded as an Influence Graph and as an Adaptive Influence Graph.
Agentic reading profile
First raw read is already t^
61.3%
Opened steps within ±1 of t^
48.4%
Multi-step investigation
86.5%
Single-step verification
11.9%
Commits from the graph alone
1.6%
Table 5: Reader behavior on the Algorithm-Generated partition (Sonnet-4, no ground-truth access). The last three rows partition executions by reading depth. t^ is the step the execution submits.
Figure 5: Each panel highlights one move in orange; chords are inheritance edges and numbers are the raw-log steps a node merges. Rather than follow transcript order, the reader walks inheritance edges from the node the graph implicates, then commits to step 6, matching the annotation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Tool
Description
Inspection ( Tlog )
get_overview
Compact metadata about the log: step count, per-step agent, and sizes.
read_full_log
Return every step concatenated.
read_step
Return one step’s content.
read_step_range
Return an inclusive range of steps.
list_agents
List every agent and its system prompt.
Appendix
Table 6: Builder tools. Inspection reads the log τ ; construction edits the graph.
Node type
Role & fields
task
The user’s question that seeds the trajectory—the root of the graph (no incoming edges). Fields: content (the question), operative_brief (the instructions the trajectory starts under), step_refs
orchestrator
A coordinating turn—an agent that plans, delegates, or judges progress and routes the work onward. Fields: agent_name , summary (what the agent did in this turn, start to end), authorship (as in agent ), step_refs
agent
A working turn—an invoked agent that executes a delegated instruction and produces a result. Fields: agent_name , summary , input (the instruction, context, or prior result it worked from), output (the concrete result, decision, tool output, or handoff it produced), authorship (whether it authored that output), defect_recorded (the node whose output fails the optional system-emitted instruction_or_question it was given, yet is committed downstream), step_refs
conclusion
The final answer the system produced—the terminal sink (no outgoing edges). Fields: content (the final answer), step_refs
Appendix
Table 7: Node types in the agentic graph. Every node carries step_refs , the raw-log steps it covers.
Method
Model
Algorithm-Generated
Hand-Crafted
Step Acc. ↑
Agent Acc. ↑
Step Acc. ↑
Agent Acc. ↑
All-at-Once
Opus-5
44.00
63.20
18.97
51.72
Step-by-Step
Opus-5
40.80
54.40
22.40
50.00
FAMAS ( Ge et al., 2025 )
Qwen2.5-72B
23.81
55.56
41.38
62.07
AgenTracer-8B ( Zhang et al., 2025a )
Qwen3-8B
42.86
69.62
20.68
69.10
CHIEF ( Wang et al., 2026b )
DS-V3.2
52.00
76.80
29.31
77.59
Appendix
Table 8: Who&When benchmark with ground-truth access at inference time. Baseline numbers are as reported by their authors. One Algorithm-Generated log is content-filtered by Opus-5; therefore, we use n=125 for this model. † Run using the authors’ released code. Our methods are shaded.
Failure attribution, i.e., identifying the responsible agent and decisive step of a failure, is particularly challenging in LLM-based multi-agent systems (MAS) due to their natural-language reasoning, nondeterministic outputs, and intricate interaction dynamics. A reliable benchmark is therefore essential to guide and evaluate attribution techniques. Yet existing benchmarks rely on partially observable traces that capture only agent outputs, omitting the inputs and context that developers actually use when debugging. We argue that failure attribution should be studied under full execution observability, aligning with real-world developer-facing scenarios where complete traces, rather than only outputs, are accessible for diagnosis. To this end, we introduce TraceElephant, a benchmark designed for failure attribution with full execution traces and reproducible environments. We then systematically evaluate failure attribution techniques across various configurations. Specifically, full traces improve attribution accuracy by up to 76% over a partial-observation counterpart, confirming that missing inputs obscure many failure causes. TraceElephant provides a foundation for follow-up failure attribution research, promoting evaluation practices that reflect real-world debugging and supporting the development of more transparent MASs.
Mengzhuo Chen, Junjie Wang, Fangwen Mu +4
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
LLM-powered Multi-Agent Systems (MAS) have demonstrated remarkable capabilities in complex domains but suffer from inherent fragility and opaque failure mechanisms. Existing failure attribution methods, whether relying on direct prompting, costly replays, or supervised fine-tuning, typically treat execution logs as flat sequences. This linear perspective fails to disentangle the intricate causal links inherent to MAS, leading to weak observability and ambiguous responsibility boundaries. To address these challenges, we propose CHIEF, a novel framework that transforms chaotic trajectories into a structured hierarchical causal graph. It then employs hierarchical oracle-guided backtracking to efficiently prune the search space via sybthesized virtual oracles. Finally, it implements counterfactual attribution via a progressive causal screening strategy to rigorously distinguish true root causes from propagated symptoms. Experiments on Who&When benchmark show that CHIEF outperforms eight strong and state-of-the-art baselines on both agent- and step-level accuracy. Ablation studies further confirm the critical role of each proposed module.
Yawen Wang, Wenjie Wu, Junjie Wang +1
Institute of Software, Chinese Academy of Sciences, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China +2
Large language model (LLM)-based multi-agent systems (MAS) often exhibit complex failure modes, which frequently cause agents to produce incorrect outcomes. This motivates the task of Agent Failure Attribution: given a failed multi-agent trajectory, identify the faulty agents and their corresponding error types. Existing approaches predominantly rely on LLMs to perform failure attribution, either through direct prompting, fine-tuning on synthetic data or complex agentic pipelines. While effective, these methods incur substantial computational overhead due to long-context processing, expensive post-training and handcrafted workflows. Moreover, empirical evidence shows that even state-of-the-art models achieve limited accuracy on existing benchmarks, suggesting that scaling model size alone is insufficient. In this work, we revisit this task and question the necessity of such expensive generative solutions. We introduce AFANet, a lightweight graph-based framework that models interaction trajectories through step-level semantic signals and agent-level relationships. We show that with significantly fewer parameters and near-zero inference cost, AFANet (i) matches or outperforms LLM-based baselines, including fine-tuned models on in-domain benchmarks, (ii) maintains robust performance across different GNN architectures and (iii) can be further improved with inexpensive test-time adaptation on the OOD benchmark. Our results suggest that effective agent failure attribution does not require heavy LLM reasoning and a lightweight, structured approach can achieve strong performance.