LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose CEG-Agent, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs Causal Error Graphs (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct CEG-Bench, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an Adversarial Agentic Adjudication Protocol (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.
Figures & tables
Figure 1: Comparison between prior paradigm and our method for agentic trace diagnosis.
Figure 2: The overall design of CEG-Agent.
Figure 3: Construction and evaluation overview of CEG-Bench.
Figure 4: AAAP–human-gold agreement per SRE dimension over all 100 traces.
Backbone
System
SRE
Structural
Textual
Utility
error
edge
chain
fail.
attr.
anom.
event
cons.
Claude-Opus-4.7
Claude Code (raw)
45.8
39.5
37.1
38.8
44.7
48.7
44.7
78.1
100.0
Claude-Opus-4.7
Claude Code (+Process)
49.6
49.2
41.8
38.2
47.7
43.9
46.0
100.0
100.0
Claude-Opus-4.7
CEG-Claude (ours)
51.6
46.7
42.4
51.0
38.8
37.1
60.0
100.0
99.9
GPT-5.5
Codex (raw)
32.0
22.8
31.8
27.4
29.9
25.5
2.0
98.0
100.0
GPT-5.5
Codex (+Process)
32.9
27.0
34.5
29.5
21.5
23.5
1.7
100.0
100.0
Table 1: Semantically relaxed evaluation ( SRE ) on CEG-Bench.
Backbone
System
SEE
Dimensions
failure
error
attach
cause
contrib.
Claude-Opus-4.7
Claude Code (raw)
13.9
56.1
5.3
3.9
1.0
3.3
Claude-Opus-4.7
Claude Code (+Process)
19.3
47.3
18.5
16.3
1.1
13.2
Claude-Opus-4.7
CEG-Claude (ours)
20.5
52.1
20.5
17.8
2.0
9.9
GPT-5.5
Codex (raw)
11.7
43.8
5.6
6.0
0.0
3.1
GPT-5.5
Codex (+Process)
10.4
33.3
8.7
6.0
0.0
4.1
Table 2: Structurally exact evaluation ( SEE ) on CEG-Bench.
Backbone
SRE
SEE
Claude-Opus-4.7
51.6
20.5
GLM-5.1
46.7
19.6
GPT-5.5
46.5
16.9
Gemini-3.1-Pro
44.2
15.0
Qwen3.7-Max
43.7
18.7
Table 3: CEG-Agent composite SRE/SEE under five LLM backbones.
Feedback
Tau2
Browse
SWE
generic
77.8
6.9
14.3
CEG: failure
91.7
10.3
17.1
CEG: errors
80.6
17.2
17.1
CEG: priority errors
83.3
17.2
8.6
Table 4: Closed-loop repair pass@1 (%).
Category
Tool
Method
Mode
Effect on St
Inspection
read_trace
Trace inspection
read-only
Read the normalized trace T .
Inspection
inspect_state
State inspection
read-only
Read the current graph Gt and todo buffer Qt .
Phase
build_events
Event construction
generator
Write event nodes VE .
Phase
find_failures
Failure analysis
generator
Write failure nodes VF .
Phase
propose_errors
Error hypothesis generation
generator
Write causally relevant error nodes VR .
Phase
build_causal_graph
Causal graph induction
generator
Write causal edges EC .
Table 5: Registered tool set U of CEG-Agent. Tools are grouped by category, linked to the corresponding method component, and summarized by their execution mode and effect on the diagnosis state St .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Tau2
SWE
Browse
All
n=36
n=35
n=29
n=100
Events
13.3
47.8
46.1
34.9
Errors
2.0
3.9
6.2
3.9
Failures
0.9
1.4
1.5
1.2
Anomalies
2.3
2.5
2.8
2.5
Edges
18.0
57.7
64.6
45.4
Appendix
Table 6: Mean per-trace component counts in CEG-Bench by source benchmark. Tau2, SWE, and Browse denote Tau2Bench, SWE-bench, and BrowseComp.
Label
Count
%
Mechanism μ ( errors , n=391 )
planning
77
19.7
evidence integration
72
18.4
execution
66
16.9
self-evaluation
61
15.6
control
53
13.6
Appendix
Table 7: Marginal distributions over the controlled vocabularies in CEG-Bench. Percentages for μ , ρ , and ρ♯ are computed over the 391 annotated errors; percentages for θ are computed over the 125 annotated failures.
Mechanism
high
med.
low
Total
planning
0
40
37
77
evidence integration
3
19
50
72
execution
3
18
45
66
self-evaluation
0
14
47
61
control
1
15
37
53
omission
2
11
20
33
Appendix
Table 8: Repair-value buckets cross-tabulated by error mechanism. Counts are computed over the 391 annotated errors in CEG-Bench.
Criterion
Dimension
Agreement
SRE
CEG-Sim (overall)
71.5
anomaly
88.1
error
76.3
failure
76.3
main_chain
61.9
attribution
58.8
Appendix
Table 9: Agreement between the AAAP annotations in CEG-Bench and the expert-curated CEG-Bench-Gold, measured with SRE and SEE over all 100 traces (%). SEE agreement by complexity tercile is 43.0/45.8/52.4 for small/medium/large traces.
Backbone
System
SRE
Structural
Textual
Utility
error
edge
chain
fail.
attr.
anom.
event
cons.
Claude-Opus-4.7
Claude Code (raw)
38.1
36.3
26.5
24.8
49.1
24.4
43.4
61.4
100.0
Claude-Opus-4.7
Claude Code (+Process)
38.1
35.6
26.0
23.6
40.5
23.8
46.5
99.0
100.0
Claude-Opus-4.7
CEG-Claude (ours)
45.4
43.2
30.2
37.1
45.0
30.6
57.6
99.0
99.9
GPT-5.5
Codex (raw)
27.7
21.4
22.2
15.1
38.3
16.5
6.9
99.0
100.0
GPT-5.5
Codex (+Process)
26.6
21.7
23.5
15.1
26.6
12.4
6.9
99.0
100.0
Appendix
Table 10: Semantically relaxed evaluation ( SRE ) against the expert-curated CEG-Bench-Gold , using the same layout as Table 1 . Best score within each backbone block is in bold . Scores are lower than those against the AAAP reference, but the within-family ranking is preserved.
Backbone
System
SEE
Dimensions
failure
error
attach
cause
contrib.
Claude-Opus-4.7
Claude Code (raw)
15.5
60.1
8.5
6.1
0.5
2.2
Claude-Opus-4.7
Claude Code (+Process)
19.5
49.0
18.4
17.4
1.3
11.3
Claude-Opus-4.7
CEG-Claude (ours)
22.6
57.9
22.4
18.3
2.0
12.4
GPT-5.5
Codex (raw)
16.3
55.3
8.9
8.7
0.0
8.6
GPT-5.5
Codex (+Process)
13.3
39.5
9.2
8.4
1.7
7.5
Appendix
Table 11: Structurally exact evaluation ( SEE ) against the expert-curated CEG-Bench-Gold , using the same layout as Table 2 . Best score within each backbone block is in bold .
Backbone
Evaluation
System
Tau2Bench
SWE-bench
BrowseComp
n=36
n=35
n=29
Claude-Opus-4.7
SRE
Claude Code (raw)
59.8
37.3
38.7
Claude-Opus-4.7
SRE
Claude Code (+Process)
67.2
38.3
41.5
Claude-Opus-4.7
SRE
CEG-Claude (ours)
60.0
47.7
45.9
Claude-Opus-4.7
SEE
Claude Code (raw)
16.8
11.1
13.8
Claude-Opus-4.7
SEE
Claude Code (+Process)
24.9
13.5
19.3
Appendix
Table 12: Per-source composite scores on CEG-Bench. Best score within each backbone–evaluator block is in bold .
Backbone
System
Small
Medium
Large
∣VE∣ˉ≈9
∣VE∣ˉ≈23
∣VE∣ˉ≈72
Claude-Opus-4.7
Claude Code (raw)
16.6
10.9
14.2
Claude-Opus-4.7
Claude Code (+Process)
23.0
14.9
20.0
Claude-Opus-4.7
CEG-Claude (ours)
22.9
17.0
21.4
GPT-5.5
Codex (raw)
10.7
9.3
15.0
GPT-5.5
Codex (+Process)
9.1
3.9
18.0
Appendix
Table 13: SEE composite by trace-complexity tercile. Best score within each backbone block is in bold .
Backbone
System
SRE
SEE
Gold-empty
empty
non-empty
empty
non-empty
abstain
error
Claude-Opus-4.7
Claude Code (raw)
64.8
42.5
0.0
16.4
1/15
14/15
Claude-Opus-4.7
Claude Code (+Process)
87.1
43.0
0.0
22.7
11/15
4/15
Claude-Opus-4.7
CEG-Claude (ours)
68.6
48.6
0.0
24.1
0/15
14/15
GPT-5.5
Codex (raw)
59.9
27.1
0.0
13.7
0/15
15/15
GPT-5.5
Codex (+Process)
60.0
28.1
0.0
12.3
0/15
15/15
Appendix
Table 14: Performance split by gold-empty and non-empty CEGs. abstain counts empty predictions; error counts predictions with at least one error on gold-empty traces.
Slice
System
SRE
error
edge
chain
fail.
attr.
anom.
event
cons.
Gold-empty ( n=15 )
empty
Claude Code (raw)
64.8
6.7
100.0
100.0
6.7
100.0
37.2
80.7
100.0
empty
Claude Code (+Process)
87.1
73.3
100.0
100.0
73.3
100.0
51.3
100.0
100.0
empty
CEG-Claude
68.6
6.7
100.0
100.0
0.0
100.0
72.8
100.0
99.9
empty
Codex (raw)
59.9
0.0
100.0
100.0
0.0
100.0
0.0
98.8
100.0
empty
Codex (+Process)
60.0
0.0
100.0
100.0
0.0
100.0
0.0
100.0
100.0
Appendix
Table 15: SRE per-dimension scores split by gold-empty ( n=15 ) and non-empty ( n=85 ) subsets, with columns aligned to main-paper Table 1 . On the empty subset the structural dimensions edge , chain , and attr. are mechanically 100 because gold and prediction are both empty, and the utility dimensions event / cons. are at or near ceiling for all systems; only error , fail. , and anom. are informative.
Slice
System
SEE
failure
error
attach
cause
contrib.
Gold-empty ( n=15 )
empty
Claude Code (raw)
0.0
0.0
0.0
0.0
0.0
0.0
empty
Claude Code (+Process)
0.0
0.0
0.0
0.0
0.0
0.0
empty
CEG-Claude
0.0
0.0
0.0
0.0
0.0
0.0
empty
Codex (raw)
0.0
0.0
0.0
0.0
0.0
0.0
empty
Codex (+Process)
0.0
0.0
0.0
0.0
0.0
0.0
Appendix
Table 16: SEE per-dimension scores split by gold-empty ( n=15 ) and non-empty ( n=85 ) subsets. On the empty subset SEE is 0 for every system because the gold graph contains no errors, failures, or edges to match.
Backbone
System
∣VE∣ˉ
∣VR∣ˉ
∣VF∣ˉ
∣VA∣ˉ
∣E∣ˉ
∣EC∣ˉ
Reference
Gold
34.9
3.91
1.25
2.53
45.4
7.49
Claude-Opus-4.7
Claude Code (raw)
10.7
3.14
1.02
1.20
18.2
5.30
Claude-Opus-4.7
Claude Code (+Process)
34.9
2.09
0.74
1.24
39.5
3.56
Claude-Opus-4.7
CEG-Claude (ours)
34.9
4.69
1.06
2.96
45.4
6.87
GPT-5.5
Codex (raw)
34.9
1.64
1.00
0.07
37.2
1.64
GPT-5.5
Codex (+Process)
34.9
1.82
1.00
0.10
37.5
1.82
Appendix
Table 17: Mean per-trace graph sizes. Bold marks the closest within-backbone system to the gold reference.
Slice
ngold
Claude-Opus-4.7
GPT-5.5
CC-raw
CC-proc
CEG-Cl.
Cx-raw
Cx-proc
CEG-Cx
Mechanism recall on gold (ε,μ) keys
planning
77
11.7
14.3
23.4
0.0
1.3
7.8
evidence integration
72
1.4
11.1
20.8
0.0
0.0
13.9
execution
66
3.0
18.2
22.7
3.0
3.0
12.1
self-evaluation
61
1.6
23.0
23.0
14.8
18.0
19.7
Appendix
Table 18: Recall (%) by error mechanism and structural role. Best score within each backbone family is in bold .
System
Completion
Correct.
Constraint
Efficiency
Safety
Other
Coverage
Gold
53
43
17
6
3
3
6/6
Claude-Opus-4.7
Claude Code (raw)
62
21
19
0
0
0
3/6
Claude Code (+Process)
49
14
10
1
0
0
4/6
CEG-Claude (ours)
42
33
21
5
0
5
5/6
GPT-5.5
Appendix
Table 19: Failure-type emission counts across 100 traces. Coverage is the number of non-empty failure types emitted.
Table 21: Relative repair-value ranking on (ε,μ) -matched errors. Pairwise agreement measures whether predicted priorities preserve the gold ordering within each trace.
Cohort
n
Tau2Bench
SWE-bench
BrowseComp
All systems have SEE =0
19
10
3
6
Gold-empty subset
15
10
3
2
Non-empty all-fail subset
4
0
0
4
Appendix
Table 22: Traces on which all six systems obtain SEE =0 , decomposed by source.
CEG-Agent backbone
SRE
Structural
Textual
error
edge
chain
fail.
attr.
anom.
Claude-Opus-4.7
51.6
46.7
42.4
51.0
38.8
37.1
60.0
GLM-5.1
46.7
45.1
38.0
39.7
41.9
37.0
42.8
GPT-5.5
46.5
45.9
38.0
41.5
34.9
27.6
51.3
Gemini-3.1-Pro
44.2
40.8
41.0
37.3
35.7
26.5
43.1
Qwen3.7-Max
43.7
41.7
37.8
37.8
34.2
32.8
35.2
Appendix
Table 23: Per-dimension SRE scores of CEG-Agent under five LLM backbones. Best score per column is in bold . Utility dimensions ( event , cons. ) are at ceiling for all backbones and are omitted; the composite SRE in the second column already incorporates them.
CEG-Agent backbone
SEE
Dimensions
failure
error
attach
cause
contrib.
Claude-Opus-4.7
20.5
52.1
20.5
17.8
2.0
9.9
GLM-5.1
19.6
54.9
18.0
15.0
1.3
8.7
Qwen3.7-Max
18.7
46.1
18.9
16.1
2.3
10.2
GPT-5.5
16.9
48.3
15.2
13.0
1.5
6.3
Gemini-3.1-Pro
15.0
42.0
15.5
11.4
1.3
4.8
Appendix
Table 24: Per-dimension SEE scores of CEG-Agent under five LLM backbones. Best score per column is in bold .
Metric
CEG-Claude
CEG-Codex
CEG-Gemini
CEG-Qwen
CEG-GLM
Mean per trace
LLM calls
18.7
19.0
23.5
23.2
24.1
Agent iterations
13.5
14.0
18.2
18.0
18.8
Input tokens
556,039
350,076
467,519
622,982
436,678
Output tokens
3,832
4,832
12,567
19,063
20,332
Wall-clock time (s)
95.0
160.0
372.6
214.4
517.5
Appendix
Table 25: Cost profile of CEG-Agent under five LLM backbones on CEG-Bench.
Tool
CEG-Claude
CEG-Codex
CEG-Gemini
CEG-Qwen
CEG-GLM
find_failures
108
101
99
106
101
propose_errors
102
100
96
124
110
find_anomalies
101
100
94
103
100
build_causal_graph
102
100
87
106
100
estimate_repair_values
100
101
88
96
90
validate_graph
112
136
98
111
121
Appendix
Table 26: Total agent-issued tool invocations across the evaluated traces for each backbone. Infrastructure tools ( read_trace , build_events , render_graph , finalize ) and scaffolding tools ( write_todo , inspect_state ) are omitted from the table, since they are not central to the tool-use analysis.
Feedback
Tau2
Browse
SWE
generic
77.8
6.9
14.3
CEG: failure
91.7
10.3
17.1
CEG: errors
80.6
17.2
17.1
CEG: priority errors
83.3
17.2
8.6
Appendix
Table 27: Closed-loop repair pass@1 (%) under the feedback conditions used in the main paper.
Feedback
pass@1
apply-fail
test-fail
churn
test%
generic
14.3
13
17
5.6
9
CEG: failure
17.1
19
10
16.5
17
CEG: errors
17.1
14
15
9.7
14
CEG: priority errors
8.6
19
13
10.0
9
Appendix
Table 28: SWE-bench closed-loop breakdown ( n=35 ). apply-fail counts patches that fail to apply, test-fail counts patches that apply but fail tests, churn is the mean number of changed lines, and test% is the percentage of patches that edit test files.
Figure 5: CEG-Claude visualization for a Tau2Bench trace.
Figure 6: CEG-Claude visualization for a SWE-bench trace.
Figure 7: CEG-Claude visualization for a BrowseComp trace.
As multi-agent AI systems are increasingly deployed in real-world settings - from automated customer support to DevOps remediation - failures become harder to diagnose due to cascading effects, hidden dependencies, and long execution traces. We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent workflows. AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals - without requiring LLM inference at debugging time. Across a diverse benchmark of multi-agent failure scenarios designed to reflect common deployment patterns, AgentTrace localizes root causes with high accuracy and sub-second latency, significantly outperforming both heuristic and LLM-based baselines. Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.
Yunjia Qi, Zehua Yin, Xintong Shi +10
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University
Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.
Bo Deng, Xinlei Zheng, Yi Wei +6
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University