From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis

Shu-Xun Yang; Yidong Wang; Zhuoer Feng; Bosi Wen; Jiayi Gui; Dayong Yang; Wenbo Yu; Haoke Zhang

cs.AISep 26, 2026

From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis

Authors: Shu-Xun Yang, Yidong Wang, Zhuoer Feng, Bosi Wen, Jiayi Gui, Dayong Yang, Wenbo Yu, Haoke Zhang, +2 more

Organizations: Beijing Institute of Technology, Beijing, China · Zhipu AI, Beijing, China · Tsinghua University, Beijing, China

Abstract

LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose CEG-Agent, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs Causal Error Graphs (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct CEG-Bench, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an Adversarial Agentic Adjudication Protocol (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.

Figures & tables

Figure 1: Comparison between prior paradigm and our method for agentic trace diagnosis.

Figure 2: The overall design of CEG-Agent.

Figure 3: Construction and evaluation overview of CEG-Bench.

Figure 4: AAAP–human-gold agreement per SRE dimension over all 100 traces.

Backbone	System	SRE	Structural			Textual			Utility
			error	edge	chain	fail.	attr.	anom.	event	cons.
Claude-Opus-4.7	Claude Code (raw)	45.8	39.5	37.1	38.8	44.7	48.7	44.7	78.1	100.0
Claude-Opus-4.7	Claude Code (+Process)	49.6	49.2	41.8	38.2	47.7	43.9	46.0	100.0	100.0
Claude-Opus-4.7	CEG-Claude (ours)	51.6	46.7	42.4	51.0	38.8	37.1	60.0	100.0	99.9
GPT-5.5	Codex (raw)	32.0	22.8	31.8	27.4	29.9	25.5	2.0	98.0	100.0
GPT-5.5	Codex (+Process)	32.9	27.0	34.5	29.5	21.5	23.5	1.7	100.0	100.0

Table 1: Semantically relaxed evaluation ( SRE ) on CEG-Bench.

Backbone	System	SEE	Dimensions
			failure	error	attach	cause	contrib.
Claude-Opus-4.7	Claude Code (raw)	13.9	56.1	5.3	3.9	1.0	3.3
Claude-Opus-4.7	Claude Code (+Process)	19.3	47.3	18.5	16.3	1.1	13.2
Claude-Opus-4.7	CEG-Claude (ours)	20.5	52.1	20.5	17.8	2.0	9.9
GPT-5.5	Codex (raw)	11.7	43.8	5.6	6.0	0.0	3.1
GPT-5.5	Codex (+Process)	10.4	33.3	8.7	6.0	0.0	4.1

Table 2: Structurally exact evaluation ( SEE ) on CEG-Bench.

Backbone	SRE	SEE
Claude-Opus-4.7	51.6	20.5
GLM-5.1	46.7	19.6
GPT-5.5	46.5	16.9
Gemini-3.1-Pro	44.2	15.0
Qwen3.7-Max	43.7	18.7

Table 3: CEG-Agent composite SRE/SEE under five LLM backbones.

Feedback	Tau2	Browse	SWE
generic	77.8	6.9	14.3
CEG: failure	91.7	10.3	17.1
CEG: errors	80.6	17.2	17.1
CEG: priority errors	83.3	17.2	8.6

Table 4: Closed-loop repair pass@1 (%).

Category	Tool	Method	Mode	Effect on $S_{t}$
Inspection	read_trace	Trace inspection	read-only	Read the normalized trace $T$ .
Inspection	inspect_state	State inspection	read-only	Read the current graph $G_{t}$ and todo buffer $\mathcal{Q}_{t}$ .
Phase	build_events	Event construction	generator	Write event nodes $V_{E}$ .
Phase	find_failures	Failure analysis	generator	Write failure nodes $V_{F}$ .
Phase	propose_errors	Error hypothesis generation	generator	Write causally relevant error nodes $V_{R}$ .
Phase	build_causal_graph	Causal graph induction	generator	Write causal edges $E_{C}$ .

Table 5: Registered tool set

\mathcal{U}

of CEG-Agent. Tools are grouped by category, linked to the corresponding method component, and summarized by their execution mode and effect on the diagnosis state

S_{t}

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Tau2	SWE	Browse	All
	$n=36$	$n=35$	$n=29$	$n=100$
Events	13.3	47.8	46.1	34.9
Errors	2.0	3.9	6.2	3.9
Failures	0.9	1.4	1.5	1.2
Anomalies	2.3	2.5	2.8	2.5
Edges	18.0	57.7	64.6	45.4

Appendix

Table 6: Mean per-trace component counts in CEG-Bench by source benchmark. Tau2, SWE, and Browse denote Tau2Bench, SWE-bench, and BrowseComp.

Label	Count	%
Mechanism $\mu$ ( errors , $n=391$ )
planning	77	19.7
evidence integration	72	18.4
execution	66	16.9
self-evaluation	61	15.6
control	53	13.6

Appendix

Table 7: Marginal distributions over the controlled vocabularies in CEG-Bench. Percentages for

\mu

\rho

, and

\rho^{\sharp}

are computed over the 391 annotated errors; percentages for

\theta

are computed over the 125 annotated failures.

Mechanism	high	med.	low	Total
planning	0	40	37	77
evidence integration	3	19	50	72
execution	3	18	45	66
self-evaluation	0	14	47	61
control	1	15	37	53
omission	2	11	20	33

Appendix

Table 8: Repair-value buckets cross-tabulated by error mechanism. Counts are computed over the 391 annotated errors in CEG-Bench.

Criterion	Dimension	Agreement
SRE	CEG-Sim (overall)	71.5
	anomaly	88.1
	error	76.3
	failure	76.3
	main_chain	61.9
	attribution	58.8

Appendix

Table 9: Agreement between the AAAP annotations in CEG-Bench and the expert-curated CEG-Bench-Gold, measured with SRE and SEE over all 100 traces (%). SEE agreement by complexity tercile is

43.0/45.8/52.4

for small/medium/large traces.

Backbone	System	SRE	Structural			Textual			Utility
			error	edge	chain	fail.	attr.	anom.	event	cons.
Claude-Opus-4.7	Claude Code (raw)	38.1	36.3	26.5	24.8	49.1	24.4	43.4	61.4	100.0
Claude-Opus-4.7	Claude Code (+Process)	38.1	35.6	26.0	23.6	40.5	23.8	46.5	99.0	100.0
Claude-Opus-4.7	CEG-Claude (ours)	45.4	43.2	30.2	37.1	45.0	30.6	57.6	99.0	99.9
GPT-5.5	Codex (raw)	27.7	21.4	22.2	15.1	38.3	16.5	6.9	99.0	100.0
GPT-5.5	Codex (+Process)	26.6	21.7	23.5	15.1	26.6	12.4	6.9	99.0	100.0

Appendix

Table 10: Semantically relaxed evaluation ( SRE ) against the expert-curated CEG-Bench-Gold , using the same layout as Table 1 . Best score within each backbone block is in bold . Scores are lower than those against the AAAP reference, but the within-family ranking is preserved.

Backbone	System	SEE	Dimensions
			failure	error	attach	cause	contrib.
Claude-Opus-4.7	Claude Code (raw)	15.5	60.1	8.5	6.1	0.5	2.2
Claude-Opus-4.7	Claude Code (+Process)	19.5	49.0	18.4	17.4	1.3	11.3
Claude-Opus-4.7	CEG-Claude (ours)	22.6	57.9	22.4	18.3	2.0	12.4
GPT-5.5	Codex (raw)	16.3	55.3	8.9	8.7	0.0	8.6
GPT-5.5	Codex (+Process)	13.3	39.5	9.2	8.4	1.7	7.5

Appendix

Table 11: Structurally exact evaluation ( SEE ) against the expert-curated CEG-Bench-Gold , using the same layout as Table 2 . Best score within each backbone block is in bold .

Backbone	Evaluation	System	Tau2Bench	SWE-bench	BrowseComp
			$n=36$	$n=35$	$n=29$
Claude-Opus-4.7	SRE	Claude Code (raw)	59.8	37.3	38.7
Claude-Opus-4.7	SRE	Claude Code (+Process)	67.2	38.3	41.5
Claude-Opus-4.7	SRE	CEG-Claude (ours)	60.0	47.7	45.9
Claude-Opus-4.7	SEE	Claude Code (raw)	16.8	11.1	13.8
Claude-Opus-4.7	SEE	Claude Code (+Process)	24.9	13.5	19.3

Appendix

Table 12: Per-source composite scores on CEG-Bench. Best score within each backbone–evaluator block is in bold .

Backbone	System	Small	Medium	Large
		$\bar{\|V_{E}\|}\approx 9$	$\bar{\|V_{E}\|}\approx 23$	$\bar{\|V_{E}\|}\approx 72$
Claude-Opus-4.7	Claude Code (raw)	16.6	10.9	14.2
Claude-Opus-4.7	Claude Code (+Process)	23.0	14.9	20.0
Claude-Opus-4.7	CEG-Claude (ours)	22.9	17.0	21.4
GPT-5.5	Codex (raw)	10.7	9.3	15.0
GPT-5.5	Codex (+Process)	9.1	3.9	18.0

Appendix

Table 13: SEE composite by trace-complexity tercile. Best score within each backbone block is in bold .

Backbone	System	SRE		SEE		Gold-empty
		empty	non-empty	empty	non-empty	abstain	error
Claude-Opus-4.7	Claude Code (raw)	64.8	42.5	0.0	16.4	1/15	14/15
Claude-Opus-4.7	Claude Code (+Process)	87.1	43.0	0.0	22.7	11/15	4/15
Claude-Opus-4.7	CEG-Claude (ours)	68.6	48.6	0.0	24.1	0/15	14/15
GPT-5.5	Codex (raw)	59.9	27.1	0.0	13.7	0/15	15/15
GPT-5.5	Codex (+Process)	60.0	28.1	0.0	12.3	0/15	15/15

Appendix

Table 14: Performance split by gold-empty and non-empty CEGs. abstain counts empty predictions; error counts predictions with at least one error on gold-empty traces.

Slice	System	SRE	error	edge	chain	fail.	attr.	anom.	event	cons.
Gold-empty ( $n=15$ )
empty	Claude Code (raw)	64.8	6.7	100.0	100.0	6.7	100.0	37.2	80.7	100.0
empty	Claude Code (+Process)	87.1	73.3	100.0	100.0	73.3	100.0	51.3	100.0	100.0
empty	CEG-Claude	68.6	6.7	100.0	100.0	0.0	100.0	72.8	100.0	99.9
empty	Codex (raw)	59.9	0.0	100.0	100.0	0.0	100.0	0.0	98.8	100.0
empty	Codex (+Process)	60.0	0.0	100.0	100.0	0.0	100.0	0.0	100.0	100.0

Appendix

Table 15: SRE per-dimension scores split by gold-empty (

n=15

) and non-empty (

n=85

) subsets, with columns aligned to main-paper Table 1 . On the empty subset the structural dimensions edge , chain , and attr. are mechanically

100

because gold and prediction are both empty, and the utility dimensions event / cons. are at or near ceiling for all systems; only error , fail. , and anom. are informative.

Slice	System	SEE	failure	error	attach	cause	contrib.
Gold-empty ( $n=15$ )
empty	Claude Code (raw)	0.0	0.0	0.0	0.0	0.0	0.0
empty	Claude Code (+Process)	0.0	0.0	0.0	0.0	0.0	0.0
empty	CEG-Claude	0.0	0.0	0.0	0.0	0.0	0.0
empty	Codex (raw)	0.0	0.0	0.0	0.0	0.0	0.0
empty	Codex (+Process)	0.0	0.0	0.0	0.0	0.0	0.0

Appendix

Table 16: SEE per-dimension scores split by gold-empty (

n=15

) and non-empty (

n=85

) subsets. On the empty subset SEE is

0

for every system because the gold graph contains no errors, failures, or edges to match.

Backbone	System	$\bar{\|V_{E}\|}$	$\bar{\|V_{R}\|}$	$\bar{\|V_{F}\|}$	$\bar{\|V_{A}\|}$	$\bar{\|E\|}$	$\bar{\|E_{C}\|}$
Reference	Gold	34.9	3.91	1.25	2.53	45.4	7.49
Claude-Opus-4.7	Claude Code (raw)	10.7	3.14	1.02	1.20	18.2	5.30
Claude-Opus-4.7	Claude Code (+Process)	34.9	2.09	0.74	1.24	39.5	3.56
Claude-Opus-4.7	CEG-Claude (ours)	34.9	4.69	1.06	2.96	45.4	6.87
GPT-5.5	Codex (raw)	34.9	1.64	1.00	0.07	37.2	1.64
GPT-5.5	Codex (+Process)	34.9	1.82	1.00	0.10	37.5	1.82

Appendix

Table 17: Mean per-trace graph sizes. Bold marks the closest within-backbone system to the gold reference.

Mechanism recall on gold $(\varepsilon,\mu)$ keys
Slice	$n_{\text{gold}}$	Claude-Opus-4.7			GPT-5.5
		CC-raw	CC-proc	CEG-Cl.	Cx-raw	Cx-proc	CEG-Cx
planning	77	11.7	14.3	23.4	0.0	1.3	7.8
evidence integration	72	1.4	11.1	20.8	0.0	0.0	13.9
execution	66	3.0	18.2	22.7	3.0	3.0	12.1
self-evaluation	61	1.6	23.0	23.0	14.8	18.0	19.7

Appendix

Table 18: Recall (%) by error mechanism and structural role. Best score within each backbone family is in bold .

System	Completion	Correct.	Constraint	Efficiency	Safety	Other	Coverage
Gold	53	43	17	6	3	3	6/6
Claude-Opus-4.7
Claude Code (raw)	62	21	19	0	0	0	3/6
Claude Code (+Process)	49	14	10	1	0	0	4/6
CEG-Claude (ours)	42	33	21	5	0	5	5/6
GPT-5.5

Appendix

Table 19: Failure-type emission counts across 100 traces. Coverage is the number of non-empty failure types emitted.

Gold	CEG-Claude prediction			CEG-Codex prediction
	low	medium	high	low	medium	high
low	34	13	5	39	0	0
medium	14	7	9	17	0	0
high	1	1	0	0	0	1
Column total	49	21	14	56	0	1

Appendix

Table 20: Absolute repair-value confusion on

(\varepsilon,\mu)

-matched errors. Diagonal cells indicate exact bucket agreement.

System	Matched errors	Pairwise agree.	Kendall’s $\tau$	Bucket acc.
Claude Code (+Process)	65	0.528	$+0.056$	0.062
Codex (+Process)	23	0.500	$\phantom{+}0.000$	0.000
CEG-Claude	84	0.700	$\boldsymbol{+0.400}$	0.488
CEG-Codex	57	0.500	$\phantom{+}0.000$	0.702

Appendix

Table 21: Relative repair-value ranking on

(\varepsilon,\mu)

-matched errors. Pairwise agreement measures whether predicted priorities preserve the gold ordering within each trace.

Cohort	$n$	Tau2Bench	SWE-bench	BrowseComp
All systems have SEE $=0$	19	10	3	6
Gold-empty subset	15	10	3	2
Non-empty all-fail subset	4	0	0	4

Appendix

Table 22: Traces on which all six systems obtain SEE

=0

, decomposed by source.

CEG-Agent backbone	SRE	Structural			Textual
		error	edge	chain	fail.	attr.	anom.
Claude-Opus-4.7	51.6	46.7	42.4	51.0	38.8	37.1	60.0
GLM-5.1	46.7	45.1	38.0	39.7	41.9	37.0	42.8
GPT-5.5	46.5	45.9	38.0	41.5	34.9	27.6	51.3
Gemini-3.1-Pro	44.2	40.8	41.0	37.3	35.7	26.5	43.1
Qwen3.7-Max	43.7	41.7	37.8	37.8	34.2	32.8	35.2

Appendix

Table 23: Per-dimension SRE scores of CEG-Agent under five LLM backbones. Best score per column is in bold . Utility dimensions ( event , cons. ) are at ceiling for all backbones and are omitted; the composite SRE in the second column already incorporates them.

CEG-Agent backbone	SEE	Dimensions
		failure	error	attach	cause	contrib.
Claude-Opus-4.7	20.5	52.1	20.5	17.8	2.0	9.9
GLM-5.1	19.6	54.9	18.0	15.0	1.3	8.7
Qwen3.7-Max	18.7	46.1	18.9	16.1	2.3	10.2
GPT-5.5	16.9	48.3	15.2	13.0	1.5	6.3
Gemini-3.1-Pro	15.0	42.0	15.5	11.4	1.3	4.8

Appendix

Table 24: Per-dimension SEE scores of CEG-Agent under five LLM backbones. Best score per column is in bold .

Metric	CEG-Claude	CEG-Codex	CEG-Gemini	CEG-Qwen	CEG-GLM
Mean per trace
LLM calls	18.7	19.0	23.5	23.2	24.1
Agent iterations	13.5	14.0	18.2	18.0	18.8
Input tokens	556,039	350,076	467,519	622,982	436,678
Output tokens	3,832	4,832	12,567	19,063	20,332
Wall-clock time (s)	95.0	160.0	372.6	214.4	517.5

Appendix

Table 25: Cost profile of CEG-Agent under five LLM backbones on CEG-Bench.

Tool	CEG-Claude	CEG-Codex	CEG-Gemini	CEG-Qwen	CEG-GLM
find_failures	108	101	99	106	101
propose_errors	102	100	96	124	110
find_anomalies	101	100	94	103	100
build_causal_graph	102	100	87	106	100
estimate_repair_values	100	101	88	96	90
validate_graph	112	136	98	111	121

Appendix

Table 26: Total agent-issued tool invocations across the evaluated traces for each backbone. Infrastructure tools ( read_trace , build_events , render_graph , finalize ) and scaffolding tools ( write_todo , inspect_state ) are omitted from the table, since they are not central to the tool-use analysis.

Feedback	Tau2	Browse	SWE
generic	77.8	6.9	14.3
CEG: failure	91.7	10.3	17.1
CEG: errors	80.6	17.2	17.1
CEG: priority errors	83.3	17.2	8.6

Appendix

Table 27: Closed-loop repair pass@1 (%) under the feedback conditions used in the main paper.

Feedback	pass@1	apply-fail	test-fail	churn	test%
generic	14.3	13	17	5.6	9
CEG: failure	17.1	19	10	16.5	17
CEG: errors	17.1	14	15	9.7	14
CEG: priority errors	8.6	19	13	10.0	9

Appendix

Table 28: SWE-bench closed-loop breakdown (

n=35

). apply-fail counts patches that fail to apply, test-fail counts patches that apply but fail tests, churn is the mean number of changed lines, and test% is the percentage of patches that edit test files.

Figure 5: CEG-Claude visualization for a Tau2Bench trace.

Figure 6: CEG-Claude visualization for a SWE-bench trace.

Figure 7: CEG-Claude visualization for a BrowseComp trace.

Explore similar work

AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
Mar 16, 2026Zhaohui Geoffrey WangAgent Failure Analysis AI Agent Reliability
USC Viterbi School of Engineering
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Aug 6, 2026Yunjia Qi, Zehua Yin, Xintong Shi +10Agent Failure Analysis Long-Horizon Agent Evaluation
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University
DeFA: Dependency-Guided Failure Attribution for LLM Agents
Oct 1, 2026Bo Deng, Xinlei Zheng, Yi Wei +6Agent Failure Analysis LLM Agent Evaluation
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University

Mar 16, 2026J/K move · Enter open · S save

AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems

Zhaohui Geoffrey Wang

USC Viterbi School of Engineering

Agent Failure Analysis AI Agent Reliability Multi-Agent Systems Causal Interpretability

Open paper AskPDF arXiv

As multi-agent AI systems are increasingly deployed in real-world settings - from automated customer support to DevOps remediation - failures become harder to diagnose due to cascading effects, hidden dependencies, and long execution traces. We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent workflows. AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals - without requiring LLM inference at debugging time. Across a diverse benchmark of multi-agent failure scenarios designed to reflect common deployment patterns, AgentTrace localizes root causes with high accuracy and sub-second latency, significantly outperforming both heuristic and LLM-based baselines. Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.

Abstract

Explore similar work

AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems

TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

DeFA: Dependency-Guided Failure Attribution for LLM Agents