Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

Do Nguyen-Thanh; Junlin Fang; Bing Li; Zhen Fang; Sean Du

cs.LGOct 9, 2026

Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

Authors: Do Nguyen-Thanh, Junlin Fang, Bing Li, Zhen Fang, Sean Du

Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, University of Technology Sydney

Abstract

Failure attribution for LLM agents aims to identify decisive steps responsible for task failure, which is critical for debugging and improving agentic systems. Existing methods rely on costly LLM prompting, step-level error annotations, or verified successful trajectories that lack failure signals for assistance. In this paper, we study failure attribution from step-unlabeled failure trajectories, where each trajectory is known to have failed but its decisive step is unknown. We propose SOAP (Spectral scOring with Attention-guided Propagation), an effective framework requiring neither step-level training labels nor a separate success-only reference set. SOAP first constructs a spectral reference space from LLM representations and assigns each step a base error score according to its projection behavior. Importantly, later steps may exhibit stronger failure signals because they inherit an earlier mistake, causing independent scorers to favor downstream consequences over the decisive error. To address this downstream contamination, SOAP derives soft step dependencies from attention patterns and propagates downstream error evidence toward its likely upstream sources. Extensive experiments show that SOAP consistently improves attribution performance over competitive baselines.

Figures & tables

Figure 1: Downstream contamination in a failed Who&When trajectory [ 26 ] . An early wrong click (step 12) lands the browser on an irrelevant page; later steps are spent on recovery attempts that derail again.

Figure 2: Overview of Soap for agentic failure attribution. Soap constructs a spectral reference space from step-unlabeled failure trajectories and assigns each step a base error score. To address downstream contamination, Soap derives soft dependencies from attention patterns and propagates downstream error evidence toward likely upstream sources to localize the decisive error.

Figure 3: Base-score distributions on a Correct-Error test split [ 23 ] . Decisive-error steps exhibit lower projection energy than ordinary steps.

GPT-4o
Method	Supervised	WW-AG	WW-HC	CE	TE-Cap	TE-Mag
Prompt-based methods
All-at-Once [ 26 ]	✗	14.81	6.90	28.39	16.28	5.80
Step-by-Step [ 26 ]	✗	20.63	13.79	44.47	13.95	13.04
Binary Search [ 26 ]	✗	22.22	4.60	9.70	9.30	6.52
CORRECT [ 23 ]	✓	15.34	4.60	35.12	10.85	13.77

Table 1: Main comparison in the without-ground-truth setting. Results are step-level attribution accuracy (%), averaged over three seeds (std in Appendix C.1 ). No method receives the gold task answer during inference. Supervised marks methods that consume step-level error annotations. Best per column within each block in bold, second best underlined .

Figure 4: (a–b) Scalability to larger backbones , which show the step-level accuracy (%) on WW-AG and WW-HC using Qwen3-14B and Qwen3.5-27B; error bars denote standard deviations. (c) Cross-distribution generalization , with source datasets as rows and target datasets as columns; outlined diagonal entries correspond to the in-distribution results in Table 1 . (d) Results with synthetic trajectories generated by Qwen3.5-9B and GPT-4o. Panels (c) and (d) use Qwen3.5-9B as LLM $\mathcal{M}$ for embedding extraction and rescoring.

Prompt-based methods
Method	Sup.	WW-AG	WW-HC
All-at-Once	✗	21.16	11.49
Step-by-Step	✗	17.46	18.39
Binary Search	✗	2.12	13.79
CORRECT	✓	32.28	4.60
CHIEF	✗	30.69	4.60

Table 2: Comparison in the with-ground-truth setting. Results are step-level accuracy (%) on Who&When , with the gold task answer provided. All methods use Qwen3.5-9B as the backbone. Marks follow Table 1 .

Figure 5: Ablation studies. Sensitivity to (a) propagation strength

\gamma

, (b) representation layer

l^{\star}

, (c) attention layer band

L^{\star}

, and (d) the amount of unlabeled reference data. All experiments use Qwen3.5-9B. Panels (a–c) vary one hyperparameter at a time from the selected WW-AG configuration used in Table 1 . Dark bars and enlarged markers denote the selected values; the orange dashed line denotes the base score without rescoring (

\gamma=0

), and the dotted curve in (b) reports the base score at each layer. Panel (d) constructs

R

using

10\%

20\%

, or

30\%

of the reference corpus and reports results on WW-AG and WW-HC . Best viewed in color.

Table 8

Figure 6: Qualitative example. Per-step scores on a TE-Cap trajectory with backbone Qwen3.5-9B. The base score (dotted) peaks late; Soap (solid) lifts the decisive error

t^{\star}

(orange marker) to the top. The large square and circle mark each score’s argmax.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	Agent system	Traj.	Turns			Agents	Decisive step
			mean	med.	max		mean
WW-AG	CaptainAgent	126	8.7	10	10	3.6	3.0
WW-HC	Magentic-One	58	51.6	32	130	5.7	15.2
TE-Cap	CaptainAgent	85	20.5	13	59	2.7	6.0
TE-Mag	Magentic-One	91	29.3	28	50	2.1	11.4
CE-ARC	CORRECT	304	5.3	2	22	2.3	2.0

Appendix

Table 5: Corpus statistics per subset: number of trajectories, turns per trajectory (mean, median, maximum), distinct agents per trajectory, and the mean position of the decisive error (0-indexed). The CE row is the unweighted mean over its seven subsets.

GPT-4o
Method	WW-AG	WW-HC	CE	TE-Cap	TE-Mag
All-at-Once	15.87	3.45	24.50	9.30	6.52
Step-by-Step	22.75	18.39	59.29	18.60	19.57
Binary Search	28.57	4.60	18.44	9.30	6.52
CORRECT	15.87	4.60	37.09	10.85	10.14
CHIEF	24.34	11.49	42.10	14.73	13.77

Appendix

Table 6: Step-level accuracy (%) in the with-ground-truth setting on all five subsets. The prompt-based rows use the judge named in the block; the training-based rows and Soap use the backbone named in the block. The best result in each column within a block is highlighted in bold .

	Fitting (s/fit)		Embedding extraction (s/forward pass)		CPU scoring (ms/trajectory)
Method	WW-AG	WW-HC	WW-AG	WW-HC	WW-AG	WW-HC
OAT	51.10	51.10	0.42	2.13	2.4	10.4
StepFinder	51.79	115.43	0.23	0.19	1.8	5.0
Soap	0.08	0.29	0.38	0.76	0.7	5.5

Appendix

Table 7: Fitting and inference time on Who&When with Qwen3.5-9B. Fitting is per training seed (mean of five) for OAT and StepFinder, one CPU SVD fit for Soap . Extraction is one forward pass of the backbone.

Method	AG	HC
All-at-Once	2.0	2.1
Step-by-Step	2.0	3.6
Binary Search	0.2	0.4
CORRECT	1.9	1.6
CHIEF	23.9	23.5
ErrorProbe	5.9	6.5

Appendix

Table 8: Time per trajectory (s) on the shared Who&When subset.

Backbone	Subset	$w=1$	$2$	$3$	$4$	$5$	all
Qwen3.5-9B	WW-AG	47.62	41.80	40.74	40.21	40.21	40.21
	WW-HC	33.33	34.48	32.18	32.18	32.18	32.18
	TE-Cap	28.68	29.46	34.88	34.88	35.66	34.88
	TE-Mag	15.94	20.29	18.84	23.19	18.84	18.12
DeepSeek-8B	WW-AG	45.50	38.62	33.33	34.39	37.57	37.57
	WW-HC	28.74	28.74	28.74	28.74	28.74	28.74

Appendix

Table 9: Context window

w

of the dependency weights. Step-level accuracy (%) when each step keeps its

w

strongest predecessors, everything else at the selected configuration; the selected

w

is highlighted in bold . DeepSeek-8B’s selected

\gamma

on WW-HC is 0, so that row is constant by construction.

Figure 7: Sensitivity of Soap on Qwen3.5-9B, all four subsets. As Figure 5 , with the window

w

added: (a) propagation strength

\gamma

, (b) window

w

, (c) representation layer

l^{\star}

(solid: after rescoring; dotted: base score at that layer; large marker: selected layer), (d) attention band

L^{\star}

. The dashed orange line denotes the base score of the selected configuration.

Figure 8: Sensitivity of Soap on DeepSeek-8B, all four subsets; panels as in Figure 7 . DeepSeek-8B sweeps every block in panel (c). On WW-HC the selected

\gamma

is 0, so the

w

and attention-band panels are flat by construction.

		Qwen3.5-9B				DeepSeek-8B
Pooling	Method	WW-AG	WW-HC	TE-Cap	TE-Mag	WW-AG	WW-HC	TE-Cap	TE-Mag
Last token	Base	29.10	22.99	24.03	21.01	29.10	22.99	24.81	24.64
	Soap	29.63	24.14	25.58	23.19	30.69	22.99	24.81	26.09
Mean (Ours)	Base	39.15	33.33	33.33	21.01	38.62	28.74	40.31	29.71
	Soap	47.62	34.48	35.66	23.19	45.50	28.74	42.64	30.43

Appendix

Table 10: Mean vs. last-token pooling of the step representation. We compare the step-level accuracy (%); the best result in each column is highlighted in bold .

Reference data	WW-AG	WW-HC	TE-Cap	TE-Mag
Qwen3.5-9B
10%	43.39	33.33	34.88	23.19
20%	44.44	33.33	34.88	23.19
30%	47.62	34.48	35.66	23.19
DeepSeek-R1-Distill-Llama-8B
10%	37.04	29.89	41.86	31.88

Appendix

Table 11: Quantity of unlabeled reference data, all cells. Step-level accuracy (%) of Soap when the reference is fit on

\{10,20,30\}\%

of the corpus, validation and test fixed, full configuration re-selected per fraction. The best result in each column within a block is highlighted in bold .

	WW-AG	WW-HC	TE-Cap	TE-Mag
Qwen3.5-9B
Longest step	19.05	19.54	10.08	11.59
Shortest step	0.53	0.00	4.65	2.90
Random	12.10	5.08	8.58	5.16
Unit-norm, anchor band	35.45	16.09	24.81	16.67
Unit-norm, re-selected	35.45	22.99	30.23	23.19

Appendix

Table 12: Trivial baselines and the unit-norm control. Step-level accuracy (%) on the test split; the best result in each column is bold .

Reference	WW-AG	WW-HC
Qwen3.5-9B
Random basis	33.33	25.29
MCP-Atlas successes	38.10	28.74
MCP-Atlas failures	42.86	28.74
Steps before $t^{\star}$ †	36.51	33.33
Random subset †	39.68	33.33

Appendix

Table 13: Effect of the reference corpus on Who&When . Step-level accuracy (%) on the test split. † Diagnostic references that read step labels on the reference split. The best result in each column within a block is highlighted in bold .

	WW-AG	WW-HC	TE-Cap	TE-Mag
Qwen3.5-9B
Probe	43.92	20.69	18.60	17.39
Soap	47.62	34.48	35.66	23.19
DeepSeek-R1-Distill-Llama-8B
Probe	35.45	19.54	20.93	20.29
Soap	45.50	28.74	42.64	30.43

Appendix

Table 14: Comparison with a supervised probe trained on validation labels. Step-level accuracy (%) on the test split. The best result in each column within a block is highlighted in bold .

GPT-4o
Method	WW-AG	WW-HC
All-at-Once	53.44	59.77
Step-by-Step	30.16	49.43
Binary Search	58.73	12.64
CORRECT	54.50	54.02
CHIEF	40.21	48.28

Appendix

Table 15: Agent-level accuracy (%) in the without-ground-truth setting on Who&When , for every row of Table 1 . Mean over the seed triple. The best result in each column within a block is highlighted in bold .

GPT-4o
Method	WW-AG	WW-HC	CE	TE-Cap	TE-Mag
All-at-Once	$14.81_{\,\pm 2.70}$	$6.90_{\,\pm 0.00}$	$28.39_{\,\pm 0.36}$	$16.28_{\,\pm 3.29}$	$5.80_{\,\pm 2.05}$
Step-by-Step	$20.63_{\,\pm 1.30}$	$13.79_{\,\pm 5.63}$	$44.47_{\,\pm 1.25}$	$13.95_{\,\pm 6.85}$	$13.04_{\,\pm 3.55}$
Binary Search	$22.22_{\,\pm 3.43}$	$4.60_{\,\pm 1.63}$	$9.70_{\,\pm 0.82}$	$9.30_{\,\pm 1.90}$	$6.52_{\,\pm 0.00}$
CORRECT	$15.34_{\,\pm 0.75}$	$4.60_{\,\pm 1.63}$	$35.12_{\,\pm 1.93}$	$10.85_{\,\pm 4.78}$	$13.77_{\,\pm 2.71}$
CHIEF	$25.93_{\,\pm 3.96}$	$14.94_{\,\pm 1.63}$	$38.56_{\,\pm 0.86}$	$13.95_{\,\pm 1.90}$	$8.70_{\,\pm 3.07}$

Appendix

Table 16: Step-level accuracy (%) in the without-ground-truth setting as mean

\pm

standard deviation over the three seeds of the frozen triple, for every row of Table 1 . OAT and StepFinder are first averaged over their five training seeds within each split.

	WW-AG			WW-HC			CE			TE-Cap			TE-Mag			MRR
Method	@1	@3	@5	@1	@3	@5	@1	@3	@5	@1	@3	@5	@1	@3	@5	WW-AG
Qwen3.5-9B
OAT	16.72	56.72	86.46	10.11	25.75	38.39	56.50	74.60	82.18	15.35	53.02	73.80	18.84	32.17	40.29	41.46
StepFinder	15.87	55.24	75.87	13.33	29.66	34.48	30.03	59.04	68.58	18.29	37.21	50.70	6.67	17.10	28.12	41.23
Soap	47.62	74.60	91.01	34.48	42.53	50.57	61.78	79.61	85.74	35.66	63.57	75.97	23.19	32.61	42.75	63.92
DeepSeek-R1-Distill-Llama-8B

Appendix

Table 17: Step-level accuracy at

k

(%) and mean reciprocal rank (MRR, %) in the without-ground-truth setting for the three methods that rank every step. The best result in each column within a block is highlighted in bold .

Method	WW-AG	WW-HC	TE-Cap	TE-Mag
Without GT
All-at-Once	18.52	4.60	10.85	10.87
Step-by-Step	24.34	18.39	17.83	6.52
Binary Search	41.80	21.84	24.03	20.29
CORRECT	24.87	13.79	20.93	23.91
CHIEF	27.51	25.29	22.48	16.67

Appendix

Table 18: Step-level accuracy (%) of the prompt-based methods with GPT-5 as the judge, without and with the ground-truth answer, on Who&When and TraceElephant . Correct-Error was not run with GPT-5. The Soap rows repeat Tables 1 and 2 . The best result in each column within a block is highlighted in bold .

Subset	Method	Qwen3.5-9B	Qwen3-14B	Qwen3.5-27B
WW-AG	OAT	$16.72_{\,\pm 1.56}$	$13.97_{\,\pm 1.48}$	$20.42_{\,\pm 1.40}$
	StepFinder	$15.87_{\,\pm 2.99}$	$22.65_{\,\pm 3.72}$	$13.54_{\,\pm 2.08}$
	Soap	$47.62_{\,\pm 2.59}$	$42.86_{\,\pm 4.67}$	$44.97_{\,\pm 2.70}$
WW-HC	OAT	$10.11_{\,\pm 1.13}$	$16.78_{\,\pm 2.00}$	$12.41_{\,\pm 1.13}$
	StepFinder	$13.33_{\,\pm 4.80}$	$14.02_{\,\pm 2.94}$	$16.55_{\,\pm 6.24}$
	Soap	$34.48_{\,\pm 2.82}$	$25.29_{\,\pm 6.50}$	$34.48_{\,\pm 2.82}$

Appendix

Table 19: Step-level accuracy (%) by backbone size on Who&When , mean

\pm

standard deviation (over the seed triple for Soap ; over the five training seeds for OAT and StepFinder). Qwen3-14B belongs to the previous model generation. The best result in each column is highlighted in bold .

Figure 9: Cross-distribution transfer, all grids. Step-level accuracy (%) of Soap with the reference fitted on the source subset (rows) and evaluated on the target subset (columns), for both backbones. The outlined diagonal is the in-distribution number of Table 1 in every grid.

Reference	WW-AG	WW-HC
Existing benchmarks	45.50	28.74
Synthetic, Qwen3.5-9B	39.68	29.89
Synthetic, GPT-4o	38.10	25.29

Appendix

Table 20: Synthetic references with DeepSeek-8B. Step-level accuracy (%) of Soap ; only the reference corpus changes.

Panel of Figure 10 — Figure 10: WW-AG CaptainAgent trajectory, scored with Qwen3.5-9B.

Panel of Figure 11 — Figure 11: WW-HC Magentic-One trajectory, scored with Qwen3.5-9B.

Panel of Figure 12 — Figure 12: TE-Cap CaptainAgent trajectory (traj-59), scored with Qwen3.5-9B. This is the trajectory whose score plot appears in Figure 6 .

Panel of Figure 13 — Figure 13: TE-Mag Magentic-One trajectory, scored with Qwen3.5-9B.

Failure Attribution for LLM Agents

See all 34 papers

Tracing Agentic Failure from the Flow of Success
Jul 14, 2026Samuel Yeh, Yiwen Zhu, Shaleen Deep +1Agent Failure Analysis Failure Attribution for LLM Agents
Department of Computer Science, University of Wisconsin-Madison · Microsoft Research
FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
May 30, 2026Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim +2Agent Failure Analysis Multi-Agent LLM Systems
SPEAR Lab Concordia University Montreal, Canada · DePaul University Chicago, USA
ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems
Oct 8, 2026Weilin Jin, Mingyu Wang, Taiyu Zhu +7Failure Attribution for LLM Agents Agent Failure Analysis
Peking University, Beijing, China · Joy Future Academy, Beijing, China · Tsinghua University, Beijing, China

Jul 14, 2026J/K move · Enter open · S save

Tracing Agentic Failure from the Flow of Success

Samuel Yeh, Yiwen Zhu, Shaleen Deep, Sharon Li

Department of Computer Science, University of Wisconsin-Madison · Microsoft Research

Agent Failure Analysis Failure Attribution for LLM Agents Anomaly Detection

Open paper AskPDF arXiv

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000

\times

faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.

Abstract

Failure Attribution for LLM Agents

Explore similar work

Tracing Agentic Failure from the Flow of Success

FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search

ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems