ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

Weilin Jin; Mingyu Wang; Taiyu Zhu; Ziqi Zhou; Wenbo Li; Haoyang Huang; Nan Duan; Yifan Wu

cs.AIOct 8, 2026

ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

Authors: Weilin Jin, Mingyu Wang, Taiyu Zhu, Ziqi Zhou, Wenbo Li, Haoyang Huang, Nan Duan, Yifan Wu, +2 more

Organizations: Peking University, Beijing, China · Joy Future Academy, Beijing, China · Tsinghua University, Beijing, China

Abstract

In LLM-based agent systems, failures can originate from early steps whose effects propagate through subsequent interactions, making their origins difficult to identify. To trace such failures back to their origin, failure attribution has been formulated as the task of identifying the earliest step responsible for the failure. Recent methods leverage LLM internal signals for failure attribution, typically using hidden states as step representations. We therefore conduct an empirical study to evaluate how effectively these representations distinguish root-cause steps from other steps and find limited separation. Motivated by this observation, we propose ReCast, a step representation learning method that transforms hidden states from a frozen LLM into attribution-oriented step representations. ReCast first selects attribution-relevant layers, then constructs complementary pattern and deviation features, and finally learns contextualized step representations through an encoder trained with contrastive and ranking objectives. We also introduce ReCast-2K, a training dataset for failure attribution. ReCast achieves the best Hit@1 across four benchmarks, surpassing the strongest baseline by 5.65 and 9.19 pp on Who&When Algorithm and Handcrafted, respectively. Code is available at https://anonymous.4open.science/r/ReCast-5FB6 .

Figures & tables

Figure 1: Comparison of text embeddings and last-layer hidden-state representations on failed trajectories from ReCast-2K and the Handcrafted and Algorithm subsets of Who&When. Lower ARS/GRS and higher SI indicate better separation. See Appendix A for metric definitions.

Figure 2: Overview of ReCast, which learns attribution-oriented step representations from frozen LLM hidden states for failure attribution.

Category	Method	Who&When		TraceElephant
Category	Method	Algorithm	Handcrafted	Captain	Magentic
Heuristic	First	16.13 $\pm$ 0.00	1.72 $\pm$ 0.00	7.06 $\pm$ 0.00	6.67 $\pm$ 0.00
Heuristic	Random	16.40 $\pm$ 4.85	5.75 $\pm$ 0.81	12.55 $\pm$ 6.10	5.93 $\pm$ 2.62
LLM-based	All-at-Once (Qwen)	26.61 $\pm$ 0.00	3.45 $\pm$ 0.00	7.06 $\pm$ 0.00	4.44 $\pm$ 0.00
	All-at-Once (DS)	22.04 $\pm$ 0.76	10.34 $\pm$ 2.44	12.55 $\pm$ 1.11	9.63 $\pm$ 0.52
	All-at-Once (GPT)	27.42 $\pm$ 0.66	18.39 $\pm$ 5.69	14.90 $\pm$ 4.74	8.15 $\pm$ 0.52

Table 1: Hit@1 attribution accuracy (%) on Who&When and TraceElephant. Results from three random seeds are summarized as mean

\pm

population standard deviation. The LLMs are Qwen3.5-27B (Qwen), DeepSeek V4 Flash (DS), and GPT-5.6 Luna (GPT). Best results are shown in bold , and second-best results are underlined .

Category	Method	AFTraj-2K	Who&When Pro
Heuristic	First	0.00 $\pm$ 0.00	7.35 $\pm$ 0.00
Heuristic	Random	11.45 $\pm$ 0.77	25.69 $\pm$ 0.69
LLM-based	All-at-Once (Qwen)	20.86 $\pm$ 0.00	21.10 $\pm$ 0.00
	All-at-Once (DS)	30.47 $\pm$ 5.02	25.95 $\pm$ 1.29
	All-at-Once (GPT)	19.02 $\pm$ 0.87	33.33 $\pm$ 1.27
	Step-by-Step (Qwen)	51.53 $\pm$ 0.00	60.33 $\pm$ 0.04

Table 2: Hit@1 attribution accuracy (%) on the two in-domain benchmark settings. Results from three random seeds are summarized as mean

\pm

population standard deviation. Best results are shown in bold , and second-best results are underlined .

Figure 3: Cumulative Hit@

k

accuracy on Who&When and TraceElephant. Stacked segments denote ranks 1, 2–3, and 4–5; error bars show the population standard deviation over three seeds.

Component	Variant	Hit@1	$\Delta$ Hit@1 (pp)
None	Full model	41.58 $\pm$ 1.04	—
Layer selection	All 64 layers	38.83 $\pm$ 2.30	$-2.75$
	Uniform 8	38.10 $\pm$ 2.55	$-3.48$
	Last 8	37.55 $\pm$ 0.93	$-4.03$
	Random 8	37.18 $\pm$ 0.69	$-4.40$
	Global probe top-8	38.83 $\pm$ 1.13	$-2.75$

Table 3: Ablation results on Who&When. Hit@1 (%) over all trajectories is reported as mean

\pm

standard deviation across three seeds.

\Delta

Hit@1 reports the change from the full-model mean in percentage points (pp).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Representation	ARS $\downarrow$	GRS $\downarrow$	SI $\uparrow$
Training	Text embedding	$0.5478$	$0.6173$	$0.1482$
	Hidden[-1] (Mean)	$0.7254$	$0.7630$	$0.0131$
	Hidden[-1] (Last)	$0.7832$	$0.7686$	$0.0228$
	Projected features	$0.5763$	$0.6001$	$0.0566$
	ReCast	$\mathbf{-0.2590}\pm 0.2116$	$\mathbf{-0.2749}\pm 0.1977$	$\mathbf{0.7032}\pm 0.1804$
Handcrafted	Text embedding	$0.5195$	$0.5885$	$0.0959$

Appendix

Table 4: Separation of step representations measured by ARS, GRS, and SI. ReCast results show mean

\pm

population standard deviation across seeds

6

20

, and

42

; other representations are fixed. Best results are bolded for each dataset and metric, based on mean values for ReCast.

Task Category	# Trajectories			# Steps		# Tokens
Task Category	Total	Train	Test	Avg.	Max	Avg.	Max
Embodied	311	249	62	30.0	30	1,444.6	1,636
STEM	1,517	1,213	304	5.6	30	4,199.5	23,786
Data Science	2,446	1,958	488	3.7	17	609.1	6,861
Deep Search	1,641	1,313	328	5.6	31	1,551.9	21,948
Coding	342	273	69	3.0	3	3,127.6	7,490

Appendix

Table 5: Composition of the Who&When Pro text-modality subset used in our experiments. Step counts refer to aligned model-action steps, and token counts are computed over the corresponding inputs.

Benchmark	MAS	Model	Initial	Final Dataset
Benchmark	MAS	Model	Runs	Succ.	Fail.	Total
AssistantBench	Captain-Agent	DeepSeek V4 Flash	165	20	53	73
		GPT-4o-0806	165	14	38	52
		Qwen3-235B	165	20	53	73
		Subtotal	495	54	144	198
GAIA	Captain-Agent	DeepSeek V4 Flash	451	257	189	446

Appendix

Table 6: Composition of our source training dataset. Initial runs are the aligned executions before filtering. The final total consists of retained successful trajectories and failed trajectories with root-cause annotations. Additional generation runs are grouped under Others.

Model	Type	# Trajectories	# Steps		# Tokens
Model	Type	# Trajectories	Avg.	Max	Avg.	Max
DeepSeek V4 Flash	Successful	454	15.7	61	7,005.5	59,153
DeepSeek V4 Flash	Failed	500	21.6	55	10,603.5	74,773
GPT-4o-0806	Successful	225	18.5	60	4,460.0	22,757
GPT-4o-0806	Failed	515	32.0	61	7,996.1	29,094
Qwen3-235B	Successful	359	18.5	62	4,568.5	21,378

Appendix

Table 7: Statistics of ReCast-2K. Steps count recorded model responses, while token counts cover the question prefix and serialized trajectory using the Qwen3.5-27B tokenizer. Overall averages are computed over all trajectories.

Component	Hyperparameter	Value
Backbone	Hidden-state dimension $d_{h}$	5,120
Layer selection	Cross-validation folds	5
	Layer budget $k$	8
	Ridge ratio	0.10
	Selected layers	$\{7,15,19,32,33,47,49,57\}$
Projection	Dimension per branch $d_{p}$	2,048

Appendix

Table 8: Hyperparameter settings for our method.

Method	LR	Weight decay	Batch	Budget
OAT	$10^{-4}$	$10^{-5}$	32	300 epochs
StepFinder	$10^{-3}$	$10^{-5}$	16	50 epochs
ASCon	$5{\times}10^{-5}$	$10^{-5}$	1	12 epochs

Appendix

Table 9: Optimization settings for trainable feature-based baselines.

Figure 4: MRR@

K

across rank cutoffs

K=1,\ldots,5

on the four benchmarks. Curves show the cumulative reciprocal-rank score as the cutoff increases; error bars denote the population standard deviation over three seeds.

Figure 5: Receiver operating characteristic (ROC) curves on the four benchmarks. Each curve is computed by pooling the candidate-step predictions within a benchmark. Solid lines show the mean over three seeds, and shaded regions indicate the population standard deviation.

Figure 6: Precision–recall (PR) curves on the four benchmarks. Each curve is computed by pooling the candidate-step predictions within a benchmark. Solid lines show the mean over three seeds, and shaded regions indicate the population standard deviation.

Backbone	# Layers	Hidden size	Selected layers
Qwen3.5-0.8B	24	1,024	3, 5, 8, 11, 13, 18, 19, 22
Qwen3.5-9B	32	4,096	3, 7, 11, 16, 17, 23, 26, 31
Qwen3.5-27B	64	5,120	7, 15, 19, 32, 33, 47, 49, 57

Appendix

Table 10: Hidden-state backbone configurations. Each backbone retains eight layers selected independently using the same source-only procedure.

Evaluation population	Backbone	Hit@1 (%)	Hit@3 (%)	Hit@5 (%)	MRR (%)	AUROC (%)	AUPRC (%)
Who&When Handcrafted	Qwen3.5-0.8B	30.46 $\pm$ 2.15	41.95 $\pm$ 2.15	55.75 $\pm$ 0.81	41.80 $\pm$ 1.07	80.00 $\pm$ 2.12	19.09 $\pm$ 0.84
	Qwen3.5-9B	29.89 $\pm$ 3.54	44.83 $\pm$ 3.72	52.87 $\pm$ 4.30	41.61 $\pm$ 0.83	79.94 $\pm$ 2.41	18.55 $\pm$ 2.91
	Qwen3.5-27B	33.33 $\pm$ 1.63	45.98 $\pm$ 0.81	57.47 $\pm$ 0.81	44.21 $\pm$ 0.77	79.22 $\pm$ 0.91	21.88 $\pm$ 0.92
Who&When Algorithm	Qwen3.5-0.8B	38.17 $\pm$ 2.31	74.73 $\pm$ 2.31	91.94 $\pm$ 1.14	59.20 $\pm$ 2.00	72.00 $\pm$ 0.38	33.42 $\pm$ 0.43
	Qwen3.5-9B	38.98 $\pm$ 0.38	74.46 $\pm$ 2.01	92.74 $\pm$ 2.37	59.85 $\pm$ 0.67	73.64 $\pm$ 1.58	35.10 $\pm$ 0.47
	Qwen3.5-27B	45.43 $\pm$ 2.01	72.31 $\pm$ 1.01	92.47 $\pm$ 1.66	63.28 $\pm$ 1.42	74.75 $\pm$ 1.00	38.52 $\pm$ 1.84

Appendix

Table 11: Effect of the hidden-state extraction backbone. Results report mean

\pm

population standard deviation over three encoder seeds. Bold indicates the best mean within each evaluation population, determined before rounding.

Figure 7: Hit@1 accuracy (%) versus measured time per trajectory on Who&When and TraceElephant. Points show three-seed means; time is in seconds on a logarithmic scale. Feature-based methods report attribution time excluding feature extraction, while LLM methods report inference time excluding backend initialization. Upper-left positions indicate higher accuracy at lower measured cost. A and S denote All-at-Once and Step-by-Step.

Figure 8: Step-level attribution scores on three HC trajectories (a–c) and three AG trajectories (d–f). Original zero-based step indices are retained; gaps indicate events without candidate scores.

Explore similar work

FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
May 30, 2026Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim +2Agent Failure Analysis Multi-Agent LLM Systems
SPEAR Lab Concordia University Montreal, Canada · DePaul University Chicago, USA
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Oct 8, 2026Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu +5Agent Failure Analysis Multi-Agent LLM Systems
College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Mechanical Engineering, Tianjin University, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology; National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China
StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
Jun 2, 2026Taiyu Zhu, Yifan Wu, Weilin Jin +2Agent Failure Analysis Multi-Agent LLM Systems
Peking University Beijing, China

May 30, 2026, cs.AIJ/K move · Enter open · S save

FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search

Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim, Zhijie Wang+1

SPEAR Lab Concordia University Montreal, Canada · DePaul University Chicago, USA

Agent Failure Analysis Multi-Agent LLM Systems

Open paper AskPDF arXiv

LLM-based agents increasingly solve complex tasks through long trajectories involving reasoning steps, tool calls, and inter-agent communication. However, when these agents fail, it is often unclear which agent caused the failure and which step introduced the decisive error. This attribution problem is challenging because mistakes can propagate across the trajectory: later actions may appear incorrect, but only because they depend on an earlier corrupted state. Therefore, failure attribution cannot be treated as independent step-level classification. We propose FALAT, a diagnostic framework for failure attribution in LLM agent trajectories. FALAT frames attribution as a dependency-guided search problem. It first constructs an expectation of how the task should be solved and uses this expectation to identify suspicious regions in the trajectory. It then traces dependencies among decisions, tool outputs, and agent messages to distinguish error-introducing steps from steps that merely inherit or propagate prior mistakes. Finally, FALAT evaluates whether correcting a candidate step would be sufficient to recover the expected outcome, allowing it to identify both the responsible agent and the decisive failure step. We evaluate FALAT on the Who&When benchmark, which includes both algorithm-generated and hand-crafted multi-agent failure trajectories. The results show that FALAT consistently improves responsible-agent and decisive-step attribution. Its best configurations achieve 46.0% step-level accuracy on algorithm-generated trajectories and 29.1% on the more challenging hand-crafted trajectories, outperforming specialized attribution baselines and direct prompting with standalone LLMs. These findings suggest that dependency-aware reasoning is essential for reliable failure diagnosis in LLM agent systems.

Abstract

Explore similar work

FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems