LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories

Yunfei Zhang; Boyu Feng; Changhua Pei; Zexin Wang; Zhihuang Peng; Xinlong Liu; Hengyue Jiang; Difeng Ma

cs.AIAug 15, 2026

LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories

Authors: Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma,

Organizations: Computer Network Information Center, Chinese Academy of Sciences · Chongqing University · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences ·

Abstract

In long agent executions, an early error can persist through later actions and checks, while evidence needed to trace its origin is dispersed across the history. Short histories offer limited tests of recovering error origins across substantial subsequent execution. We introduce LongRCA Bench: 1,140 complete failed trajectories from five sources, all human-annotated for responsible roles and earliest decisive root-cause steps. Reference roots precede completion by a median of 48 recorded steps; 28.4% of trajectories contain over 100 subsequent steps. With DeepSeek-V4-Flash on the full benchmark, the strongest of five evaluated baselines achieves 13.2% exact root-step accuracy. We propose Root-Cause Trajectory Attribution (RCTA), a training-free method that organizes original candidate records and explicit handoff instructions for attribution. Segment summaries and a trajectory outline guide candidate retrieval; available handoff records supply upstream instruction context for the final instruction-execution comparison. With the same backbone and scoring protocol, RCTA reaches 24.1% exact root-step accuracy and 51.1% responsible-role accuracy. LongRCA-Mini provides 200 fixed trajectories for lower-cost comparative screening. Even with RCTA, fewer than one quarter of reference roots are recovered exactly.

Figures & tables

Figure 1: All six evaluated methods score lower on LongRCA across Agent Acc, Step Acc, and Step

\pm 5

. Accuracy (%) uses DS-V4-Flash on the complete Who&When (

n=184

) and LongRCA (

n=1{,}140

) sets, with each trajectory equally weighted within its dataset.

Benchmark	Error Source	Diagnostic Targets	Trajectories Total (Reviewed)	Avg. length (steps)
RootSE ( Wang et al., 2026a )	Natural	Step a	102 (102)	51.7
HORIZON ( Wang et al., 2026b )	Natural	Type	1995 (40)	–
TraceElephant ( Chen et al., 2026 )	Natural	Component, Step b	220 (220)	–
Failure as a Process ( Zhao et al., 2026 )	Natural	Steps c , Type	1184 (1184)	42.0
Who&When ( Zhang et al., 2025 )	Natural	Agent, Step a	184 (184)	22.2
Who&When Pro ( Liu et al., 2026 )	Synthetic	Agent, Step a , Type	12,326 (100)	7.5

Table 1: Comparison of resources for agent failure analysis. LongRCA supports responsible-role and earliest-root-step attribution over long execution trajectories collected without error injection and fully reviewed by humans.

Figure 2: A price-unit misinterpretation persists through planning and verification despite a local budget correction. The human reference identifies stay_agent and step 73 within a single output containing candidate records at steps 73–77. The displayed path follows the ultimately selected property from step 74. Green arrows indicate retrospective diagnosis using the final itinerary and TravelPlanner’s nightly pricing rule.

Figure 3: Long-horizon characteristics (

n=1,140

). (a) Medians (circles), means (diamonds), minimum–P90 ranges, and maxima of trajectory length, reference root step, and root-to-end distance. (b) Fractions with more than 50, 100, or 200 recorded steps after the reference root.

Figure 4: RCTA has two stages: Stage I constructs the candidate-information package (A1); Stage II retrieves available handoff evidence (A2) and performs backward instruction–execution reasoning (A3). Output and evidence validation (A4) checks the resulting attribution output and its evidence provenance.

Method	Overall			Per-source Step Acc
	Agent Acc	Step Acc	Step $\pm 5$	SWE	TB2	TP	Vita	WA
All-at-once	26.2	7.6	19.9	1.6	7.1	4.5	19.4	16.9
Step-by-step	22.2	5.3	16.9	0.8	4.8	3.5	12.0	11.3
Binary search	23.0	3.4	13.3	0.8	4.8	2.0	12.0	5.1
ECHO	27.5	13.2	24.7	7.8	21.4	8.9	22.2	26.0
FALAT	19.0	2.8	12.5	2.3	2.4	2.9	3.7	2.3

Table 2: Full-benchmark accuracies (%) using DS-V4-Flash. Overall weights trajectories equally; per-source columns report Step Acc. Bold marks the highest observed value in each column.

Figure 5: Step Acc on all 1,140 trajectories with DS-V4-Flash. Methods share length-only groups without splitting equal lengths. Panel (a) shows all groups; panel (b) magnifies the 0–12% region of groups 2–5. Unequal-width groups are equally spaced; connecting lines are visual guides.

Method	V4-Flash		V4-Pro		GPT-5.6 Sol		Kimi K3		Qwen3.5 35B-A3B
	Agent	Step	Agent	Step	Agent	Step	Agent	Step	Agent	Step
All-at-once	42.5	7.5	50.0	11.5	53.0	17.0	54.5	11.0	51.0	3.5
Step-by-step	38.5	6.5	47.5	10.5	49.0	13.0	44.0	9.5	46.5	4.5
Binary search	40.5	7.5	45.5	10.0	51.5	15.5	42.0	12.0	40.0	5.0
ECHO	45.0	13.0	53.5	18.0	55.5	18.5	55.0	20.5	48.5	10.0
FALAT	35.5	2.5	48.0	14.5	52.0	14.0	52.0	18.0	41.0	11.0

Table 3: Agent and Step Acc (%, Macro-5) on the fixed 200-trajectory Mini (40 per source). V4 denotes DeepSeek-V4. Bold marks the highest point estimate per backbone and metric; complete metrics and paired analyses are in Appendix B .

Setting	Agent Acc $\uparrow$	Step Acc $\uparrow$	Step $\pm 5\,\uparrow$	Cov. $\uparrow$	$p_{\mathrm{Holm}}$
RCTA (all components)	56.0	28.0	41.0	97.0	–
w/o candidate-information package (A1)	53.5	22.0	39.0	91.5	.17
w/o handoff evidence (A2)	53.5	24.5	38.0	97.0	.75
w/o backward attribution (A3)	51.0	25.0	41.0	97.5	.75
w/o validation (A4)	54.0	26.5	39.5	89.0	.75

Table 4: Paired ablations on the fixed Mini sample (Macro-5 percentages). Cov. measures valid-output coverage;

p_{\mathrm{Holm}}

adjusts the four Step Acc tests.

Figure 6: Component diagnostics. (a) Reference-root hit rates and final Step Acc (all

n=200

). (b) Step Acc on handoff-containing trajectories by reference root: handoff (

n=41

) or non-handoff (

n=39

). Full and A3 share the original excerpts. Error bars: 95% Wilson intervals; subgroup contrasts are exploratory.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Mini calibration with cached predictions and 10,000 source-stratified draws per size: maximum per-pair median and 95th-percentile absolute Macro-5 errors across 18 method–metric pairs.

Size	Quotas	Max. median	Max. p95	Pass
75	15/15/15/15/15	3.865	10.532	No
100	20/20/20/20/20	3.135	8.865	No
125	25/25/25/25/25	2.535	7.335	No
150	30/30/30/30/30	2.327	6.532	No
175	35/35/35/35/35	2.151	5.849	No
200	40/40/40/40/40	1.890	5.365	Yes

Appendix

Table 5: Six-method sample-size calibration with equal-quota simple-random sampling. Quotas follow the source order SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench, and WebArena Verified. For each of 18 method–metric pairs, absolute score errors are summarized over 10,000 draws. “Max. median” and “Max. p95” report the maximum per-pair median and 95th-percentile errors (pp). “Pass” requires only max. median

\leq 2

pp and max. p95

\leq 6

pp.

Method	Agent Acc	Step Acc	Step $\pm 5$
All-at-once	$48.2\rightarrow 48.5\ (0.3)$	$9.9\rightarrow 8.5\ (1.4)$	$23.7\rightarrow 21.5\ (2.2)$
Step-by-step	$41.8\rightarrow 41.5\ (0.3)$	$6.5\rightarrow 6.5\ (0.0)$	$19.8\rightarrow 17.5\ (2.3)$
Binary search	$40.8\rightarrow 39.5\ (1.3)$	$4.9\rightarrow 7.5\ (2.6)$	$18.1\rightarrow 20.0\ (1.9)$
ECHO	$49.4\rightarrow 46.5\ (2.9)$	$17.3\rightarrow 13.5\ (3.8)$	$31.2\rightarrow 29.0\ (2.2)$
FALAT	$33.1\rightarrow 34.5\ (1.4)$	$2.7\rightarrow 3.0\ (0.3)$	$15.5\rightarrow 18.0\ (2.5)$
RCTA	$\mathbf{60.9}\rightarrow\mathbf{61.0}\ (0.1)$	$\mathbf{28.3}\rightarrow\mathbf{27.0}\ (1.3)$	$\mathbf{44.1}\rightarrow\mathbf{41.0}\ (3.1)$

Appendix

Table 6: Fidelity of the fixed LongRCA-Mini manifest. Each cell reports Full

\rightarrow

Mini (

|\Delta|

) : Macro-5 accuracy (%) and absolute difference (pp). Across the 18 method–metric comparisons, the mean absolute deviation is 1.665 pp and the maximum deviation is 3.771 pp. All results use DS-V4-Flash.

Structural quantity	Full	Mini	Comparison
Trajectories	1,140	200	17.54% retained
Recorded steps	178,137	32,317	18.14% retained
History characters	207,731,218	41,784,984	20.11% retained
Macro-5 mean trajectory length	167.7	161.6	$-6.1$ steps
Macro-5 mean root-to-end distance	104.9	104.4	$-0.5$ steps
Macro-5 mean normalized root position	0.420	0.399	$-0.021$

Appendix

Table 7: Structural comparison of LongRCA Bench and the fixed LongRCA-Mini manifest. The KS statistic is the maximum across within-source comparisons of trajectory length, root-to-end distance, and normalized root position. History characters are used as a model-independent input-cost proxy.

Source	Steps	Input kchars	Root-to-end	Root position	KS
SWE	287.0/263.0	450.2/432.9	215.4/218.5	0.250/0.181	0.177
TB2	272.1/265.6	297.2/294.9	186.9/179.5	0.263/0.268	0.040
TP	165.9/167.8	163.3/154.1	65.9/67.8	0.597/0.569	0.134
Vita	54.6/50.8	23.2/26.9	29.7/26.0	0.445/0.473	0.134
WA	58.8/60.8	131.2/135.8	26.8/30.1	0.543/0.502	0.147

Appendix

Table 8: Within-source structural audit of the fixed manifest. Entries are Full/Mini means; input length is measured in thousands of history characters, and normalized root position is

r^{*}/(T-1)

. KS is the largest empirical CDF distance across the four reported characteristics. SWE, TB2, TP, Vita, and WA denote the five sources in the order used in Table 5 .

Method	Metric	SWE	TB2	TP	Vita	WA
All-at-once	Agent Acc	47.7/42.5	61.9/62.5	7.2/7.5	82.4/85.0	41.8/45.0
	Step Acc	1.6/2.5	7.1/5.0	4.5/5.0	19.4/20.0	16.9/10.0
	Step $\pm 5$	4.7/7.5	21.4/20.0	15.2/10.0	41.7/42.5	35.6/27.5
Step-by-step	Agent Acc	38.3/30.0	40.5/40.0	2.8/5.0	83.3/85.0	44.1/47.5
	Step Acc	0.8/2.5	4.8/5.0	3.5/0.0	12.0/12.5	11.3/12.5
	Step $\pm 5$	10.2/7.5	16.7/15.0	13.0/10.0	30.6/27.5	28.8/27.5

Appendix

Table 9: Per-source Full/Mini accuracy (%) for the fixed manifest. Each cell reports the full source score followed by the Mini source score. Source abbreviations follow Table 8 . These values are an audit of source-level variation, not evidence that 40 trajectories support definitive per-source comparisons.

Method	Agent Acc	Step Acc	Step $\pm 5$
All-at-once	$26.2\rightarrow 26.6\ (0.4)$	$7.6\rightarrow 6.9\ (0.7)$	$19.9\rightarrow 15.9\ (4.0)$
Step-by-step	$22.2\rightarrow 23.3\ (1.1)$	$5.3\rightarrow 3.6\ (1.7)$	$16.9\rightarrow 14.3\ (2.7)$
Binary search	$23.0\rightarrow 22.4\ (0.6)$	$3.4\rightarrow 7.7\ (4.3)$	$13.3\rightarrow 17.2\ (3.9)$
ECHO	$27.5\rightarrow 24.7\ (2.9)$	$13.2\rightarrow 9.6\ (3.5)$	$24.7\rightarrow 22.4\ (2.3)$
FALAT	$19.0\rightarrow 21.2\ (2.2)$	$2.8\rightarrow 2.8\ (0.0)$	$12.5\rightarrow 14.0\ (1.5)$
RCTA	$51.1\rightarrow 54.6\ (3.5)$	$24.1\rightarrow 24.7\ (0.5)$	$37.4\rightarrow 34.2\ (3.2)$

Appendix

Table 10: Original-mixture fidelity audit. Cells report Full

\rightarrow

post-stratified Mini (

|\Delta|

), in accuracy percentages and pp. Both sides use the original source weights, not Macro-5.

DS-V4-Flash
Method	Agent Acc $\uparrow$	Step Acc $\uparrow$	Step $\pm 5$ $\uparrow$	Joint Exact $\uparrow$	Joint $\pm 5$ $\uparrow$	MAE $\downarrow$
All-at-once	42.5	7.5	20.5	7.5	13.0	44.915
Step-by-step	38.5	6.5	18.0	6.5	10.0	41.340
Binary search	40.5	7.5	20.5	7.5	12.0	43.465
ECHO	45.0	13.0	28.0	13.0	18.0	40.930
FALAT	35.5	2.5	17.5	2.5	10.5	53.685

Appendix

Table 11: Complete six-metric record for the fixed 200-trajectory cross-backbone experiment. Accuracies are Macro-5 percentages; Step MAE is the reported valid-output mean absolute root-step error in steps. Bold marks the best value within each backbone and metric (including ties). Shading identifies RCTA. Evaluation settings and prediction coverage are detailed in the accompanying text.

Backbone	$\Delta$	95% CI	W/L	Holm-20 $p$	Holm-40 $p$
DS-V4-Flash	$+14.0$	$[7.5,20.5]$	39/11	0.00072	0.00217
DS-V4-Pro	$+10.0$	$[3.5,17.0]$	36/16	0.00779	0.06007
GPT-5.6 Sol	$+11.5$	$[4.5,19.0]$	40/17	0.00640	0.04162
Kimi K3	$+11.0$	$[5.0,17.0]$	31/9	0.00322	0.01155

Appendix

Table 12: Paired RCTA–ECHO Step Acc differences on LongRCA-Mini (

n=200

\Delta

and pointwise 95% intervals are in percentage points; W/L counts RCTA-only/ECHO-only correct predictions. Holm-20 adjusts the Step Acc family; Holm-40 jointly adjusts Agent Acc and Step Acc.

Excluded benchmark	Mean gain	95% CI
SWE-bench Pro	$+8.3$	$[3.6,13.1]$
Terminal-Bench 2	$+13.0$	$[8.6,17.5]$
TravelPlanner	$+11.4$	$[6.4,16.4]$
VitaBench	$+13.0$	$[7.8,18.1]$
WebArena Verified	$+12.5$	$[7.3,17.8]$

Appendix

Table 13: Leave-one-benchmark-out sensitivity of the four-backbone mean RCTA–ECHO Step Acc gain. Each row retains 160 trajectories from four benchmarks; differences and intervals are in percentage points.

Figure 8: Exact-root correctness on LongRCA-Mini (

n=200

per backbone). Both: RCTA and at least one of five baselines are correct; RCTA only: RCTA is correct and all five baselines are wrong; Baseline only: RCTA is wrong and at least one baseline is correct; Neither: all six methods are wrong. Segment labels are trajectory counts.

Field	Meaning
question_ID	Source-prefixed trajectory identifier
history	Ordered trajectory records containing step , name , role , and content ; includes the task instruction
mistake_agent	Reference responsible role, scored independently of the root-cause step
mistake_step	Reference 0-based root-cause step, scored independently of the responsible role
mistake_reason	Trajectory-grounded annotation rationale, withheld during prediction and not scored

Appendix

Table 14: Released LongRCA Bench data schema. Evaluated methods receive question_ID and history , which includes the task instruction. The three reference annotation fields are withheld during prediction.

Source	$N$	Task domain	Agent organization
SWE-bench Pro	128	Software repair	Diagnosis–execution–verification team
Terminal Bench 2	42	Terminal tasks	Diagnosis–execution–verification team
TravelPlanner	685	Travel planning	Magentic-One group chat
VitaBench	108	Service-oriented tool use	Action expert–computer terminal workflow
WebArena Verified	177	Web interaction	Planner–web-surfer–critic workflow
Total	1,140	5 task domains

Appendix

Table 15: Task domains and agent organizations of the retained LongRCA Bench trajectories. Counts are reported by source.

Source	$N$	Workflow	M2.5	K2.5	Q3.5+
SWE-bench Pro	128	Custom DiagnostAgent – ActionAgent – JudgeAgent team	128	–	–
Terminal-Bench 2	42	Custom DiagnostAgent – ActionAgent – JudgeAgent team	42	–	–
TravelPlanner	685	AutoGen MagenticOneGroupChat with specialist roles	265	260	160
VitaBench	108	Action_Expert – Computer_terminal workflow	69	16	23
WebArena Verified	177	AutoGen Planner – WebSurfer – Critic controller	177	–	–
Total	1,140		681	276	183

Appendix

Table 16: Execution provenance and generator counts for LongRCA Bench. The SWE-bench Pro and Terminal-Bench 2 execution records specify their custom role workflow and generator, but no separate third-party scaffold package. Generator columns correspond to MiniMax-M2.5 (M2.5), Kimi-K2.5 (K2.5), and Qwen3.5-Plus (Q3.5+). A dash indicates that the source contains no trajectory from that generator.

Figure 9: Step Acc (%) across five sources using DS-V4-Flash. All six methods use the same complete source subsets (

n

below source labels; total 1,140); invalid step outputs count as errors.

Method	SWE-bench Pro	Terminal Bench 2	TravelPlanner	VitaBench	WebArena Verified
All-at-once	1.6	7.1	4.5	19.4	16.9
Step-by-step	0.8	4.8	3.5	12.0	11.3
Binary search	0.8	4.8	2.0	12.0	5.1
ECHO	7.8	21.4	8.9	22.2	26.0
FALAT	2.3	2.4	2.9	3.7	2.3
RCTA	38.3	26.2	18.8	25.0	33.3

Appendix

Table 17: Full-benchmark Step Acc (%) by source, underlying Figure 9 . Source sample sizes are 128, 42, 685, 108, and 177, respectively. Every method uses the complete corresponding source subset; bold marks the highest observed accuracy in each source.

Method	10–63	64–129	130–172	173–220	221–728
All-at-once	21.4	6.2	2.8	0.9	1.9
Step-by-step	13.4	3.3	2.8	3.3	0.5
Binary search	7.2	3.8	0.5	2.8	1.4
ECHO	27.9	11.4	7.1	7.9	6.1
FALAT	4.1	3.8	0.0	3.3	2.4
RCTA	34.8	18.5	21.2	18.6	23.6

Appendix

Table 18: Step Acc (%) across the five shared trajectory-length groups in Figure 5 . The groups contain 290, 211, 212, 215, and 212 trajectories, respectively. Bold marks the highest observed accuracy in each group.

Figure 10: Within-source Step Acc with DS-V4-Flash by (a) root-to-end distance and (b) log text volume. All six methods share the trajectories in each group. Labels give actual ranges and sample sizes; unequal-width groups are equally spaced, and connecting lines are visual guides. Terminal-Bench 2 remains a single group under the minimum-size rule.

Root-to-end distance (steps)
Source	Group	Actual range	$n$	All-at- once	Step-by- step	Binary search	ECHO	FALAT	RCTA
SWE	1	13–137	43	0.0 / 7.0	0.0 / 7.0	0.0 / 4.7	9.3 / 18.6	2.3 / 2.3	53.5 / 62.8
	2	138–234	42	0.0 / 2.4	2.4 / 11.9	0.0 / 2.4	7.1 / 19.0	2.4 / 9.5	31.0 / 38.1
	3	235–605	43	4.7 / 4.7	0.0 / 11.6	2.3 / 4.7	7.0 / 16.3	2.3 / 20.9	30.2 / 39.5
TB2	1	26–491	42	7.1 / 21.4	4.8 / 16.7	4.8 / 23.8	21.4 / 38.1	2.4 / 16.7	26.2 / 45.2
TP	1	0–16	227	4.8 / 10.1	4.0 / 11.9	0.0 / 2.2	7.9 / 13.2	1.8 / 9.3	16.3 / 31.7

Appendix

Table 19: Within-source localization with DS-V4-Flash. Each method cell reports Step Acc / Step

\pm 5

(%). All methods share the trajectories in each row. SWE, TB2, TP, Vita, and WA denote SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench, and WebArena Verified, respectively.

Stage	Input	Required output	Validation
Local	Segment, task, and overlap	Purpose and candidate steps with evidence IDs	All IDs occur in the segment
Outline	Segment summaries and verifier-signal summary	Contiguous subgoal phases with step IDs	All IDs occur in the trajectory
Attribution	Outline, summaries, candidate text, and handoffs	Independent role and root fields, evidence IDs, and any cited handoff	Role and IDs occur in the trajectory; cited text is grounded

Appendix

Table 20: Prompt requirements and programmatic membership checks for the three RCTA prompt templates. The local-summary template is applied once per segment; the final role and root fields are validated independently.

Figure 11: Construction and annotation workflow for LongRCA Bench. Source runs are filtered and normalized into step-indexed histories before annotation, label resolution, and final quality control.

Full-scale: 294 trajectories
Role relation	Pairs	Role	Step $\pm 5$	Median [Q1, Q3]	0	1–5	6–20	$>20$
All pairs	310	66.1	50.3	5 [0, 46]	40.3	10.0	11.6	38.1
Same role	205	100.0	67.3	0 [0, 15]	59.0	8.3	8.3	24.4
Different role	105	0.0	17.1	42 [13, 78]	3.8	13.3	18.1	64.8
Calibration: 100 trajectories
All pairs	296	53.7	37.8	17.5 [1, 54.25]	24.7	13.2	15.5	46.6

Appendix

Table 21: Pre-resolution disagreement among completed, structurally valid annotations. Each unordered pair of different annotators has equal weight. Role agreement uses the evaluation normalization. Step-gap percentages sum to 100 within each row; quartiles use linear interpolation. Calibration and full-scale annotations are summarized separately.

Method / comparison	Step Acc	Step $\pm 5$
All-at-once	7.8 $\to$ 8.3 [7.1, 9.5]	21.1 $\to$ 22.1 [19.7, 24.5]
Step-by-step	6.1 $\to$ 6.5 [5.4, 7.5]	15.6 $\to$ 17.3 [15.3, 19.4]
Binary search	3.7 $\to$ 3.9 [3.1, 4.8]	15.0 $\to$ 15.9 [13.9, 18.0]
ECHO	17.0 $\to$ 17.6 [15.6, 19.4]	27.9 $\to$ 31.1 [28.6, 33.7]
FALAT	1.0 $\to$ 2.2 [1.4, 3.1]	13.6 $\to$ 14.4 [12.2, 16.7]
RCTA	21.8 $\to$ 23.5 [21.1, 25.9]	35.4 $\to$ 36.2 [33.7, 38.8]

Appendix

Table 22: Root-step sensitivity to observed human labels on 294 full-scale trajectories with DS-V4-Flash. Accuracy cells show final-label score

\to

resampled mean [2.5th, 97.5th percentiles] (%). Gap rows use RCTA minus the strongest baseline in each draw (pp). Rank-1 rates require a strict lead; ties do not count as wins. Intervals describe label sensitivity on fixed samples, not population confidence intervals.

Method	Agent Acc $\uparrow$	Step Acc $\uparrow$	Step $\pm 5$ $\uparrow$	Root MAE $\downarrow$
All-at-once	26.2%	7.6%	19.9%	55.9
Step-by-step	22.2%	5.3%	16.9%	52.3
Binary search	23.0%	3.4%	13.3%	61.7
ECHO	27.5%	13.2%	24.7%	50.4
FALAT	19.0%	2.8%	12.5%	66.6
RCTA (ours)	51.1%	24.1%	37.4%	38.6

Appendix

Table 23: Full-benchmark results on all 1,140 trajectories. All methods use DS-V4-Flash. Agent Acc measures agreement with the reference responsible role. Root MAE is the source-weighted valid-output mean absolute step error. Higher values are better for the three accuracy metrics, whereas lower values are better for MAE.

Source	$K$	Reference roles (counts)	Majority	RCTA
SWE	5	DiagnostAgent (79), ActionAgent (41), JudgeAgent (4), human (3), Computer_terminal (1)	61.7	63.3
TB2	5	DiagnostAgent (20), ActionAgent (18), JudgeAgent (2), Computer_terminal (1), human (1)	47.6	57.1
TP	6 [3–7]	stay_agent (174), manager_agent (162), Task_Planner (103), transport_agent (84), verifier_agent (67), food_attr_agent (56), Computer_terminal (39)	25.4	41.2
Vita	2	Action_Expert (89), Computer_terminal (19)	82.4	82.4
WA	4	Action_Expert (97), Task_Planner (43), Verification_Expert (29), Computer_terminal (8)	54.8	60.5
Overall	–	–	40.3	51.1

Appendix

Table 24: Reference-role distributions and majority-role baselines on Full (

N=1{,}140

). Role counts are listed in descending order.

K

is the number of distinct normalized roles in each history, reported as median [range] when variable. Majority and RCTA columns report Agent Acc (%); Overall weights trajectories equally. Source abbreviations follow Table 2 .

Setting	Agent Acc $\uparrow$	Step Acc $\uparrow$	Step $\pm 5\,\uparrow$	MAE $\downarrow$	Cov. $\uparrow$	$\Delta$ [95% CI]	$p_{\mathrm{Holm}}$
RCTA (all components)	56.0	28.0	41.0	28.3	97.0	–	–
w/o candidate-information package (A1)	53.5	22.0	39.0	30.5	91.5	$-6.0$ [ $-11.0$ , $-1.0$ ]	.17
w/o handoff evidence (A2)	53.5	24.5	38.0	29.7	97.0	$-3.5$ [ $-9.0$ , $+2.0$ ]	.75
w/o backward attribution (A3)	51.0	25.0	41.0	31.4	97.5	$-3.0$ [ $-8.5$ , $+2.5$ ]	.75
w/o validation (A4)	54.0	26.5	39.5	29.2	89.0	$-1.5$ [ $-3.5$ , $0.0$ ]	.75

Appendix

Table 25: Component ablations on the fixed 200-trajectory sample. Rates are source-macro percentages. MAE uses non-abstaining outputs with parseable in-range steps (

n{=}196/188/196/197/192

); Cov. uses all trajectories as its denominator and requires non-abstaining outputs that pass the condition’s validation checks (Appendix L.1 ).

\Delta

is ablation Step Acc minus that of RCTA (all components). Intervals are pointwise;

p_{\mathrm{Holm}}

controls the four-test Step Acc family.

Root Cause Analysis

Root cause analysis identifies the initiating faults behind observed failures by tracing symptoms through causal and system dependencies.

See all 80 papers

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
Sep 11, 2026Harsh Raj, David Lee, Anas Mahmoud +7Agent Failure Analysis Agentic Search
Scale AI · Scale AI Research
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Aug 6, 2026Yunjia Qi, Zehua Yin, Xintong Shi +10Agent Failure Analysis Long-Horizon Agent Evaluation
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University
TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution
Aug 27, 2026Tommaso Bendinelli, Artur Dox, Christian HolzLLM Agent Evaluation Root Cause Analysis
Department of Computer Science ETH Zürich, Switzerland · CSEM SA, Switzerland · Independent Researcher

Sep 11, 2026J/K move · Enter open · S save

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang+6

Scale AI · Scale AI Research

Agent Failure Analysis Agentic Search Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from

0.471

0.608

. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.

Abstract

Root Cause Analysis

Explore similar work

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution