Organizations: Computer Network Information Center, Chinese Academy of Sciences · Chongqing University · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Singapore Management University · Tongyi Lab, Alibaba Group · Tsinghua University
In long agent executions, an early error can persist through later actions and checks, while evidence needed to trace its origin is dispersed across the history. Short histories offer limited tests of recovering error origins across substantial subsequent execution. We introduce LongRCA Bench: 1,140 complete failed trajectories from five sources, all human-annotated for responsible roles and earliest decisive root-cause steps. Reference roots precede completion by a median of 48 recorded steps; 28.4% of trajectories contain over 100 subsequent steps. With DeepSeek-V4-Flash on the full benchmark, the strongest of five evaluated baselines achieves 13.2% exact root-step accuracy. We propose Root-Cause Trajectory Attribution (RCTA), a training-free method that organizes original candidate records and explicit handoff instructions for attribution. Segment summaries and a trajectory outline guide candidate retrieval; available handoff records supply upstream instruction context for the final instruction-execution comparison. With the same backbone and scoring protocol, RCTA reaches 24.1% exact root-step accuracy and 51.1% responsible-role accuracy. LongRCA-Mini provides 200 fixed trajectories for lower-cost comparative screening. Even with RCTA, fewer than one quarter of reference roots are recovered exactly.
Figures & tables
Figure 1: All six evaluated methods score lower on LongRCA across Agent Acc, Step Acc, and Step ±5 . Accuracy (%) uses DS-V4-Flash on the complete Who&When ( n=184 ) and LongRCA ( n=1,140 ) sets, with each trajectory equally weighted within its dataset.
Benchmark
Error Source
Diagnostic Targets
Trajectories Total (Reviewed)
Avg. length (steps)
RootSE ( Wang et al., 2026a )
Natural
Step a
102 (102)
51.7
HORIZON ( Wang et al., 2026b )
Natural
Type
1995 (40)
–
TraceElephant ( Chen et al., 2026 )
Natural
Component, Step b
220 (220)
–
Failure as a Process ( Zhao et al., 2026 )
Natural
Steps c , Type
1184 (1184)
42.0
Who&When ( Zhang et al., 2025 )
Natural
Agent, Step a
184 (184)
22.2
Who&When Pro ( Liu et al., 2026 )
Synthetic
Agent, Step a , Type
12,326 (100)
7.5
Table 1: Comparison of resources for agent failure analysis. LongRCA supports responsible-role and earliest-root-step attribution over long execution trajectories collected without error injection and fully reviewed by humans.
Figure 2: A price-unit misinterpretation persists through planning and verification despite a local budget correction. The human reference identifies stay_agent and step 73 within a single output containing candidate records at steps 73–77. The displayed path follows the ultimately selected property from step 74. Green arrows indicate retrospective diagnosis using the final itinerary and TravelPlanner’s nightly pricing rule.
Figure 3: Long-horizon characteristics ( n=1,140 ). (a) Medians (circles), means (diamonds), minimum–P90 ranges, and maxima of trajectory length, reference root step, and root-to-end distance. (b) Fractions with more than 50, 100, or 200 recorded steps after the reference root.
Figure 4: RCTA has two stages: Stage I constructs the candidate-information package (A1); Stage II retrieves available handoff evidence (A2) and performs backward instruction–execution reasoning (A3). Output and evidence validation (A4) checks the resulting attribution output and its evidence provenance.
Method
Overall
Per-source Step Acc
Agent Acc
Step Acc
Step ±5
SWE
TB2
TP
Vita
WA
All-at-once
26.2
7.6
19.9
1.6
7.1
4.5
19.4
16.9
Step-by-step
22.2
5.3
16.9
0.8
4.8
3.5
12.0
11.3
Binary search
23.0
3.4
13.3
0.8
4.8
2.0
12.0
5.1
ECHO
27.5
13.2
24.7
7.8
21.4
8.9
22.2
26.0
FALAT
19.0
2.8
12.5
2.3
2.4
2.9
3.7
2.3
Table 2: Full-benchmark accuracies (%) using DS-V4-Flash. Overall weights trajectories equally; per-source columns report Step Acc. Bold marks the highest observed value in each column.
Figure 5: Step Acc on all 1,140 trajectories with DS-V4-Flash. Methods share length-only groups without splitting equal lengths. Panel (a) shows all groups; panel (b) magnifies the 0–12% region of groups 2–5. Unequal-width groups are equally spaced; connecting lines are visual guides.
Method
V4-Flash
V4-Pro
GPT-5.6 Sol
Kimi K3
Qwen3.5 35B-A3B
Agent
Step
Agent
Step
Agent
Step
Agent
Step
Agent
Step
All-at-once
42.5
7.5
50.0
11.5
53.0
17.0
54.5
11.0
51.0
3.5
Step-by-step
38.5
6.5
47.5
10.5
49.0
13.0
44.0
9.5
46.5
4.5
Binary search
40.5
7.5
45.5
10.0
51.5
15.5
42.0
12.0
40.0
5.0
ECHO
45.0
13.0
53.5
18.0
55.5
18.5
55.0
20.5
48.5
10.0
FALAT
35.5
2.5
48.0
14.5
52.0
14.0
52.0
18.0
41.0
11.0
Table 3: Agent and Step Acc (%, Macro-5) on the fixed 200-trajectory Mini (40 per source). V4 denotes DeepSeek-V4. Bold marks the highest point estimate per backbone and metric; complete metrics and paired analyses are in Appendix B .
Setting
Agent Acc ↑
Step Acc ↑
Step ±5↑
Cov. ↑
pHolm
RCTA (all components)
56.0
28.0
41.0
97.0
–
w/o candidate-information package (A1)
53.5
22.0
39.0
91.5
.17
w/o handoff evidence (A2)
53.5
24.5
38.0
97.0
.75
w/o backward attribution (A3)
51.0
25.0
41.0
97.5
.75
w/o validation (A4)
54.0
26.5
39.5
89.0
.75
Table 4: Paired ablations on the fixed Mini sample (Macro-5 percentages). Cov. measures valid-output coverage; pHolm adjusts the four Step Acc tests.
Figure 6: Component diagnostics. (a) Reference-root hit rates and final Step Acc (all n=200 ). (b) Step Acc on handoff-containing trajectories by reference root: handoff ( n=41 ) or non-handoff ( n=39 ). Full and A3 share the original excerpts. Error bars: 95% Wilson intervals; subgroup contrasts are exploratory.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Mini calibration with cached predictions and 10,000 source-stratified draws per size: maximum per-pair median and 95th-percentile absolute Macro-5 errors across 18 method–metric pairs.
Size
Quotas
Max. median
Max. p95
Pass
75
15/15/15/15/15
3.865
10.532
No
100
20/20/20/20/20
3.135
8.865
No
125
25/25/25/25/25
2.535
7.335
No
150
30/30/30/30/30
2.327
6.532
No
175
35/35/35/35/35
2.151
5.849
No
200
40/40/40/40/40
1.890
5.365
Yes
Appendix
Table 5: Six-method sample-size calibration with equal-quota simple-random sampling. Quotas follow the source order SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench, and WebArena Verified. For each of 18 method–metric pairs, absolute score errors are summarized over 10,000 draws. “Max. median” and “Max. p95” report the maximum per-pair median and 95th-percentile errors (pp). “Pass” requires only max. median ≤2 pp and max. p95 ≤6 pp.
Method
Agent Acc
Step Acc
Step ±5
All-at-once
48.2→48.5(0.3)
9.9→8.5(1.4)
23.7→21.5(2.2)
Step-by-step
41.8→41.5(0.3)
6.5→6.5(0.0)
19.8→17.5(2.3)
Binary search
40.8→39.5(1.3)
4.9→7.5(2.6)
18.1→20.0(1.9)
ECHO
49.4→46.5(2.9)
17.3→13.5(3.8)
31.2→29.0(2.2)
FALAT
33.1→34.5(1.4)
2.7→3.0(0.3)
15.5→18.0(2.5)
RCTA
60.9→61.0(0.1)
28.3→27.0(1.3)
44.1→41.0(3.1)
Appendix
Table 6: Fidelity of the fixed LongRCA-Mini manifest. Each cell reports Full → Mini ( ∣Δ∣ ) : Macro-5 accuracy (%) and absolute difference (pp). Across the 18 method–metric comparisons, the mean absolute deviation is 1.665 pp and the maximum deviation is 3.771 pp. All results use DS-V4-Flash.
Structural quantity
Full
Mini
Comparison
Trajectories
1,140
200
17.54% retained
Recorded steps
178,137
32,317
18.14% retained
History characters
207,731,218
41,784,984
20.11% retained
Macro-5 mean trajectory length
167.7
161.6
−6.1 steps
Macro-5 mean root-to-end distance
104.9
104.4
−0.5 steps
Macro-5 mean normalized root position
0.420
0.399
−0.021
Appendix
Table 7: Structural comparison of LongRCA Bench and the fixed LongRCA-Mini manifest. The KS statistic is the maximum across within-source comparisons of trajectory length, root-to-end distance, and normalized root position. History characters are used as a model-independent input-cost proxy.
Source
Steps
Input kchars
Root-to-end
Root position
KS
SWE
287.0/263.0
450.2/432.9
215.4/218.5
0.250/0.181
0.177
TB2
272.1/265.6
297.2/294.9
186.9/179.5
0.263/0.268
0.040
TP
165.9/167.8
163.3/154.1
65.9/67.8
0.597/0.569
0.134
Vita
54.6/50.8
23.2/26.9
29.7/26.0
0.445/0.473
0.134
WA
58.8/60.8
131.2/135.8
26.8/30.1
0.543/0.502
0.147
Appendix
Table 8: Within-source structural audit of the fixed manifest. Entries are Full/Mini means; input length is measured in thousands of history characters, and normalized root position is r∗/(T−1) . KS is the largest empirical CDF distance across the four reported characteristics. SWE, TB2, TP, Vita, and WA denote the five sources in the order used in Table 5 .
Method
Metric
SWE
TB2
TP
Vita
WA
All-at-once
Agent Acc
47.7/42.5
61.9/62.5
7.2/7.5
82.4/85.0
41.8/45.0
Step Acc
1.6/2.5
7.1/5.0
4.5/5.0
19.4/20.0
16.9/10.0
Step ±5
4.7/7.5
21.4/20.0
15.2/10.0
41.7/42.5
35.6/27.5
Step-by-step
Agent Acc
38.3/30.0
40.5/40.0
2.8/5.0
83.3/85.0
44.1/47.5
Step Acc
0.8/2.5
4.8/5.0
3.5/0.0
12.0/12.5
11.3/12.5
Step ±5
10.2/7.5
16.7/15.0
13.0/10.0
30.6/27.5
28.8/27.5
Appendix
Table 9: Per-source Full/Mini accuracy (%) for the fixed manifest. Each cell reports the full source score followed by the Mini source score. Source abbreviations follow Table 8 . These values are an audit of source-level variation, not evidence that 40 trajectories support definitive per-source comparisons.
Method
Agent Acc
Step Acc
Step ±5
All-at-once
26.2→26.6(0.4)
7.6→6.9(0.7)
19.9→15.9(4.0)
Step-by-step
22.2→23.3(1.1)
5.3→3.6(1.7)
16.9→14.3(2.7)
Binary search
23.0→22.4(0.6)
3.4→7.7(4.3)
13.3→17.2(3.9)
ECHO
27.5→24.7(2.9)
13.2→9.6(3.5)
24.7→22.4(2.3)
FALAT
19.0→21.2(2.2)
2.8→2.8(0.0)
12.5→14.0(1.5)
RCTA
51.1→54.6(3.5)
24.1→24.7(0.5)
37.4→34.2(3.2)
Appendix
Table 10: Original-mixture fidelity audit. Cells report Full → post-stratified Mini ( ∣Δ∣ ), in accuracy percentages and pp. Both sides use the original source weights, not Macro-5.
Method
Agent Acc ↑
Step Acc ↑
Step ±5↑
Joint Exact ↑
Joint ±5↑
MAE ↓
DS-V4-Flash
All-at-once
42.5
7.5
20.5
7.5
13.0
44.915
Step-by-step
38.5
6.5
18.0
6.5
10.0
41.340
Binary search
40.5
7.5
20.5
7.5
12.0
43.465
ECHO
45.0
13.0
28.0
13.0
18.0
40.930
FALAT
35.5
2.5
17.5
2.5
10.5
53.685
Appendix
Table 11: Complete six-metric record for the fixed 200-trajectory cross-backbone experiment. Accuracies are Macro-5 percentages; Step MAE is the reported valid-output mean absolute root-step error in steps. Bold marks the best value within each backbone and metric (including ties). Shading identifies RCTA. Evaluation settings and prediction coverage are detailed in the accompanying text.
Backbone
Δ
95% CI
W/L
Holm-20 p
Holm-40 p
DS-V4-Flash
+14.0
[7.5,20.5]
39/11
0.00072
0.00217
DS-V4-Pro
+10.0
[3.5,17.0]
36/16
0.00779
0.06007
GPT-5.6 Sol
+11.5
[4.5,19.0]
40/17
0.00640
0.04162
Kimi K3
+11.0
[5.0,17.0]
31/9
0.00322
0.01155
Appendix
Table 12: Paired RCTA–ECHO Step Acc differences on LongRCA-Mini ( n=200 ). Δ and pointwise 95% intervals are in percentage points; W/L counts RCTA-only/ECHO-only correct predictions. Holm-20 adjusts the Step Acc family; Holm-40 jointly adjusts Agent Acc and Step Acc.
Excluded benchmark
Mean gain
95% CI
SWE-bench Pro
+8.3
[3.6,13.1]
Terminal-Bench 2
+13.0
[8.6,17.5]
TravelPlanner
+11.4
[6.4,16.4]
VitaBench
+13.0
[7.8,18.1]
WebArena Verified
+12.5
[7.3,17.8]
Appendix
Table 13: Leave-one-benchmark-out sensitivity of the four-backbone mean RCTA–ECHO Step Acc gain. Each row retains 160 trajectories from four benchmarks; differences and intervals are in percentage points.
Figure 8: Exact-root correctness on LongRCA-Mini ( n=200 per backbone). Both: RCTA and at least one of five baselines are correct; RCTA only: RCTA is correct and all five baselines are wrong; Baseline only: RCTA is wrong and at least one baseline is correct; Neither: all six methods are wrong. Segment labels are trajectory counts.
Field
Meaning
question_ID
Source-prefixed trajectory identifier
history
Ordered trajectory records containing step , name , role , and content ; includes the task instruction
mistake_agent
Reference responsible role, scored independently of the root-cause step
mistake_step
Reference 0-based root-cause step, scored independently of the responsible role
mistake_reason
Trajectory-grounded annotation rationale, withheld during prediction and not scored
Appendix
Table 14: Released LongRCA Bench data schema. Evaluated methods receive question_ID and history , which includes the task instruction. The three reference annotation fields are withheld during prediction.
Source
N
Task domain
Agent organization
SWE-bench Pro
128
Software repair
Diagnosis–execution–verification team
Terminal Bench 2
42
Terminal tasks
Diagnosis–execution–verification team
TravelPlanner
685
Travel planning
Magentic-One group chat
VitaBench
108
Service-oriented tool use
Action expert–computer terminal workflow
WebArena Verified
177
Web interaction
Planner–web-surfer–critic workflow
Total
1,140
5 task domains
Appendix
Table 15: Task domains and agent organizations of the retained LongRCA Bench trajectories. Counts are reported by source.
Source
N
Workflow
M2.5
K2.5
Q3.5+
SWE-bench Pro
128
Custom DiagnostAgent – ActionAgent – JudgeAgent team
128
–
–
Terminal-Bench 2
42
Custom DiagnostAgent – ActionAgent – JudgeAgent team
42
–
–
TravelPlanner
685
AutoGen MagenticOneGroupChat with specialist roles
265
260
160
VitaBench
108
Action_Expert – Computer_terminal workflow
69
16
23
WebArena Verified
177
AutoGen Planner – WebSurfer – Critic controller
177
–
–
Total
1,140
681
276
183
Appendix
Table 16: Execution provenance and generator counts for LongRCA Bench. The SWE-bench Pro and Terminal-Bench 2 execution records specify their custom role workflow and generator, but no separate third-party scaffold package. Generator columns correspond to MiniMax-M2.5 (M2.5), Kimi-K2.5 (K2.5), and Qwen3.5-Plus (Q3.5+). A dash indicates that the source contains no trajectory from that generator.
Figure 9: Step Acc (%) across five sources using DS-V4-Flash. All six methods use the same complete source subsets ( n below source labels; total 1,140); invalid step outputs count as errors.
Method
SWE-bench Pro
Terminal Bench 2
TravelPlanner
VitaBench
WebArena Verified
All-at-once
1.6
7.1
4.5
19.4
16.9
Step-by-step
0.8
4.8
3.5
12.0
11.3
Binary search
0.8
4.8
2.0
12.0
5.1
ECHO
7.8
21.4
8.9
22.2
26.0
FALAT
2.3
2.4
2.9
3.7
2.3
RCTA
38.3
26.2
18.8
25.0
33.3
Appendix
Table 17: Full-benchmark Step Acc (%) by source, underlying Figure 9 . Source sample sizes are 128, 42, 685, 108, and 177, respectively. Every method uses the complete corresponding source subset; bold marks the highest observed accuracy in each source.
Method
10–63
64–129
130–172
173–220
221–728
All-at-once
21.4
6.2
2.8
0.9
1.9
Step-by-step
13.4
3.3
2.8
3.3
0.5
Binary search
7.2
3.8
0.5
2.8
1.4
ECHO
27.9
11.4
7.1
7.9
6.1
FALAT
4.1
3.8
0.0
3.3
2.4
RCTA
34.8
18.5
21.2
18.6
23.6
Appendix
Table 18: Step Acc (%) across the five shared trajectory-length groups in Figure 5 . The groups contain 290, 211, 212, 215, and 212 trajectories, respectively. Bold marks the highest observed accuracy in each group.
Figure 10: Within-source Step Acc with DS-V4-Flash by (a) root-to-end distance and (b) log text volume. All six methods share the trajectories in each group. Labels give actual ranges and sample sizes; unequal-width groups are equally spaced, and connecting lines are visual guides. Terminal-Bench 2 remains a single group under the minimum-size rule.
Source
Group
Actual range
n
All-at- once
Step-by- step
Binary search
ECHO
FALAT
RCTA
Root-to-end distance (steps)
SWE
1
13–137
43
0.0 / 7.0
0.0 / 7.0
0.0 / 4.7
9.3 / 18.6
2.3 / 2.3
53.5 / 62.8
2
138–234
42
0.0 / 2.4
2.4 / 11.9
0.0 / 2.4
7.1 / 19.0
2.4 / 9.5
31.0 / 38.1
3
235–605
43
4.7 / 4.7
0.0 / 11.6
2.3 / 4.7
7.0 / 16.3
2.3 / 20.9
30.2 / 39.5
TB2
1
26–491
42
7.1 / 21.4
4.8 / 16.7
4.8 / 23.8
21.4 / 38.1
2.4 / 16.7
26.2 / 45.2
TP
1
0–16
227
4.8 / 10.1
4.0 / 11.9
0.0 / 2.2
7.9 / 13.2
1.8 / 9.3
16.3 / 31.7
Appendix
Table 19: Within-source localization with DS-V4-Flash. Each method cell reports Step Acc / Step ±5 (%). All methods share the trajectories in each row. SWE, TB2, TP, Vita, and WA denote SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench, and WebArena Verified, respectively.
Stage
Input
Required output
Validation
Local
Segment, task, and overlap
Purpose and candidate steps with evidence IDs
All IDs occur in the segment
Outline
Segment summaries and verifier-signal summary
Contiguous subgoal phases with step IDs
All IDs occur in the trajectory
Attribution
Outline, summaries, candidate text, and handoffs
Independent role and root fields, evidence IDs, and any cited handoff
Role and IDs occur in the trajectory; cited text is grounded
Appendix
Table 20: Prompt requirements and programmatic membership checks for the three RCTA prompt templates. The local-summary template is applied once per segment; the final role and root fields are validated independently.
Figure 11: Construction and annotation workflow for LongRCA Bench. Source runs are filtered and normalized into step-indexed histories before annotation, label resolution, and final quality control.
Role relation
Pairs
Role
Step ±5
Median [Q1, Q3]
0
1–5
6–20
>20
Full-scale: 294 trajectories
All pairs
310
66.1
50.3
5 [0, 46]
40.3
10.0
11.6
38.1
Same role
205
100.0
67.3
0 [0, 15]
59.0
8.3
8.3
24.4
Different role
105
0.0
17.1
42 [13, 78]
3.8
13.3
18.1
64.8
Calibration: 100 trajectories
All pairs
296
53.7
37.8
17.5 [1, 54.25]
24.7
13.2
15.5
46.6
Appendix
Table 21: Pre-resolution disagreement among completed, structurally valid annotations. Each unordered pair of different annotators has equal weight. Role agreement uses the evaluation normalization. Step-gap percentages sum to 100 within each row; quartiles use linear interpolation. Calibration and full-scale annotations are summarized separately.
Method / comparison
Step Acc
Step ±5
All-at-once
7.8 → 8.3 [7.1, 9.5]
21.1 → 22.1 [19.7, 24.5]
Step-by-step
6.1 → 6.5 [5.4, 7.5]
15.6 → 17.3 [15.3, 19.4]
Binary search
3.7 → 3.9 [3.1, 4.8]
15.0 → 15.9 [13.9, 18.0]
ECHO
17.0 → 17.6 [15.6, 19.4]
27.9 → 31.1 [28.6, 33.7]
FALAT
1.0 → 2.2 [1.4, 3.1]
13.6 → 14.4 [12.2, 16.7]
RCTA
21.8 → 23.5 [21.1, 25.9]
35.4 → 36.2 [33.7, 38.8]
Appendix
Table 22: Root-step sensitivity to observed human labels on 294 full-scale trajectories with DS-V4-Flash. Accuracy cells show final-label score → resampled mean [2.5th, 97.5th percentiles] (%). Gap rows use RCTA minus the strongest baseline in each draw (pp). Rank-1 rates require a strict lead; ties do not count as wins. Intervals describe label sensitivity on fixed samples, not population confidence intervals.
Method
Agent Acc ↑
Step Acc ↑
Step ±5↑
Root MAE ↓
All-at-once
26.2%
7.6%
19.9%
55.9
Step-by-step
22.2%
5.3%
16.9%
52.3
Binary search
23.0%
3.4%
13.3%
61.7
ECHO
27.5%
13.2%
24.7%
50.4
FALAT
19.0%
2.8%
12.5%
66.6
RCTA (ours)
51.1%
24.1%
37.4%
38.6
Appendix
Table 23: Full-benchmark results on all 1,140 trajectories. All methods use DS-V4-Flash. Agent Acc measures agreement with the reference responsible role. Root MAE is the source-weighted valid-output mean absolute step error. Higher values are better for the three accuracy metrics, whereas lower values are better for MAE.
Source
K
Reference roles (counts)
Majority
RCTA
SWE
5
DiagnostAgent (79), ActionAgent (41), JudgeAgent (4), human (3), Computer_terminal (1)
61.7
63.3
TB2
5
DiagnostAgent (20), ActionAgent (18), JudgeAgent (2), Computer_terminal (1), human (1)
Table 24: Reference-role distributions and majority-role baselines on Full ( N=1,140 ). Role counts are listed in descending order. K is the number of distinct normalized roles in each history, reported as median [range] when variable. Majority and RCTA columns report Agent Acc (%); Overall weights trajectories equally. Source abbreviations follow Table 2 .
Setting
Agent Acc ↑
Step Acc ↑
Step ±5↑
MAE ↓
Cov. ↑
Δ [95% CI]
pHolm
RCTA (all components)
56.0
28.0
41.0
28.3
97.0
–
–
w/o candidate-information package (A1)
53.5
22.0
39.0
30.5
91.5
−6.0 [ −11.0 , −1.0 ]
.17
w/o handoff evidence (A2)
53.5
24.5
38.0
29.7
97.0
−3.5 [ −9.0 , +2.0 ]
.75
w/o backward attribution (A3)
51.0
25.0
41.0
31.4
97.5
−3.0 [ −8.5 , +2.5 ]
.75
w/o validation (A4)
54.0
26.5
39.5
29.2
89.0
−1.5 [ −3.5 , 0.0 ]
.75
Appendix
Table 25: Component ablations on the fixed 200-trajectory sample. Rates are source-macro percentages. MAE uses non-abstaining outputs with parseable in-range steps ( n=196/188/196/197/192 ); Cov. uses all trajectories as its denominator and requires non-abstaining outputs that pass the condition’s validation checks (Appendix L.1 ). Δ is ablation Step Acc minus that of RCTA (all components). Intervals are pointwise; pHolm controls the four-test Step Acc family.
The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from 0.471 to 0.608. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.
Yunjia Qi, Zehua Yin, Xintong Shi +10
Department of Computer Science and Technology, BNRist, Tsinghua University · Tencent Hunyuan · Shenzhen International Graduate School, Tsinghua University
LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however, their performance on these tasks has not been systematically evaluated under controlled conditions. We introduce TraceBench, a simulation-based framework for generating controlled root-cause attribution tasks. In each generated task, an agent receives time-series observations produced by simulating a physical dynamical system and must determine whether a system parameter was altered during the simulation and, if so, which one. Using TraceBench, we generate tasks from three interpretable mechanical systems and systematically evaluate four LLM agents across controlled experimental conditions, yielding new insights into how these agents analyze time-series observations from dynamical systems. Our results show that agents benefit substantially from domain context and explore data primarily through numerical console output rather than visualizations. We also find that agents generally perform worse when required to produce a Python script that maps each time-series sample to a predicted root-cause label than when they submit predictions directly. We release our datasets, agent trajectories, experimental results, and a leaderboard on our website, tracebench.github.io.
Tommaso Bendinelli, Artur Dox, Christian Holz
Department of Computer Science ETH Zürich, Switzerland · CSEM SA, Switzerland · Independent Researcher