Structural Baselines Matter for Agent Failure Detection and Attribution
Authors: Ramesh Arvind Naagarajan, Stefan Streif
Organizations: Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany
An LLM agent using tools can fail silently. It stops before the task is done, raises no error, and answers fluently. Two kinds of monitor address this, one deciding whether a run failed and the other naming the step where failure became unavoidable. Much of the reported performance on both tasks comes from regularities that need no model. Failing runs are longer, and the step labelled decisive is usually the last. A rule that always names the last step is correct 67% to 96% of the time here, without reading the run at all. That ceiling reflects how the labels were written, not how agents break. On failures whose decisive step replay can confirm, it falls to near chance. Reporting gain over that rule therefore changes what a score means. Under that correction, methods tested here gain nothing on give-up failures, while several published systems keep a clear advantage on other splits. For detection, the agent's internal states beat transcript length and text similarity on the agent benchmark, and hold up where the text baseline does not. Precision stays low when failures are rare, so such a monitor should rank runs for review, not replace it.
Figures & tables
Figure 1: The label-free pipeline, with detection AUROC on the regenerated corpus, rounded to two digits from Table 1 . Calibration uses trusted successful runs only, leave-one-out, so no failure label enters the monitor. ∗ tool grounding is measured on the greedy corpus.
Label-free signal
pooled
95% CI
resid.
airline
retail
structural baselines (no calibration)
Transcript length
0.640
[.564,.723]
–
0.740
0.456
Step count
0.716
[.652,.777]
–
0.769
0.585
Give-up keyword ∗
0.594
[.564,.627]
–
–
–
prior trajectory anomaly detectors
Cycle detection, hybrid
0.670
[.592,.750]
–
–
–
Table 1: Detection AUROC, regenerated τ -bench corpus ( 660 trajectories, 64 trusted successes). resid. residualises on log transcript length. ∗ scored on the greedy corpus ( 165 runs), the only vintage with a tool oracle. Every other row is the regenerated corpus. † scored by the probability the judge assigns to its own failure verdict, since its emitted verdict is near-constant (Appendix C.3 ).
Benchmark / split
n
pos.
best pos.
learned
SOTA
lift
lift ho
τ 8B airline
34
.98
last 0.88
0.53
–
−0.35
−0.36
τ 8B retail
110
1.00
last 0.96
0.12
–
−0.85
−0.85
τ 70B airline
39
.84
last 0.67
0.31
–
−0.36
−0.36
τ Qwen airline
47
.66
last 0.15
0.32
–
+0.17
+0.22
τ Gemma airline
27
.83
last 0.41
0.44
–
+0.04‡
+0.04‡
W&W hand-crafted
58
.46
last 0.12
0.17
0.28
+0.05‡
+0.07‡
Table 2: Positional-lift audit of step attribution. pos. mean gold relative position, lift learned minus best positional baseline, lift ho with the rule chosen on a held-out half, SOTA best published accuracy ( Zhu et al., 2026a ) . ‡ the interval on skill touches zero (permutation p≥0.25 ), so these cells are null results rather than small gains. The normalised skill score, with intervals, is in Appendix E.3 .
Benchmark / domain
Agent
n
% recov.
ceiling
τ -bench airline
Llama-3.1-8B
34
35.3
0.88
τ -bench airline
Llama-3.1-70B
39
25.6
0.67
τ -bench airline
Qwen2.5-7B
47
25.5
0.11
τ -bench airline
Gemma-2-27B
27
14.8
0.40
τ -bench airline
Mistral-24B
30
0.0
–
τ -bench airline
Llama-3.1-8B
154
36.4
–
Table 3: How often a failure has a decisive step that replay can verify. % recov. share of failures where some prefix makes the goal unreachable, ceiling positional ceiling of that split. Methods and intervals in Appendices D.7 and D.8 .
Corpus
trusted
dict.
give-up
best latent
full
GSM8K Qwen-7B
1866
7168
0
0.61
0.73
GSM8K Mistral-24B
881
10240
0
0.60
0.64
τ -bench 8B
261
8192
1
0.63
0.80
FinQA Llama-8B
165
8192
3
0.65
0.75
Table 4: Sparse dictionaries fitted on trusted steps only. give-up latents active on over 10% of failing and under 1% of trusted steps, best latent best single-latent AUROC, full full-space distance.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Agent
Domain
runs
succ.
fail
Llama-3.1-8B
airline
50
16
34
Llama-3.1-8B
retail
115
5
110
Llama-3.1-70B
airline
50
11
39
Appendix
Table 5: Greedy corpus composition. Tool-error rate is 0.44 on airline and 0.72 on retail, over 654 executed tool returns.
Figure 2: Two silent-failure modes from the datasets, with real trajectory text.
Quantity
Corpus
Where
Detection AUROC, all signals
regenerated
Table 1
Length and step-count baselines
regenerated
Table 1
Tool-grounding signal
greedy
Table 1
Give-up keyword row
greedy
Table 1
Domain transfer
regenerated
§ 5.6
Recoverability fraction
regenerated
App. D.7
Appendix
Table 6: Which data vintage underlies which reported quantity.
Task (failure)
Model
n (succ.)
length
SBERT
internal
p
τ -bench (omission)
Llama-8B
660 ( 64 )
0.64
0.55
0.79
<0.001
GSM8K (wrong ans.)
Llama-8B
100 ( 29 )
0.76
0.62
0.65
0.720
GSM8K (wrong ans.)
Qwen-7B
1319 ( 852 )
0.58
0.58
0.73
<.001
GSM8K (wrong ans.)
Mistral-24B
300 ( 233 )
0.68
0.67
0.63
0.314
GSM8K (wrong ans.)
Qwen-32B
300 ( 256 )
0.67
0.63
0.81
0.007
FinQA (wrong ans.)
Llama-8B
191 ( 58 )
0.73
0.70
0.73
0.508
Appendix
Table 7: Detection across tasks and model families, leave-one-out calibration. length is the transcript-length baseline. p tests internal against SBERT.
Open-pipeline corpus
n
succ.
TF-IDF
SBERT
internal
Mistral-24B airline
50
20
0.40
0.71 [.55,.85]
0.64 [.48,.80]
Llama-8B airline + retail
150
13
0.52
0.68 [.51,.82]
0.83 [.65,.98]
Appendix
Table 8: Detection AUROC on τ -bench datasets from a fully open pipeline (Qwen2.5-32B user simulator), bootstrap 95% CIs in brackets.
Figure 3: Where the gold decisive step falls, as a fraction of the way through a run. On τ -bench the mass sits at the end, which is what makes an always-last rule strong there. Single-step runs are excluded, since for them first and last coincide (Appendix B.1 ).
Figure 4: Decisive-step accuracy on the give-up gold ( 95% bootstrap CIs). The always-last baseline (grey) dominates the learned localizers and the judge.
Figure 5: Detection AUROC vs residual-stream layer, with text baselines marked.
Figure 6: Internal anomaly score, failing vs successful trajectories (8B).
Corpus
n
succ.
internal
SBERT
p
airline only, open simulator
greedy
50
9
0.799
0.745
0.455
temp 0.3
200
37
0.824
0.753
0.073
temp 0.7
200
46
0.783
0.694
0.044
airline + retail
greedy, GPT-4o sim
165
21
0.781
0.805
0.589
Appendix
Table 9: Isolating decoding from the user simulator and from domain composition. The greedy GPT-4o row is scored leave-one-out, giving 0.781/0.805 rather than the in-sample 0.842/0.857 .
Detector
AUROC
95% CI
p vs internal
Cycle, structural
0.570
[.525,.613]
<0.001
Cycle, semantic
0.660
[.588,.733]
<0.001
Cycle, hybrid
0.670
[.592,.750]
<0.001
Graph rarity
0.619
[.565,.670]
<0.001
Internal, RMS dist.
0.787
[.708,.857]
–
Appendix
Table 10: Prior trajectory anomaly detectors on the regenerated corpus ( 660 trajectories, 64 trusted successes), leave-one-out, threshold-free. DeLong tests are paired against the internal signal, whose row repeats the Table 1 interval.
Corpus
n
max
mean
top-3
last
τ 8B airline + retail
165
0.781
0.762
0.765
0.710
τ 70B airline
50
0.834
0.802
0.851
0.816
τ 8B retail
115
0.473
0.440
0.413
0.482
GSM8K Llama-8B
100
0.645
0.659
0.681
0.646
GSM8K Qwen-7B
1319
0.733
0.754
0.752
0.718
GSM8K Mistral-24B
300
0.631
0.657
0.655
0.669
Appendix
Table 11: Detection AUROC by run-level aggregator, leave-one-out throughout. The maximum, which the main results use, is best or tied on 3 of the 13 datasets scored. Ten are shown, and the remaining three follow the same pattern. The 70B row rests on 11 trusted successes and is underpowered (Section 5.6 ).
Figure 7: Detection AUROC versus trusted-reference size for the internal and SBERT signals (mean ± std over 50 resamples). Both degrade smoothly, and a few tens of successful trajectories suffice.
Split
Signal
N=2
3
5
8
16
Airline
Internal
–
0.809
0.824
0.819
0.818
SBERT
–
0.768
0.765
0.766
0.772
Retail
Internal
0.706
0.723
0.715
–
–
SBERT
0.822
0.830
0.860
–
–
Appendix
Table 12: Per-domain trusted-reference size sweep (8B, mean over 50 resamples, each domain scored on itself). Each sweep stops at that domain’s success count.
Contamination
0%
4.5%
8.7%
19.2%
30%
Internal
0.842
0.846
0.844
0.848
0.849
SBERT
0.857
0.852
0.844
0.816
0.753
Appendix
Table 13: Detection AUROC with the trusted calibration set contaminated by unlabeled failing trajectories (pooled 8 B, mean over 50 resamples).
Agent config
Reference
AUROC
Original (greedy)
Original
0.818
Drifted (temp 0.7)
Stale original
0.793
Drifted (temp 0.7)
Recalibrated
0.814
Appendix
Table 14: Staleness under agent drift (8B airline, internal RMS- z ).
Signal
FPR
Precision
Recall
F1
Internal RMS-z
5%
0.986
0.472
0.638
10%
0.976
0.556
0.708
20%
0.960
0.674
0.792
SBERT
5%
0.986
0.493
0.657
10%
0.978
0.625
0.763
20%
0.966
0.785
0.866
Appendix
Table 15: Detection precision, recall, and F1 at three benign-calibrated FPR operating points (pooled Llama-3.1-8B airline + retail). The threshold is set on benign traffic alone.
Detector
AUROC
Label-free internal RMS- z
0.842
Label-free SBERT
0.857
Label-free equal-weight fusion
0.859
Supervised probe (mean-pool act.)
0.793±0.028
Supervised fusion (three signals)
0.862±0.008
Appendix
Table 16: Supervised upper bound versus label-free detection (pooled 8B, supervised rows five-fold CV over five seeds).
Failure regime
n
internal
SBERT
DeLong p
Acted (write call, wrong)
82
0.902
0.938
0.39
Gave up (transfer / no write)
62
0.763
0.750
0.80
Appendix
Table 17: Detection AUROC by failure regime (pooled 8 B corpus, each regime’s failures against all 21 successes). The internal SBERT tie holds within both regimes.
Split
Method
step-acc [95% CI]
agent
Hand-crafted
Label-free
0.172[0.086,0.276]
0.655
GPT-4o judge
0.172
0.500
Always-last
0.121[0.034,0.207]
–
Random
0.042
–
Algorithm-gen.
Label-free
0.127[0.071,0.190]
0.278
GPT-4o judge
0.302
0.429
Appendix
Table 18: Who&When attribution (text-only, no gold seen). Bootstrap CIs in brackets.
Method
split
acc
lift
lift 95% CI
clears
algorithm-generated, always-first 0.159 , n=126
RAFFLES
alg
0.516
+0.357
[+0.25,+0.46]
yes
AgenTracer-8B (w/G)
alg
0.429
+0.270
[+0.16,+0.37]
yes
AgenTracer-8B (w/oG)
alg
0.373
+0.214
[+0.11,+0.32]
yes
CORRECT
alg
0.381
+0.222
[+0.12,+0.33]
yes
Who&When best
alg
0.254
+0.095
[−0.01,+0.19]
no
Appendix
Table 19: Published Who&When results under the positional correction. Accuracies are exact integer counts on the full split, transcribed from the primary sources. Intervals resample trajectories, recomputing the best content-blind rule on each draw, so the baseline’s own uncertainty is included. A fixed-baseline binomial test would credit two more rows.
Protocol
Split
acc
reported
lift
p
all-at-once
alg-gen
0.151
0.125
−0.008
0.61
all-at-once
hand
0.052
0.053
−0.069
0.95
step-by-step ∗
alg-gen
0.040
0.255
−0.119
1.00
step-by-step ∗
hand
0.035
0.070
−0.086
0.98
Appendix
Table 20: A published method run under its own released code. reported is the accuracy its paper gives for that cell, lift the gain over the matched positional rule. ∗ this protocol did not reproduce its reported accuracy here, so these two rows support no conclusion about the method.
Model
Split
acc
base
lift
95% CI
p
Reimplementation
alg-gen
0.254
0.159
+0.095
[0.01,0.18]
0.032
hand
0.155
0.121
+0.034
[−0.09,0.16]
0.38
Reasoning model
alg-gen
0.214
0.159
+0.056
[−0.03,0.14]
0.148
hand
0.138
0.121
+0.017
[−0.09,0.12]
0.495
Appendix
Table 21: Two attribution models scored against their matched positional baselines. base is the best content-blind rule on that split. No row survives the audit’s corrected threshold of p≤0.025 .
Figure 8: Positional lift of the best learned localizer across nine attribution settings, sorted. Hatching marks external benchmarks.
Benchmark
split
n
steps
pos.
ceil.
rand.
SILENT-Bench
τ 70B airline
39
8.6
0.84
0.666
0.138
τ 8B airline
34
5.5
0.98
0.884
0.502
τ 8B retail
110
4.8
0.99
0.964
0.309
τ Qwen airline
47
9.4
0.66
0.106
0.111
τ Gemma airline
27
9.6
0.82
0.400
0.106
Who&When
hand-crafted
58
51.6
0.46
0.106
0.042
Appendix
Table 22: Per-split positional audit. ceil. held-out positional ceiling, rand. random baseline, pos. mean gold relative position.
Qwen-7B
Mistral-24B
label-free
Subset
ceil.
acc
lift
acc
lift
acc
lift
GSM8K
0.193
0.520
+0.327
0.485
+0.292
0.220
+0.027
MATH
0.189
0.380
+0.191
0.330
+0.141
0.165
−0.024
OlympiadBench
0.184
0.320
+0.136
0.275
+0.091
0.165
−0.019
OmniMATH
0.225
0.330
+0.105
0.320
+0.095
0.175
−0.050
Appendix
Table 23: Discriminant check on ProcessBench ( 200 erroneous instances per subset, no unparseable replies). Both judge families earn positive lift on every subset, while a content-blind-in-practice text heuristic does not.
Corpus / domain
failures
state-verified
%
Regenerated, airline
154
56
36.4
Regenerated, retail
442
10
2.3
Greedy pull, all splits
257
39
15.2
Regenerated, total
596
66
11.1
Appendix
Table 24: Failures admitting a state-verified decisive step, by corpus and domain.
Benchmark
Domain
Control
n
% recov.
τ2 -bench
retail
116/120
60
63.3[0.51,0.74]
τ2 -bench
telecom
100/100
199
87.4[0.82,0.91]
τ2 -bench
airline
80/100
–
not reported
SWE-bench
code
198/203
198
7.6[0.04,0.12]
Appendix
Table 25: Recoverability in two further environments. Control is the share of full-length trajectories on which the probe reproduces the benchmark’s own scoring. A domain is reported only if its control passes.
TRAIL split
pos.
best pos.
learned
lift
PSS
GAIA (retr.)
.53
mid 0.10
0.00
−0.10
−0.11
SWE (code)
.26
mid 0.03
0.00
−0.03
−0.03
Appendix
Table 26: TRAIL positional-lift audit ( 143 real agent traces). A GPT-4o judge fails to clear the weak positional baselines despite a non-positional gold.
Figure 9: Relative position of the gold give-up step and the GPT-4o judge’s prediction, 70B airline split.
Trace length
n
pred. in first two
exact match
Long ( ≥6 steps)
32
18 ( 56% )
6 ( 19% )
Short ( ≤5 steps)
7
1
7 ( 100% )
Appendix
Table 27: GPT-4o judge predicted decisive-step positions on the 70B airline split, by trace length (gold give-up step is late, mean relative position 0.87 ).
Split
prompt
acc
pred. pos.
last
lift
8B airline
naive
0.41
0.14
0.88
−0.47
aware
0.85
0.54
−0.03
70B airline
naive
0.05
0.28
0.67
−0.62
aware
0.31
0.73
−0.36
Appendix
Table 28: Give-up-aware prompt ablation, open judge (Mistral-Small-24B). pred. pos. mean predicted relative position, last always-last accuracy.
Split
n
3-rule
fixed idx
length map
best k
W&W hand-crafted
58
0.104
0.134
0.116
8
W&W algorithm-gen
126
0.142
0.260
0.200
1
PB gsm8k
207
0.191
0.276
0.255
1
PB math
594
0.227
0.240
0.261
1
PB olympiadbench
661
0.170
0.221
0.203
2
PB omnimath
759
0.157
0.238
0.241
1
Appendix
Table 29: Larger content-blind reference families, half-split held-out accuracy averaged over 2000 splits. 3-rule is the paper’s family P , fixed idx the best absolute index, length map the length-conditional table, best k the in-sample best fixed index (zero-indexed).
Benchmark / split
PSS
95% CI
floored ( ϵ=0.10 )
τ 8B airline
−3.0
[−14.0,−1.1]
−3.0
τ 8B retail
−23.3
[−97.0,−10.9]
−8.5
τ 70B airline
−1.1
[−2.5,−0.4]
−1.1
τ Qwen airline
+0.20
–
+0.20
τ Gemma airline
+0.06
[0.00,0.20]
+0.06
W&W hand-crafted
+0.06
[0.00,0.14]
+0.06
Appendix
Table 30: Normalised skill score for the Table 2 splits, unfloored and floored. Only the saturated retail split changes materially, and every sign is preserved.
Corpus
median
p 95
share >3
GSM8K Qwen-7B
0.04
2.75
4.1%
FinQA Qwen-7B
0.08
5.07
11.7%
τ -bench 8B
−0.64
0.90
0.4%
FinQA Llama-8B
−0.16
1.46
1.0%
GSM8K Llama-8B
−0.19
2.43
3.4%
Appendix
Table 31: Per-coordinate excess kurtosis of the standardized trusted activations, testing assumption (B1). A Gaussian coordinate has excess kurtosis 0 .
LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.
Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.
Jiale Liu, Huajun Xi, Shaokun Zhang +6
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.