Structural Baselines Matter for Agent Failure Detection and Attribution

Ramesh Arvind Naagarajan; Stefan Streif

cs.AIOct 12, 2026

Structural Baselines Matter for Agent Failure Detection and Attribution

Authors: Ramesh Arvind Naagarajan, Stefan Streif

Organizations: Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany

Abstract

An LLM agent using tools can fail silently. It stops before the task is done, raises no error, and answers fluently. Two kinds of monitor address this, one deciding whether a run failed and the other naming the step where failure became unavoidable. Much of the reported performance on both tasks comes from regularities that need no model. Failing runs are longer, and the step labelled decisive is usually the last. A rule that always names the last step is correct $67\%$ to $96\%$ of the time here, without reading the run at all. That ceiling reflects how the labels were written, not how agents break. On failures whose decisive step replay can confirm, it falls to near chance. Reporting gain over that rule therefore changes what a score means. Under that correction, methods tested here gain nothing on give-up failures, while several published systems keep a clear advantage on other splits. For detection, the agent's internal states beat transcript length and text similarity on the agent benchmark, and hold up where the text baseline does not. Precision stays low when failures are rare, so such a monitor should rank runs for review, not replace it.

Figures & tables

Panel of Figure 1 — Figure 1: The label-free pipeline, with detection AUROC on the regenerated corpus, rounded to two digits from Table 1 . Calibration uses trusted successful runs only, leave-one-out, so no failure label enters the monitor. $\ast$ tool grounding is measured on the greedy corpus.

Label-free signal	pooled	95% CI	resid.	airline	retail
structural baselines (no calibration)
Transcript length	$0.640$	$[.564,.723]$	–	$0.740$	$0.456$
Step count	$0.716$	$[.652,.777]$	–	$0.769$	$0.585$
Give-up keyword ∗	$0.594$	$[.564,.627]$	–	–	–
prior trajectory anomaly detectors
Cycle detection, hybrid	$0.670$	$[.592,.750]$	–	–	–

Table 1: Detection AUROC, regenerated

\tau

-bench corpus (

660

trajectories,

64

trusted successes). resid. residualises on log transcript length. ∗ scored on the greedy corpus (

165

runs), the only vintage with a tool oracle. Every other row is the regenerated corpus. † scored by the probability the judge assigns to its own failure verdict, since its emitted verdict is near-constant (Appendix C.3 ).

Benchmark / split	$n$	pos.	best pos.	learned	SOTA	lift	lift ${}^{\text{ho}}$
$\tau$ 8B airline	$34$	$.98$	last $0.88$	$0.53$	–	$-0.35$	$-0.36$
$\tau$ 8B retail	$110$	$1.00$	last $0.96$	$0.12$	–	$-0.85$	$-0.85$
$\tau$ 70B airline	$39$	$.84$	last $0.67$	$0.31$	–	$-0.36$	$-0.36$
$\tau$ Qwen airline	$47$	$.66$	last $0.15$	$0.32$	–	$+0.17$	$+0.22$
$\tau$ Gemma airline	$27$	$.83$	last $0.41$	$0.44$	–	$+0.04^{\ddagger}$	$+0.04^{\ddagger}$
W&W hand-crafted	$58$	$.46$	last $0.12$	$0.17$	$0.28$	$+0.05^{\ddagger}$	$+0.07^{\ddagger}$

Table 2: Positional-lift audit of step attribution. pos. mean gold relative position, lift learned minus best positional baseline, lift

{}^{\text{ho}}

with the rule chosen on a held-out half, SOTA best published accuracy ( Zhu et al., 2026a ) . ‡ the interval on skill touches zero (permutation

p\geq 0.25

), so these cells are null results rather than small gains. The normalised skill score, with intervals, is in Appendix E.3 .

Benchmark / domain	Agent	$n$	% recov.	ceiling
$\tau$ -bench airline	Llama-3.1-8B	$34$	$35.3$	$0.88$
$\tau$ -bench airline	Llama-3.1-70B	$39$	$25.6$	$0.67$
$\tau$ -bench airline	Qwen2.5-7B	$47$	$25.5$	$0.11$
$\tau$ -bench airline	Gemma-2-27B	$27$	$14.8$	$0.40$
$\tau$ -bench airline	Mistral-24B	$30$	$0.0$	–
$\tau$ -bench airline	Llama-3.1-8B	$154$	$36.4$	–

Table 3: How often a failure has a decisive step that replay can verify. % recov. share of failures where some prefix makes the goal unreachable, ceiling positional ceiling of that split. Methods and intervals in Appendices D.7 and D.8 .

Corpus	trusted	dict.	give-up	best latent	full
GSM8K Qwen-7B	$1866$	$7168$	$0$	$0.61$	$0.73$
GSM8K Mistral-24B	$881$	$10240$	$0$	$0.60$	$0.64$
$\tau$ -bench 8B	$261$	$8192$	$1$	$0.63$	$0.80$
FinQA Llama-8B	$165$	$8192$	$3$	$0.65$	$0.75$

Table 4: Sparse dictionaries fitted on trusted steps only. give-up latents active on over

10\%

of failing and under

1\%

of trusted steps, best latent best single-latent AUROC, full full-space distance.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Agent	Domain	runs	succ.	fail
Llama-3.1-8B	airline	$50$	$16$	$34$
Llama-3.1-8B	retail	$115$	$5$	$110$
Llama-3.1-70B	airline	$50$	$11$	$39$

Appendix

Table 5: Greedy corpus composition. Tool-error rate is

0.44

on airline and

0.72

on retail, over

654

executed tool returns.

Panel of Figure 2 — Figure 2: Two silent-failure modes from the datasets, with real trajectory text.

Quantity	Corpus	Where
Detection AUROC, all signals	regenerated	Table 1
Length and step-count baselines	regenerated	Table 1
Tool-grounding signal	greedy	Table 1
Give-up keyword row	greedy	Table 1
Domain transfer	regenerated	§ 5.6
Recoverability fraction	regenerated	App. D.7

Appendix

Table 6: Which data vintage underlies which reported quantity.

Task (failure)	Model	$n$ (succ.)	length	SBERT	internal	$p$
$\tau$ -bench (omission)	Llama-8B	$660$ ( $64$ )	0.64	0.55	$\mathbf{0.79}$	$<0.001$
GSM8K (wrong ans.)	Llama-8B	$100$ ( $29$ )	0.76	0.62	0.65	$0.720$
GSM8K (wrong ans.)	Qwen-7B	$1319$ ( $852$ )	0.58	0.58	$\mathbf{0.73}$	$<.001$
GSM8K (wrong ans.)	Mistral-24B	$300$ ( $233$ )	0.68	0.67	0.63	$0.314$
GSM8K (wrong ans.)	Qwen-32B	$300$ ( $256$ )	0.67	0.63	$\mathbf{0.81}$	$0.007$
FinQA (wrong ans.)	Llama-8B	$191$ ( $58$ )	0.73	0.70	0.73	$0.508$

Appendix

Table 7: Detection across tasks and model families, leave-one-out calibration. length is the transcript-length baseline.

p

tests internal against SBERT.

Open-pipeline corpus	$n$	succ.	TF-IDF	SBERT	internal
Mistral-24B airline	$50$	$20$	0.40	0.71 [.55,.85]	0.64 [.48,.80]
Llama-8B airline $+$ retail	$150$	$13$	0.52	0.68 [.51,.82]	0.83 [.65,.98]

Appendix

Table 8: Detection AUROC on

\tau

-bench datasets from a fully open pipeline (Qwen2.5-32B user simulator), bootstrap

95\%

CIs in brackets.

Figure 3: Where the gold decisive step falls, as a fraction of the way through a run. On

\tau

-bench the mass sits at the end, which is what makes an always-last rule strong there. Single-step runs are excluded, since for them first and last coincide (Appendix B.1 ).

Figure 4: Decisive-step accuracy on the give-up gold (

95\%

bootstrap CIs). The always-last baseline (grey) dominates the learned localizers and the judge.

Figure 5: Detection AUROC vs residual-stream layer, with text baselines marked.

Figure 6: Internal anomaly score, failing vs successful trajectories (8B).

airline only, open simulator
Corpus	$n$	succ.	internal	SBERT	$p$
greedy	$50$	$9$	$0.799$	$0.745$	$0.455$
temp $0.3$	$200$	$37$	$0.824$	$0.753$	$0.073$
temp $0.7$	$200$	$46$	$0.783$	$0.694$	$0.044$
airline $+$ retail
greedy, GPT-4o sim	$165$	$21$	$0.781$	$0.805$	$0.589$

Appendix

Table 9: Isolating decoding from the user simulator and from domain composition. The greedy GPT-4o row is scored leave-one-out, giving

0.781/0.805

rather than the in-sample

0.842/0.857

Detector	AUROC	$95\%$ CI	$p$ vs internal
Cycle, structural	$0.570$	$[.525,.613]$	$<0.001$
Cycle, semantic	$0.660$	$[.588,.733]$	$<0.001$
Cycle, hybrid	$0.670$	$[.592,.750]$	$<0.001$
Graph rarity	$0.619$	$[.565,.670]$	$<0.001$
Internal, RMS dist.	$\mathbf{0.787}$	$[.708,.857]$	–

Appendix

Table 10: Prior trajectory anomaly detectors on the regenerated corpus (

660

trajectories,

64

trusted successes), leave-one-out, threshold-free. DeLong tests are paired against the internal signal, whose row repeats the Table 1 interval.

Corpus	$n$	max	mean	top-3	last
$\tau$ 8B airline $+$ retail	$165$	$\mathbf{0.781}$	$0.762$	$0.765$	$0.710$
$\tau$ 70B airline	$50$	$0.834$	$0.802$	$\mathbf{0.851}$	$0.816$
$\tau$ 8B retail	$115$	$0.473$	$0.440$	$0.413$	$\mathbf{0.482}$
GSM8K Llama-8B	$100$	$0.645$	$0.659$	$\mathbf{0.681}$	$0.646$
GSM8K Qwen-7B	$1319$	$0.733$	$\mathbf{0.754}$	$0.752$	$0.718$
GSM8K Mistral-24B	$300$	$0.631$	$0.657$	$0.655$	$\mathbf{0.669}$

Appendix

Table 11: Detection AUROC by run-level aggregator, leave-one-out throughout. The maximum, which the main results use, is best or tied on 3 of the 13 datasets scored. Ten are shown, and the remaining three follow the same pattern. The 70B row rests on

11

trusted successes and is underpowered (Section 5.6 ).

Figure 7: Detection AUROC versus trusted-reference size for the internal and SBERT signals (mean

\pm

std over

50

resamples). Both degrade smoothly, and a few tens of successful trajectories suffice.

Split	Signal	$N{=}2$	$3$	$5$	$8$	$16$
Airline	Internal	–	$0.809$	$0.824$	$0.819$	$0.818$
Airline	SBERT	–	$0.768$	$0.765$	$0.766$	$0.772$
Retail	Internal	$0.706$	$0.723$	$0.715$	–	–
Retail	SBERT	$0.822$	$0.830$	$0.860$	–	–

Appendix

Table 12: Per-domain trusted-reference size sweep (8B, mean over

50

resamples, each domain scored on itself). Each sweep stops at that domain’s success count.

Contamination	$0\%$	$4.5\%$	$8.7\%$	$19.2\%$	$30\%$
Internal	$0.842$	$0.846$	$0.844$	$0.848$	$0.849$
SBERT	$0.857$	$0.852$	$0.844$	$0.816$	$0.753$

Appendix

Table 13: Detection AUROC with the trusted calibration set contaminated by unlabeled failing trajectories (pooled

8

B, mean over

50

resamples).

Agent config	Reference	AUROC
Original (greedy)	Original	$0.818$
Drifted (temp 0.7)	Stale original	$0.793$
Drifted (temp 0.7)	Recalibrated	$0.814$

Appendix

Table 14: Staleness under agent drift (8B airline, internal RMS-

z

Signal	FPR	Precision	Recall	$F_{1}$
Internal RMS-z	$5\%$	$0.986$	$0.472$	$0.638$
	$10\%$	$0.976$	$0.556$	$0.708$
	$20\%$	$0.960$	$0.674$	$0.792$
SBERT	$5\%$	$0.986$	$0.493$	$0.657$
	$10\%$	$0.978$	$0.625$	$0.763$
	$20\%$	$0.966$	$0.785$	$0.866$

Appendix

Table 15: Detection precision, recall, and

F_{1}

at three benign-calibrated FPR operating points (pooled Llama-3.1-8B airline

+

retail). The threshold is set on benign traffic alone.

Detector	AUROC
Label-free internal RMS- $z$	$0.842$
Label-free SBERT	$0.857$
Label-free equal-weight fusion	$0.859$
Supervised probe (mean-pool act.)	$0.793\pm 0.028$
Supervised fusion (three signals)	$0.862\pm 0.008$

Appendix

Table 16: Supervised upper bound versus label-free detection (pooled 8B, supervised rows five-fold CV over five seeds).

Failure regime	$n$	internal	SBERT	DeLong $p$
Acted (write call, wrong)	$82$	$0.902$	$0.938$	$0.39$
Gave up (transfer / no write)	$62$	$0.763$	$0.750$	$0.80$

Appendix

Table 17: Detection AUROC by failure regime (pooled

8

B corpus, each regime’s failures against all

21

successes). The internal SBERT tie holds within both regimes.

Split	Method	step-acc [95% CI]	agent
Hand-crafted	Label-free	$0.172$ $[0.086,0.276]$	$\mathbf{0.655}$
	GPT-4o judge	$0.172$	$0.500$
	Always-last	$0.121$ $[0.034,0.207]$	–
	Random	$0.042$	–
Algorithm-gen.	Label-free	$0.127$ $[0.071,0.190]$	$0.278$
	GPT-4o judge	$\mathbf{0.302}$	$0.429$

Appendix

Table 18: Who&When attribution (text-only, no gold seen). Bootstrap CIs in brackets.

Method	split	acc	lift	lift $95\%$ CI	clears
algorithm-generated, always-first $0.159$ , $n{=}126$
RAFFLES	alg	$0.516$	$+0.357$	$[+0.25,+0.46]$	yes
AgenTracer-8B (w/G)	alg	$0.429$	$+0.270$	$[+0.16,+0.37]$	yes
AgenTracer-8B (w/oG)	alg	$0.373$	$+0.214$	$[+0.11,+0.32]$	yes
CORRECT	alg	$0.381$	$+0.222$	$[+0.12,+0.33]$	yes
Who&When best	alg	$0.254$	$+0.095$	$[-0.01,+0.19]$	no

Appendix

Table 19: Published Who&When results under the positional correction. Accuracies are exact integer counts on the full split, transcribed from the primary sources. Intervals resample trajectories, recomputing the best content-blind rule on each draw, so the baseline’s own uncertainty is included. A fixed-baseline binomial test would credit two more rows.

Protocol	Split	acc	reported	lift	$p$
all-at-once	alg-gen	$0.151$	$0.125$	$-0.008$	$0.61$
all-at-once	hand	$0.052$	$0.053$	$-0.069$	$0.95$
step-by-step $\ast$	alg-gen	$0.040$	$0.255$	$-0.119$	$1.00$
step-by-step $\ast$	hand	$0.035$	$0.070$	$-0.086$	$0.98$

Appendix

Table 20: A published method run under its own released code. reported is the accuracy its paper gives for that cell, lift the gain over the matched positional rule. ∗ this protocol did not reproduce its reported accuracy here, so these two rows support no conclusion about the method.

Model	Split	acc	base	lift	$95\%$ CI	$p$
Reimplementation	alg-gen	$0.254$	$0.159$	$+0.095$	$[0.01,0.18]$	$0.032$
Reimplementation	hand	$0.155$	$0.121$	$+0.034$	$[-0.09,0.16]$	$0.38$
Reasoning model	alg-gen	$0.214$	$0.159$	$+0.056$	$[-0.03,0.14]$	$0.148$
Reasoning model	hand	$0.138$	$0.121$	$+0.017$	$[-0.09,0.12]$	$0.495$

Appendix

Table 21: Two attribution models scored against their matched positional baselines. base is the best content-blind rule on that split. No row survives the audit’s corrected threshold of

p\leq 0.025

Figure 8: Positional lift of the best learned localizer across nine attribution settings, sorted. Hatching marks external benchmarks.

Benchmark	split	$n$	steps	pos.	ceil.	rand.
SILENT-Bench	$\tau$ 70B airline	$39$	$8.6$	$0.84$	$0.666$	$0.138$
	$\tau$ 8B airline	$34$	$5.5$	$0.98$	$0.884$	$0.502$
	$\tau$ 8B retail	$110$	$4.8$	$0.99$	$0.964$	$0.309$
	$\tau$ Qwen airline	$47$	$9.4$	$0.66$	$0.106$	$0.111$
	$\tau$ Gemma airline	$27$	$9.6$	$0.82$	$0.400$	$0.106$
Who&When	hand-crafted	$58$	$51.6$	$0.46$	$0.106$	$0.042$

Appendix

Table 22: Per-split positional audit. ceil. held-out positional ceiling, rand. random baseline, pos. mean gold relative position.

		Qwen-7B		Mistral-24B		label-free
Subset	ceil.	acc	lift	acc	lift	acc	lift
GSM8K	$0.193$	$0.520$	$+0.327$	$0.485$	$+0.292$	$0.220$	$+0.027$
MATH	$0.189$	$0.380$	$+0.191$	$0.330$	$+0.141$	$0.165$	$-0.024$
OlympiadBench	$0.184$	$0.320$	$+0.136$	$0.275$	$+0.091$	$0.165$	$-0.019$
OmniMATH	$0.225$	$0.330$	$+0.105$	$0.320$	$+0.095$	$0.175$	$-0.050$

Appendix

Table 23: Discriminant check on ProcessBench (

200

erroneous instances per subset, no unparseable replies). Both judge families earn positive lift on every subset, while a content-blind-in-practice text heuristic does not.

Corpus / domain	failures	state-verified	%
Regenerated, airline	$154$	$56$	$36.4$
Regenerated, retail	$442$	$10$	$2.3$
Greedy pull, all splits	$257$	$39$	$15.2$
Regenerated, total	$596$	$66$	$11.1$

Appendix

Table 24: Failures admitting a state-verified decisive step, by corpus and domain.

Benchmark	Domain	Control	$n$	% recov.
$\tau^{2}$ -bench	retail	$116/120$	$60$	$63.3$ $[0.51,0.74]$
$\tau^{2}$ -bench	telecom	$100/100$	$199$	$87.4$ $[0.82,0.91]$
$\tau^{2}$ -bench	airline	$80/100$	–	not reported
SWE-bench	code	$198/203$	$198$	$7.6$ $[0.04,0.12]$

Appendix

Table 25: Recoverability in two further environments. Control is the share of full-length trajectories on which the probe reproduces the benchmark’s own scoring. A domain is reported only if its control passes.

TRAIL split	pos.	best pos.	learned	lift	PSS
GAIA (retr.)	$.53$	mid $0.10$	$0.00$	$-0.10$	$-0.11$
SWE (code)	$.26$	mid $0.03$	$0.00$	$-0.03$	$-0.03$

Appendix

Table 26: TRAIL positional-lift audit (

143

real agent traces). A GPT-4o judge fails to clear the weak positional baselines despite a non-positional gold.

Figure 9: Relative position of the gold give-up step and the GPT-4o judge’s prediction, 70B airline split.

Trace length	$n$	pred. in first two	exact match
Long ( $\geq 6$ steps)	$32$	$18$ ( $56\%$ )	$6$ ( $19\%$ )
Short ( $\leq 5$ steps)	$7$	$1$	$7$ ( $100\%$ )

Appendix

Table 27: GPT-4o judge predicted decisive-step positions on the 70B airline split, by trace length (gold give-up step is late, mean relative position

0.87

Split	prompt	acc	pred. pos.	last	lift
8B airline	naive	$0.41$	$0.14$	$0.88$	$-0.47$
8B airline	aware	$0.85$	$0.54$	$0.88$	$-0.03$
70B airline	naive	$0.05$	$0.28$	$0.67$	$-0.62$
70B airline	aware	$0.31$	$0.73$	$0.67$	$-0.36$

Appendix

Table 28: Give-up-aware prompt ablation, open judge (Mistral-Small-24B). pred. pos. mean predicted relative position, last always-last accuracy.

Split	$n$	3-rule	fixed idx	length map	best $k$
W&W hand-crafted	$58$	$0.104$	$0.134$	$0.116$	$8$
W&W algorithm-gen	$126$	$0.142$	$0.260$	$0.200$	$1$
PB gsm8k	$207$	$0.191$	$0.276$	$0.255$	$1$
PB math	$594$	$0.227$	$0.240$	$0.261$	$1$
PB olympiadbench	$661$	$0.170$	$0.221$	$0.203$	$2$
PB omnimath	$759$	$0.157$	$0.238$	$0.241$	$1$

Appendix

Table 29: Larger content-blind reference families, half-split held-out accuracy averaged over

2000

splits. 3-rule is the paper’s family

\mathcal{P}

, fixed idx the best absolute index, length map the length-conditional table, best

k

the in-sample best fixed index (zero-indexed).

Benchmark / split	PSS	$95\%$ CI	floored ( $\epsilon{=}0.10$ )
$\tau$ 8B airline	$-3.0$	$[-14.0,-1.1]$	$-3.0$
$\tau$ 8B retail	$-23.3$	$[-97.0,-10.9]$	$-8.5$
$\tau$ 70B airline	$-1.1$	$[-2.5,-0.4]$	$-1.1$
$\tau$ Qwen airline	$+0.20$	–	$+0.20$
$\tau$ Gemma airline	$+0.06$	$[0.00,0.20]$	$+0.06$
W&W hand-crafted	$+0.06$	$[0.00,0.14]$	$+0.06$

Appendix

Table 30: Normalised skill score for the Table 2 splits, unfloored and floored. Only the saturated retail split changes materially, and every sign is preserved.

Corpus	median	p $95$	share $>3$
GSM8K Qwen-7B	$0.04$	$2.75$	$4.1\%$
FinQA Qwen-7B	$0.08$	$5.07$	$11.7\%$
$\tau$ -bench 8B	$-0.64$	$0.90$	$0.4\%$
FinQA Llama-8B	$-0.16$	$1.46$	$1.0\%$
GSM8K Llama-8B	$-0.19$	$2.43$	$3.4\%$

Appendix

Table 31: Per-coordinate excess kurtosis of the standardized trusted activations, testing assumption (B1). A Gaussian coordinate has excess kurtosis

0

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 152 papers

Real-Time Detection and Repair of LLM Agent Failures
Aug 3, 2026Sunny DubeyAI Agent Reliability AI Agent Monitoring
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jul 10, 2026Jiale Liu, Huajun Xi, Shaokun Zhang +6Agent Failure Analysis LLM Agent Evaluation
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
Feb 2, 2026Shraddha Barke, Arnav Goyal, Alind Khare +3Fault Diagnosis AI Agent Reliability
1Microsoft Research · 2Microsoft · 3UIUC

Aug 3, 2026J/K move · Enter open · S save

Real-Time Detection and Repair of LLM Agent Failures

Sunny Dubey

AI Agent Reliability AI Agent Monitoring LLM Agent Reliability Runtime Verification

Open paper AskPDF arXiv

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

Abstract

Agent Failure Analysis

Explore similar work

Real-Time Detection and Repair of LLM Agent Failures

Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories