Executable Prefix Values for Locating Agent Failures

Bingyu Shen; Jiaqi Tang; Lan Wei; Yi Sheng; J. Jenny Li; Boyang Li

cs.AIOct 13, 2026

Executable Prefix Values for Locating Agent Failures

Authors: Bingyu Shen, Jiaqi Tang, Lan Wei, Yi Sheng, J. Jenny Li, Boyang Li

Organizations: Independent Researcher · Department of Computer Science and Technology, Kean University, Union, NJ, USA · Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, FL, USA

Abstract

Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.

Figures & tables

Figure 1: Symptom or cause? A coding agent fails a task (top; one square per step). A judge that reads the transcript points at the first visible mistake, typically the first traceback (middle), but by then the task was already lost. We restart the agent from every step of its own run and watch its chance of success (bottom): it collapses at the step where the agent adopted the wrong idea, which is the cause. Figure 2 shows measured curves.

Judge	no mistake	flagged	position	3 passes
	(failed)	decoys	median	identical
GPT-6.1 Sol	1/47	8/10	0.23	15/20
GPT-6 Astra	1/47	8/10	0.23	19/20
GPT-5.5	11/47	4/10	0.39	6/20
OpenAI o3	13/47	3/10	0.47	5/20
GPT-6 Luna (contrast)	9/47	3/10	0.44	8/20

Table 1: Judges on the 47 failed trajectories and 10 decoys (pass 1). Position is the labeled step as a fraction of the trajectory. A label on a decoy is over-flagging. Self-consistency on the 20-trajectory subset.

Figure 2: V-curves of three resolved trajectories (largest-

n

estimate at every point, Wilson 95% band) with the first-mistake labels of the three main judges of different lineages drawn between

h_{m-1}

and

h_{m}

. Every curve of the study is in Appendix W , every trajectory’s numbers in Table 9 .

Source	pre	on	post	post-	none	zero-	symptom
				PNR		drop
GPT-6.1 Sol	23	3	12	8	1	30/46	78%
GPT-6 Astra	24	3	12	7	1	30/46	78%
GPT-5.5	11	1	13	11	11	21/36	81%
OpenAI o3	9	0	8	17	13	26/34	91%
four judges	67	7	45	43	26	107/162	81%

Table 2: Where the labels fall on the 47 resolved trajectories: before the decisive interval

(a,b]

, inside it, after it, after the certified point of no return, or “no mistake”; zero-drop counts labeled steps whose probed drop

V(h_{m-1})-V(h_{m})

\leq 0

; the last column is the share of labels that are post-cause or zero-drop.

Figure 3: Both policies. Left: where the four main judges’ labels fall relative to the decisive interval (share of labels; before it, inside it, after it, after the certified point of no return). Right: gain over never restarting of each restart policy on a random run of the affected tasks, evaluated offline (§ 6.5 ); shaded: restarting at a judge’s label (GPT-6.1 Sol, GPT-6 Astra, GPT-5.5, OpenAI o3).

Appendix figures & tables67 assets

Supplementary material from the paper’s appendix.

Appendix

Input: a failed trajectory

h_{T}

, anchors

A

, the judges’ labeled steps

M

1. Screen: for

t\in A

K=8

continuations from

h_{t}

;

V(h_{T})=0

exactly.

2. Bisect: for the up to three adjacent anchor pairs with

\hat{V}(a)-\hat{V}(b)\geq 0.25

, in order of drop size: sample

K=8

\lfloor(a+b)/2\rfloor

and recurse into the half with the larger drop until

b-a=1

3. Confirm: raise both ends of the largest bracketed drop to

K=32

; a second resolved drop of at least 0.25 is confirmed too.

4. Point of no return: at the first anchor with no success, raise to

K=24

; if its one-sided Wilson upper bound is below

0.15

(certified), bisect backward to the last earlier point with a success and raise the final candidate to

K=24

;

t_{\text{pnr}}

is the earliest certified point. If the anchor does not certify, try the next anchor with no success.

5. Label probes: for each

m\in M

(at most four), raise

m-1

and

m

K=16

(to 32 if the drop there is at least 0.20).

Output: points

(t,n,\text{successes})

, isotonic fit,

t^{*}

with its interval,

C

, shape,

t_{\text{on}}

t_{\text{pnr}}

Appendix

Table 3: The adaptive schedule of Stage 4. Continuations are keyed by (trajectory,

t

, seed) and reused across stages; raising

K

at a point adds continuations with the next seeds.

Check	Criterion	Value
Replay match	$\geq 70\%$ of selected	73%
Harness flake rate	double-eval if $>3\%$	0/46
$t=0$ sanity, 5 traj.	Fisher exact, $p\geq.05$	5/5
Test–retest, $K=8$	median gap $\leq 0.2$	0.00 (30 anchors; max 0.38)
Cap hits, continuations	$<35\%$	31% of 15,576 (L0+L1 6%; L2 62%)
Confirmed drop or certified $t_{\text{pnr}}$	$\geq 60\%$ of traj.	47/47

Appendix

Table 4: Criteria the method must meet before any claim is made, and their values on the frozen set (§ 6.10 ).

Judge	no mistake	flagged	position	3 passes
	(failed)	decoys	median	identical
GPT-6.1 Sol	1/60	10/10	0.19	15/20
GPT-6 Astra	1/60	10/10	0.15	18/20
GPT-5.5	6/60	3/10	0.28	10/20
OpenAI o3	15/60	3/10	0.41	5/20
GPT-6 Luna (contrast)	5/60	7/10	0.23	10/20

Appendix

Table 5: Judges on the second policy’s 60 failed trajectories and 10 decoys (pass 1), as in Table 1 .

Pair	exact	within 3
GPT-6.1 Sol vs. GPT-6 Astra	52/57	53/57
GPT-6.1 Sol vs. GPT-5.5	16/57	18/57
GPT-6.1 Sol vs. OpenAI o3	9/57	12/57
GPT-5.5 vs. OpenAI o3	21/57	24/57
GPT-5.5 vs. GPT-6 Astra	15/57	17/57
OpenAI o3 vs. GPT-6 Astra	9/57	12/57

Appendix

Table 6: Agreement of pass-1 labels over 57 trajectories (failed and decoys). Two “no mistake” answers agree; “no mistake” never agrees with a step.

Parameter	Value
Budget $B$ (steps)	100
Base runs per task	8
Selection band (successes of 8)	2–6
Trajectories per task, at most	3
Decoys / recovery pool	10 / 20
Anchors	every 5 to step 50, every 10 after, $T-1$

Appendix

Table 7: Estimator and schedule parameters ( configs/thresholds.yaml ).

Quantity	Value
Tasks, runs	150, 1200
pass@1	0.683
pass@2 / @4 / @8	0.63 / 0.58 / 0.51
Doomed (0–1 of 8)	0.24
Saturated (7–8 of 8)	0.62
Band (2–6 of 8)	0.14 (21 tasks)

Appendix

Table 8: Base-run pool of the main policy (left) and tasks by number of successes out of 8 (right).

Trajectory	Str.	$T$	$\hat{V}(h_{0})$	Shape	$C$	$t^{*}$	$(a,b]$	$t_{\text{pnr}}$	$t_{\text{edit}}$	$t_{\text{sym}}$	$n$
django-11555 s2	L1	88	0.48	erosion	0.30	35	(34,35]	71	44	45	376
django-11555 s5	L2	100	0.40	erosion	0.39	86	(85,90]	93	52	54	440
django-11555 s7	L1	99	0.53	erosion	0.23	6	(5,10]	82	40	41	392
django-12325 s1	L1	67	0.07	erosion	0.40	11	(10,11]	15	30	39	248
django-12325 s2	L1	90	0.06	cliff	0.60	2	(1,2]	2	38	46	296
django-12325 s6	L1	67	0.05	cliff	1.00	16	(15,20]	0	31	36	200

Appendix

Table 9: Curve quantities of every trajectory of the frozen set (1 of 2);

\dagger

: both endpoints of the interval at

K\geq 32

Trajectory	Str.	$T$	$\hat{V}(h_{0})$	Shape	$C$	$t^{*}$	$(a,b]$	$t_{\text{pnr}}$	$t_{\text{edit}}$	$t_{\text{sym}}$	$n$
seaborn-3069 s2	L2	100	0.50	cliff	0.53	14	(13,14] †	91	44	56	368
seaborn-3069 s3	L2	100	0.43	erosion	0.42	2	(1,2]	30	59	82	384
seaborn-3069 s5	L2	100	0.54	erosion	0.39	62	(61,62]	62	29	30	344
requests-6028 s0	L0	64	0.12	cliff	0.50	26	(25,26]	26	38	–	248
requests-6028 s1	L0	57	0.24	erosion	0.35	13	(12,13] †	15	23	39	240
requests-6028 s4	L0	61	0.29	cliff	0.65	25	(24,25]	29	35	–	360

Appendix

Table 10: Curve quantities of every trajectory of the frozen set (2 of 2);

\dagger

: both endpoints of the interval at

K\geq 32

			GPT-6.1	GPT-6	GPT-5.5	OpenAI	GPT-6	Claude	Human
Trajectory	$t^{*}$	$t_{\text{pnr}}$	Sol	Astra		o3	Luna	Opus 5.5
django-11555 s2	35	71	13	13	56	44	44	$\cdot$	$\cdot$
django-11555 s5	86	93	12	12	58	none	52	none	none
django-11555 s7	6	82	13	13	82	82	65	$\cdot$	$\cdot$
django-12325 s1	11	15	27	27	none	28	none	none	none
django-12325 s2	2	2	38	38	none	22	none	none	38

Appendix

Table 11: First-mistake labels of every source next to

t^{*}

and

t_{\text{pnr}}

, first half of the set (“none” = no mistake;

\cdot

= not labelled by that source).

			GPT-6.1	GPT-6	GPT-5.5	OpenAI	GPT-6	Claude	Human
Trajectory	$t^{*}$	$t_{\text{pnr}}$	Sol	Astra		o3	Luna	Opus 5.5
seaborn-3069 s2	14	91	14	14	14	45	45	$\cdot$	$\cdot$
seaborn-3069 s3	2	30	16	16	96	59	59	96	96
seaborn-3069 s5	62	62	16	16	29	29	56	$\cdot$	$\cdot$
requests-6028 s0	26	26	18	18	18	none	18	$\cdot$	$\cdot$
requests-6028 s1	13	15	14	14	14	21	14	15	14

Appendix

Table 12: First-mistake labels of every source next to

t^{*}

and

t_{\text{pnr}}

, second half of the set.

Stage	Cont.	Share	Per traj.
screen	5,240	34%	111
confirm	3,384	22%	72
pnr	2,728	18%	58
label	2,576	17%	55
bisect	1,648	11%	35
all	15,576	100%	331

Appendix

Table 13: Continuations by schedule stage over the frozen set.

Group	$n$	Repl.	Rate
selected	64	47	73%
decoy	10	8	80%
recovery	20	15	75%
L0-submit	24	17	71%
L1-submit	20	12	60%
L2-cap	20	18	90%

Appendix

Table 14: Replay by role and stratum (left); mismatch types and command kinds at the first mismatch (right).

sorted_lines	exact	char_multiset
32	28	10

Appendix

Table 15: Comparison level at which the observations of replayable trajectories matched (steps).

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6
	Sol		o3	Astra	Luna
GPT-6.1 Sol	–	12/13/14	5/7/8	44/44/45	10/11/16
GPT-5.5	12/13/14	–	14/14/17	11/12/13	16/18/22
OpenAI o3	5/7/8	14/14/17	–	5/7/8	14/17/18
GPT-6 Astra	44/44/45	11/12/13	5/7/8	–	11/11/16
GPT-6 Luna	10/11/16	16/18/22	14/17/18	11/11/16	–

Appendix

Table 16: Pairwise agreement on the 47 failed trajectories: exact / within 1 / within 3 steps.

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6
	Sol		o3	Astra	Luna
labels (pass 1)	57	57	57	57	57
unparsed answers	0	0	0	0	0
“no mistake” (failed)	1/47	11/47	13/47	1/47	9/47
flagged decoys	8/10	4/10	3/10	8/10	3/10
position $m/T$ , Q1	0.11	0.29	0.39	0.11	0.27

Appendix

Table 17: Judges: labels, answers not in the required format, “no mistake” on failed runs, flagged decoys, label-position quartiles, self-consistency over three passes on 20 trajectories, spend.

Trajectory	At $h_{0}$	$p(x)$
django-15128 s5	4/8	0.50
django-16938 s4	4/8	0.50
requests-6028 s1	0/8	0.25
sympy-15875 s4	4/8	0.75
sympy-19040 s3	2/8	0.25

Appendix

Table 18:

t=0

sanity: successes of the continuations from the empty history against the task’s base pass rate.

Trajectory	$t$	$\hat{V}_{1}$	$\hat{V}_{2}$	$\|\Delta\|$
django-15128 s5	0	0.50	0.50	0.00
django-15128 s5	35	0.25	0.50	0.25
django-15128 s5	83	0.00	0.00	0.00
django-16938 s4	0	0.50	0.50	0.00
django-16938 s4	30	0.00	0.00	0.00
django-16938 s4	64	0.00	0.00	0.00

Appendix

Table 19: Test–retest: screening estimate at three anchors from the main schedule (

\hat{V}_{1}

) and from fresh seeds (

\hat{V}_{2}

K=8

each.

Trajectory	$T$	$t^{*}_{B-t}$	sh.	$t^{*}_{B}$	sh.	final
django-15128 s5	84	6	cli	21	ero	9
django-16938 s4	65	6	cli	6	cli	11
matplotlib-25960 s5	57	6	cli	41	cli	6
requests-6028 s1	57	11	cli	11	cli	13
requests-6028 s4	61	21	cli	21	cli	25
sympy-13877 s1	58	16	cli	16	cli	6

Appendix

Table 20: Budget control: screening with

B-t

and with the full budget

B

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6	Claude	Human	Pooled
	Sol		o3	Astra	Luna	Opus 5.5
trajectories	47	47	47	47	47	30	30	47
labels	46	36	34	46	38	19	23	162
pre	23	11	9	24	16	3	3	67
on	3	1	0	3	1	1	2	7
post	12	13	8	12	12	8	8	45

Appendix

Table 21: Labels against the curves on the frozen set (47 trajectories), every source and quantity.

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6	Claude	Human	Pooled
	Sol		o3	Astra	Luna	Opus 5.5
trajectories	29	29	29	29	29	19	19	29
labels	28	21	21	28	20	14	18	98
pre	9	5	6	10	7	2	1	30
on	1	0	0	1	0	1	2	2
post	11	9	3	11	7	7	8	34

Appendix

Table 22: The same without the L2 stratum (29 trajectories whose base run submitted).

Policy	Restarts	$\bar{t}$	Failed	Random	Gain
never	0/47	–	0.00	0.44	+0.00
restart point $t_{r}$	30/47	7	0.29	0.57	+0.13
$t_{\text{pnr}}$	47/47	43	0.20	0.53	+0.09
$t^{*}$	47/47	29	0.26	0.56	+0.12
label, GPT-6.1 Sol	46/47	19	0.32	0.40	-0.04
label, GPT-5.5	36/47	39	0.16	0.36	-0.08

Appendix

Table 23: Restart policies on the frozen set (47 trajectories).

\bar{t}

: mean restart step; Failed / Random: expected success on failed runs / on a random run.

Judge	Decoys	Restarted	Loss
GPT-6.1 Sol	10	8	0.46
GPT-5.5	10	4	0.36
OpenAI o3	10	3	0.23
GPT-6 Astra	10	8	0.53

Appendix

Table 24: Judge policies on successful decoys: labelled decoys are restarted; expected loss

1-R(x,B-m)

per decoy.

Policy	Restarts	$\bar{t}$	Failed	Random	Gain
never	0/29	–	0.00	0.43	+0.00
restart point $t_{r}$	24/29	8	0.36	0.60	+0.17
$t_{\text{pnr}}$	29/29	28	0.31	0.57	+0.14
$t^{*}$	29/29	19	0.34	0.59	+0.16
label, GPT-6.1 Sol	28/29	17	0.38	0.42	-0.01
label, GPT-5.5	21/29	29	0.23	0.39	-0.04

Appendix

Table 25: Restart policies without the L2 stratum (29 trajectories).

Trajectory	$p(x)$	$t_{r}$	$t_{\text{pnr}}$	$t^{*}$
django-11555 s2	0.50	–	71/0.00	35/0.00
django-11555 s5	0.50	–	93/0.00	86/0.00
django-11555 s7	0.50	–	82/0.00	6/0.50
django-12325 s1	0.25	0/0.25	15/0.25	11/0.25
django-12325 s2	0.25	0/0.25	2/0.25	2/0.25
django-12325 s6	0.25	0/0.25	0/0.25	16/0.25

Appendix

Table 26: Value-based policies per trajectory: restart step / expected success

R(x,B-t)

(– = no restart).

	GPT-6.1	GPT-6	GPT-5.5	OpenAI
Trajectory	Sol	Astra		o3
django-11555 s2	13/0.25	13/0.25	56/0.00	44/0.00
django-11555 s5	12/0.25	12/0.25	58/0.00	–
django-11555 s7	13/0.25	13/0.25	82/0.00	82/0.00
django-12325 s1	27/0.25	27/0.25	–	28/0.25
django-12325 s2	38/0.12	38/0.12	–	22/0.25

Appendix

Table 27: Judge-label policies per trajectory: restart step / expected success (– = no mistake, no restart).

Trajectory	$V$	$R$ pred.	Obs.
django-12325 s1	0.07	0.25	0/8
django-12325 s2	0.06	0.25	0/8
django-12325 s6	0.05	0.25	0/8
django-13807 s2	0.44	0.62	5/8
django-13807 s3	0.50	0.62	2/8
django-13807 s7	0.25	0.62	3/8

Appendix

Table 28: Online restart check: observed successes of eight fresh runs with budget

B-t_{r}

, next to

V(h_{t_{r}})

and the predicted

R(x,B-t_{r})

Schedule	$n$	Exact	$\pm 1$	$\pm 3$	Found	Capt.
full schedule	276	28%	36%	47%	64%	0.44
screening + bisection	147	16%	25%	34%	82%	0.30
screening at $K=4$	203	16%	24%	34%	84%	0.30
sparse anchors	230	28%	38%	50%	57%	0.45
judge-guided, GPT-6.1 Sol	255	27%	36%	47%	65%	0.44
judge-guided, GPT-5.5	255	27%	37%	47%	66%	0.42

Appendix

Table 29: Cheaper schedules: continuations, agreement with the full curve’s decisive step, drops found, value captured.

Schedule	Paths
judge-guided, GPT-6.1 Sol	fallback full 471, label verified 40, local search 409, no label: full 20
judge-guided, GPT-5.5	fallback full 346, label verified 17, local search 357, no label: full 220
judge-guided, OpenAI o3	fallback full 364, label verified 7, local search 309, no label: full 260
judge-guided, GPT-6 Astra	fallback full 475, label verified 47, local search 398, no label: full 20

Appendix

Table 30: Paths taken by the judge-guided schedules (replications).

Shape	Schedule	Exact	$\pm 3$	Capt.
cli	full schedule	39%	53%	0.45
cli	screening + bisection	23%	40%	0.29
cli	screening at $K=4$	27%	40%	0.35
cli	sparse anchors	44%	58%	0.50
cli	judge-guided, GPT-6.1 Sol	44%	60%	0.52
cli	judge-guided, GPT-5.5	45%	58%	0.52

Appendix

Table 31: Cheaper schedules by failure shape.

Localizer	$n$	VC 0	VC 1	VC 3	A 0	A 1	A 3	$t^{*}_{0}$	$t^{*}_{3}$
GPT-6 Astra	47	0.07	0.16	0.26	0.17	0.20	0.22	0.06	0.32
GPT-6.1 Sol	47	0.07	0.16	0.23	0.18	0.21	0.23	0.06	0.32
Human annotator	30	0.08	0.14	0.22	0.18	0.23	0.33	0.03	0.17
GPT-5.5	47	0.04	0.09	0.13	0.26	0.28	0.31	0.02	0.11
Claude Opus 5.5 (annotator)	30	0.03	0.08	0.19	0.25	0.28	0.33	0.00	0.13
GPT-6 Luna (contrast)	47	0.02	0.07	0.12	0.27	0.30	0.38	0.02	0.09

Appendix

Table 32: Localizers by value captured (VC k ), label agreement (A k ) and agreement with

t^{*}

; sorted by VC 1 .

Family	$n$	Acc.	Cliff	Drop	Onset	PNR	Cont.
cliff	500	79%	79%	72%	81%	100%	327
erosion	500	99%	1%	5%	27%	100%	349
flat	500	–	71%	3%	19%	0%	292
plateau_cliff	500	97%	97%	87%	91%	100%	316

Appendix

Table 33: Estimator validation on synthetic curves with known

t^{*}

and shape.

Judge	$n$	Exact	$\pm 1$	$\pm 3$
GPT-6.1 Sol	40	6	7	8
GPT-5.5	40	19	21	23
OpenAI o3	40	14	14	15
GPT-6 Astra	40	8	9	10
GPT-6 Luna (contrast)	40	12	13	16

Appendix

Table 34: Claude Opus 5.5 (annotator protocol): agreement with each judge, exact / within 1 / within 3 steps.

Source	$n$	Exact	$\pm 1$	$\pm 3$
GPT-6.1 Sol	30	6	8	12
GPT-6 Astra	30	5	6	10
GPT-5.5	30	9	8	10
OpenAI o3	30	2	1	3
GPT-6 Luna (contrast)	30	4	3	6
Claude Opus 5.5 (annotator)	30	15	12	14

Appendix

Table 35: Human annotator: agreement with each judge and with Claude Opus 5.5, exact / within 1 / within 3 steps.

Figure 4: V-curves of the frozen set (1 of 4); one diagnostic note per curve follows.

Figure 5: V-curves of the frozen set (2 of 4); one diagnostic note per curve follows.

Figure 6: V-curves of the frozen set (3 of 4); one diagnostic note per curve follows.

Figure 7: V-curves of the frozen set (4 of 4); one diagnostic note per curve follows.

Quantity	Value
Tasks, runs	150, 1200
pass@1	0.340
pass@2 / @4 / @8	0.25 / 0.16 / 0.09
Doomed (0–1 of 8)	0.52
Saturated (7–8 of 8)	0.19
Band (2–6 of 8)	0.29 (44 tasks)

Appendix

Table 36: Base-run pool of the second policy (left) and tasks by number of successes out of 8 (right).

Trajectory	Str.	$T$	$\hat{V}(h_{0})$	Shape	$C$	$t^{*}$	$(a,b]$	$t_{\text{pnr}}$	$t_{\text{edit}}$	$t_{\text{sym}}$	$n$
django-11333 s6	L2	100	0.38	erosion	0.38	7	(6,7] †	36	22	27	288
django-11451 s0	L0	81	0.88	erosion	0.14	1	(0,5]	71	19	22	352
django-11451 s2	L0	76	0.88	cliff	0.75	76	(75,76] †	76	21	27	264
django-11999 s0	L2	100	0.56	erosion	0.26	71	(70,71]	60	28	35	448
django-11999 s1	L2	100	0.36	erosion	0.35	40	(39,40] †	20	43	53	376
django-11999 s6	L2	100	0.44	erosion	0.38	21	(20,21]	48	–	–	416

Appendix

Table 37: Curve quantities of every trajectory of the second policy’s frozen set (1 of 3);

\dagger

: both endpoints of the interval at

K\geq 32

Trajectory	Str.	$T$	$\hat{V}(h_{0})$	Shape	$C$	$t^{*}$	$(a,b]$	$t_{\text{pnr}}$	$t_{\text{edit}}$	$t_{\text{sym}}$	$n$
django-16899 s3	L0	86	0.72	cliff	0.70	39	(38,39] †	39	39	45	368
django-16899 s7	L0	81	0.88	cliff	0.75	35	(34,35] †	35	37	–	352
xarray-3305 s1	L2	100	0.81	erosion	0.15	6	(5,6]	49	48	63	464
xarray-3305 s5	L2	100	0.81	erosion	0.15	26	(25,26] †	45	44	51	384
xarray-3305 s7	L2	100	0.56	erosion	0.30	6	(5,9]	46	34	37	256
xarray-4695 s3	L2	100	0.16	cliff	0.53	34	(33,35]	35	56	–	312

Appendix

Table 38: Curve quantities of every trajectory of the second policy’s frozen set (2 of 3);

\dagger

: both endpoints of the interval at

K\geq 32

Trajectory	Str.	$T$	$\hat{V}(h_{0})$	Shape	$C$	$t^{*}$	$(a,b]$	$t_{\text{pnr}}$	$t_{\text{edit}}$	$t_{\text{sym}}$	$n$
sphinx-8459 s0	L2	100	0.25	cliff	0.50	1	(0,4]	10	61	–	384
sphinx-9320 s7	L0	40	0.85	cliff	1.00	40	(39,40] †	40	20	–	128
sphinx-9658 s1	L0	85	0.26	cliff	0.71	26	(25,26]	60	42	–	424
sphinx-9658 s3	L2	100	0.29	erosion	0.29	6	(5,6] †	50	85	–	288
sphinx-9658 s6	L2	100	0.47	erosion	0.48	36	(35,37]	100	55	–	360
sympy-13615 s1	L2	100	0.38	erosion	0.33	6	(5,6]	61	30	–	416

Appendix

Table 39: Curve quantities of every trajectory of the second policy’s frozen set (3 of 3);

\dagger

: both endpoints of the interval at

K\geq 32

Trajectory	$t^{*}$	$t_{\text{pnr}}$	Sol	Astra		o3	Luna
			GPT-6.1	GPT-6	GPT-5.5	OpenAI	GPT-6
django-11333 s6	7	36	7	7	7	22	7
django-11451 s0	1	71	19	17	19	19	19
django-11451 s2	76	76	21	21	21	21	21
django-11999 s0	71	60	28	15	99	99	28
django-11999 s1	40	20	11	11	43	43	11

Appendix

Table 40: First-mistake labels of every source next to

t^{*}

and

t_{\text{pnr}}

, first half of the set (“none” = no mistake;

\cdot

= not labelled by that source).

			GPT-6.1	GPT-6	GPT-5.5	OpenAI	GPT-6
Trajectory	$t^{*}$	$t_{\text{pnr}}$	Sol	Astra		o3	Luna
xarray-4695 s4	32	35	6	6	6	8	4
xarray-7393 s0	7	36	34	20	82	82	34
xarray-7393 s4	46	96	50	13	50	50	11
pytest-5631 s0	1	25	4	4	13	18	13
pytest-5631 s6	1	40	6	3	18	46	18

Appendix

Table 41: First-mistake labels of every source next to

t^{*}

and

t_{\text{pnr}}

, second half of the set.

Stage	Cont.	Share	Per traj.
screen	7,328	35%	122
confirm	4,608	22%	77
label	3,272	16%	55
pnr	3,128	15%	52
bisect	2,384	12%	40
all	20,720	100%	345

Appendix

Table 42: Continuations by schedule stage over the frozen set.

Group	$n$	Repl.	Rate
selected	81	60	74%
decoy	10	7	70%
recovery	20	13	65%
L0-submit	22	16	73%
L2-cap	58	43	74%
L2-submit	1	1	100%

Appendix

Table 43: Replay by role and stratum (left); mismatch types and command kinds at the first mismatch (right).

sorted_lines	exact	char_multiset
31	45	4

Appendix

Table 44: Comparison level at which the observations of replayable trajectories matched (steps).

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6
	Sol		o3	Astra	Luna
GPT-6.1 Sol	–	30/30/31	14/14/17	47/49/51	27/27/31
GPT-5.5	30/30/31	–	23/24/28	27/28/30	29/31/33
OpenAI o3	14/14/17	23/24/28	–	8/9/13	17/17/20
GPT-6 Astra	47/49/51	27/28/30	8/9/13	–	21/24/30
GPT-6 Luna	27/27/31	29/31/33	17/17/20	21/24/30	–

Appendix

Table 45: Pairwise agreement on the 60 failed trajectories: exact / within 1 / within 3 steps.

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6
	Sol		o3	Astra	Luna
labels (pass 1)	70	70	70	70	70
unparsed answers	0	0	0	0	0
“no mistake” (failed)	1/60	6/60	15/60	1/60	5/60
flagged decoys	10/10	3/10	3/10	10/10	7/10
position $m/T$ , Q1	0.09	0.12	0.21	0.09	0.10

Appendix

Table 46: Judges: labels, answers not in the required format, “no mistake” on failed runs, flagged decoys, label-position quartiles, self-consistency over three passes on 20 trajectories, spend.

Trajectory	At $h_{0}$	$p(x)$
django-13821 s7	5/8	0.75
django-13837 s2	2/8	0.38
xarray-3305 s5	6/8	0.50
xarray-4695 s4	1/8	0.25
sympy-13615 s1	2/8	0.25

Appendix

Table 47:

t=0

sanity: successes of the continuations from the empty history against the task’s base pass rate.

Trajectory	$t$	$\hat{V}_{1}$	$\hat{V}_{2}$	$\|\Delta\|$
django-11999 s0	0	0.25	0.38	0.12
django-11999 s0	40	0.88	0.62	0.25
django-11999 s0	99	0.00	0.00	0.00
django-12713 s3	0	0.50	0.75	0.25
django-12713 s3	40	0.00	0.12	0.12
django-12713 s3	99	0.00	0.00	0.00

Appendix

Table 48: Test–retest: screening estimate at three anchors from the main schedule (

\hat{V}_{1}

) and from fresh seeds (

\hat{V}_{2}

K=8

each.

Trajectory	$T$	$t^{*}_{B-t}$	sh.	$t^{*}_{B}$	sh.	final
django-11999 s0	100	41	cli	91	cli	71
django-12713 s3	100	16	cli	31	ero	34
django-13028 s6	100	11	cli	51	cli	12
django-13821 s7	100	81	ero	100	cli	71
django-13837 s2	100	6	cli	16	ero	26
django-16116 s0	79	71	cli	79	cli	79

Appendix

Table 49: Budget control: screening with

B-t

and with the full budget

B

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6	Pooled
	Sol		o3	Astra	Luna
trajectories	60	60	60	60	60	60
labels	59	54	45	59	55	217
pre	29	17	11	31	24	88
on	4	4	1	5	4	14
post	19	20	14	19	19	72

Appendix

Table 50: Labels against the curves on the frozen set (60 trajectories), every source and quantity.

	GPT-6.1	GPT-5.5	OpenAI	GPT-6	GPT-6	Pooled
	Sol		o3	Astra	Luna
trajectories	17	17	17	17	17	17
labels	16	14	8	16	13	54
pre	10	7	5	11	8	33
on	0	1	0	1	0	2
post	5	5	1	4	4	15

Appendix

Table 51: The same without the L2 stratum (17 trajectories whose base run submitted).

Policy	Restarts	$\bar{t}$	Failed	Random	Gain
never	0/60	–	0.00	0.45	+0.00
restart point $t_{r}$	15/60	6	0.11	0.50	+0.05
$t_{\text{pnr}}$	50/60	49	0.04	0.47	+0.02
$t^{*}$	51/60	25	0.17	0.54	+0.09
label, GPT-6.1 Sol	59/60	23	0.24	0.33	-0.13
label, GPT-5.5	54/60	33	0.16	0.42	-0.03

Appendix

Table 52: Restart policies on the frozen set (60 trajectories).

\bar{t}

: mean restart step; Failed / Random: expected success on failed runs / on a random run.

Judge	Decoys	Restarted	Loss
GPT-6.1 Sol	10	10	0.51
GPT-5.5	10	3	0.23
OpenAI o3	10	3	0.20
GPT-6 Astra	10	10	0.51

Appendix

Table 53: Judge policies on successful decoys: labelled decoys are restarted; expected loss

1-R(x,B-m)

per decoy.

Trajectory	$p(x)$	$t_{r}$	$t_{\text{pnr}}$	$t^{*}$
django-11333 s6	0.38	7/0.25	36/0.00	7/0.25
django-11451 s0	0.50	–	71/0.00	1/0.50
django-11451 s2	0.50	–	–	–
django-11999 s0	0.25	–	60/0.00	71/0.00
django-11999 s1	0.25	–	20/0.00	40/0.00
django-11999 s6	0.25	–	48/0.00	21/0.00

Appendix

Table 54: Value-based policies per trajectory: restart step / expected success

R(x,B-t)

(– = no restart).

	GPT-6.1	GPT-6	GPT-5.5	OpenAI
Trajectory	Sol	Astra		o3
django-11333 s6	7/0.25	7/0.25	7/0.25	22/0.12
django-11451 s0	19/0.50	17/0.50	19/0.50	19/0.50
django-11451 s2	21/0.50	21/0.50	21/0.50	21/0.50
django-11999 s0	28/0.00	15/0.12	99/0.00	99/0.00
django-11999 s1	11/0.12	11/0.12	43/0.00	43/0.00

Appendix

Table 55: Judge-label policies per trajectory: restart step / expected success (– = no mistake, no restart).

Policy	Restarts	$\bar{t}$	Failed	Random	Gain
never	0/17	–	0.00	0.52	+0.00
restart point $t_{r}$	2/17	12	0.07	0.55	+0.02
$t_{\text{pnr}}$	7/17	48	0.02	0.53	+0.01
$t^{*}$	8/17	24	0.10	0.57	+0.05
label, GPT-6.1 Sol	16/17	21	0.31	0.36	-0.16
label, GPT-5.5	14/17	27	0.21	0.48	-0.04

Appendix

Table 56: Restart policies without the L2 stratum (17 trajectories).

Schedule	$n$	Exact	$\pm 1$	$\pm 3$	Found	Capt.
full schedule	289	30%	36%	46%	73%	0.47
screening + bisection	162	25%	31%	39%	90%	0.40
screening at $K=4$	213	24%	30%	36%	94%	0.37
sparse anchors	241	32%	38%	46%	66%	0.49
judge-guided, GPT-6.1 Sol	265	31%	38%	47%	72%	0.48
judge-guided, GPT-5.5	258	33%	37%	47%	74%	0.50

Appendix

Table 57: Cheaper schedules: continuations, agreement with the full curve’s decisive step, drops found, value captured.

Schedule	Paths
judge-guided, GPT-6.1 Sol	fallback full 502, label verified 25, local search 653, no label: full 20
judge-guided, GPT-5.5	fallback full 430, label verified 49, local search 601, no label: full 120
judge-guided, OpenAI o3	fallback full 396, label verified 12, local search 492, no label: full 300
judge-guided, GPT-6 Astra	fallback full 489, label verified 53, local search 638, no label: full 20

Appendix

Table 58: Paths taken by the judge-guided schedules (replications).

Shape	Schedule	Exact	$\pm 3$	Capt.
cli	full schedule	66%	76%	0.70
cli	screening + bisection	54%	63%	0.58
cli	screening at $K=4$	50%	59%	0.53
cli	sparse anchors	69%	80%	0.74
cli	judge-guided, GPT-6.1 Sol	66%	76%	0.70
cli	judge-guided, GPT-5.5	71%	81%	0.75

Appendix

Table 59: Cheaper schedules by failure shape.

Localizer	$n$	VC 0	VC 1	VC 3	A 0	A 1	A 3	$t^{*}_{0}$	$t^{*}_{3}$
GPT-6 Astra	60	0.05	0.10	0.18	0.29	0.31	0.36	0.08	0.22
GPT-5.5	60	0.05	0.08	0.15	0.44	0.46	0.49	0.07	0.20
GPT-6.1 Sol	60	0.04	0.08	0.16	0.37	0.37	0.40	0.07	0.22
GPT-6 Luna (contrast)	60	0.03	0.07	0.11	0.39	0.41	0.48	0.07	0.13
step 1	60	0.03	0.06	0.07	0.00	0.01	0.02	0.12	0.13
OpenAI o3	60	0.01	0.04	0.10	0.25	0.26	0.32	0.02	0.13

Appendix

Table 60: Localizers by value captured (VC k ), label agreement (A k ) and agreement with

t^{*}

; sorted by VC 1 .

Figure 8: V-curves of the second policy’s frozen set (1 of 5); one diagnostic note per curve follows.

Figure 9: V-curves of the second policy’s frozen set (2 of 5); one diagnostic note per curve follows.

Figure 10: V-curves of the second policy’s frozen set (3 of 5); one diagnostic note per curve follows.

Figure 11: V-curves of the second policy’s frozen set (4 of 5); one diagnostic note per curve follows.

Figure 12: V-curves of the second policy’s frozen set (5 of 5); one diagnostic note per curve follows.

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 154 papers

Oct 12, 2026

Structural Baselines Matter for Agent Failure Detection and Attribution

An LLM agent using tools can fail silently. It stops before the task is done, raises no error, and answers fluently. Two kinds of monitor address this, one deciding whether a run failed and the other naming the step where failure became unavoidable. Much of the reported performance on both tasks comes from regularities that need no model. Failing runs are longer, and the step labelled decisive is usually the last. A rule that always names the last step is correct

67\%

96\%

of the time here, without reading the run at all. That ceiling reflects how the labels were written, not how agents break. On failures whose decisive step replay can confirm, it falls to near chance. Reporting gain over that rule therefore changes what a score means. Under that correction, methods tested here gain nothing on give-up failures, while several published systems keep a clear advantage on other splits. For detection, the agent's internal states beat transcript length and text similarity on the agent benchmark, and hold up where the text baseline does not. Precision stays low when failures are rare, so such a monitor should rank runs for review, not replace it.

Ramesh Arvind Naagarajan, Stefan Streif

Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany

Agent Failure Analysis Failure Attribution for LLM Agents LLM Agent Evaluation LLM Agent Reliability

Ask PDF

Oct 9, 2026

Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

Failure attribution for LLM agents aims to identify decisive steps responsible for task failure, which is critical for debugging and improving agentic systems. Existing methods rely on costly LLM prompting, step-level error annotations, or verified successful trajectories that lack failure signals for assistance. In this paper, we study failure attribution from step-unlabeled failure trajectories, where each trajectory is known to have failed but its decisive step is unknown. We propose SOAP (Spectral scOring with Attention-guided Propagation), an effective framework requiring neither step-level training labels nor a separate success-only reference set. SOAP first constructs a spectral reference space from LLM representations and assigns each step a base error score according to its projection behavior. Importantly, later steps may exhibit stronger failure signals because they inherit an earlier mistake, causing independent scorers to favor downstream consequences over the decisive error. To address this downstream contamination, SOAP derives soft step dependencies from attention patterns and propagates downstream error evidence toward its likely upstream sources. Extensive experiments show that SOAP consistently improves attribution performance over competitive baselines.

Do Nguyen-Thanh, Junlin Fang, Bing Li +2

College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, University of Technology Sydney

Failure Attribution for LLM Agents Agent Failure Analysis LLM Agents

Ask PDF

Sep 28, 2026

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measure recovery by comparing which action is selected as best across independent runs, a quantity known as set agreement. However, set agreement is a purely ordinal measure that records which action wins without reflecting the absolute level of performance. When all actions fail, they tie at zero reward, and independent runs produce the same tied set with high probability, creating an illusion of stability that masks near-zero recovery success. We formalize this limitation through a set-path symmetry result, proving that for equal-cost Bernoulli actions the success probabilities (0.9, 0.8) and (0.2, 0.1) yield identical best-action-set distributions at every sample size. No procedure based solely on which action wins can distinguish these two regimes. We further prove that certifying exact population ties is impossible in finite time, and that the assignment of outcomes to checkpoints carries information beyond marginal outcome distributions. The pooled success probability is the missing scalar that resolves the ordinal ambiguity. Experiments on 864 frozen RecoveryBench episodes and two planning cohorts totaling 3,456 responses confirm the theoretical predictions. Agreement and held-out quality can move in opposite directions, and permuting checkpoint-to-action bindings changes 8 to 13 percent of cell-level conclusions. Based on these findings, we propose reporting four diagnostic quantities (agreement, all-zero fraction, held-out success, and pooled success) that expose this failure mode with no additional data collection.

Dong Xu, Zhangfan Yang, Jiantao Wu +5

School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

LLM Agent Evaluation AI Agent Benchmarks LLM Agent Reliability

Ask PDF

Abstract

Agent Failure Analysis

Explore similar work

Structural Baselines Matter for Agent Failure Detection and Attribution

Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures