Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

Dong Xu; Zhangfan Yang; Jiantao Wu; Shipeng Zhang; Zexuan Zhu; Jiangqiang Li; Jun Zhang; Junkai Ji

cs.AISep 28, 2026

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

Authors: Dong Xu, Zhangfan Yang, Jiantao Wu, Shipeng Zhang, Zexuan Zhu, Jiangqiang Li, Jun Zhang, Junkai Ji

Organizations: School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

Abstract

Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measure recovery by comparing which action is selected as best across independent runs, a quantity known as set agreement. However, set agreement is a purely ordinal measure that records which action wins without reflecting the absolute level of performance. When all actions fail, they tie at zero reward, and independent runs produce the same tied set with high probability, creating an illusion of stability that masks near-zero recovery success. We formalize this limitation through a set-path symmetry result, proving that for equal-cost Bernoulli actions the success probabilities (0.9, 0.8) and (0.2, 0.1) yield identical best-action-set distributions at every sample size. No procedure based solely on which action wins can distinguish these two regimes. We further prove that certifying exact population ties is impossible in finite time, and that the assignment of outcomes to checkpoints carries information beyond marginal outcome distributions. The pooled success probability is the missing scalar that resolves the ordinal ambiguity. Experiments on 864 frozen RecoveryBench episodes and two planning cohorts totaling 3,456 responses confirm the theoretical predictions. Agreement and held-out quality can move in opposite directions, and permuting checkpoint-to-action bindings changes 8 to 13 percent of cell-level conclusions. Based on these findings, we propose reporting four diagnostic quantities (agreement, all-zero fraction, held-out success, and pooled success) that expose this failure mode with no additional data collection.

Figures & tables

Figure 1: Overview of the checkpoint evaluation pipeline. Step 1. A failed checkpoint state is frozen and paired with multiple candidate recovery actions (e.g., tool call, replan, retry, switch strategy). Step 2. Each action is executed multiple times independently from the same checkpoint with different random seeds. Step 3. Binary outcomes (success or failure) are recorded for every action–run pair, forming an outcome matrix. Step 4. For each run, the best-action set is computed by selecting all actions with the highest total score (ties retained), yielding a collection of best-action sets that serves as the basis for agreement and quality analysis.

	Agreement			All-zero pairs			Held-out success
Model	$k=1$	$k=2$	$k=4$	$k=1$	$k=2$	$k=4$	$k=1$	$k=2$	$k=4$
Luna	0.854	0.750	0.667	0.854	0.750	0.583	0.014	0.014	0.014
Terra	0.776	0.688	0.583	0.776	0.688	0.583	0.045	0.035	0.014
Sol	0.734	0.627	0.417	0.708	0.590	0.417	0.083	0.075	0.063

Table 1: RecoveryBench validation on 12 failed checkpoints. Each model row reports agreement, all-zero pairs, and held-out success for

k=1,2,4

draws per action. Agreement compares native-reward maximizer sets across blocks; all-zero pairs is the fraction of matching comparisons where every action scores zero in both blocks. Held-out success evaluates the selected set; intervals are in Appendix Table 47 .

Cohort	Model	$k=1$	$k=2$	$k=4$	$k=1$	$k=2$	$k=4$	Fixed verify
		Set agreement			Selected strict success
Study	Terra	0.318	0.204	0.167	0.699	0.695	0.710	0.771
Study	Luna	0.234	0.150	0.083	0.575	0.568	0.538	0.635
Study	Sol	0.424	0.331	0.167	0.722	0.729	0.750	0.729
Replication	Terra	0.404	0.238	0.083	0.765	0.754	0.762	0.771
Replication	Luna	0.299	0.212	0.250	0.727	0.728	0.719	0.760

Table 2: GPT-5.6 planning results on two frozen cohorts (point estimates; intervals in Appendix Tables 39 and 40 ). Agreement compares public-score maximizer sets across blocks. Selected strict success evaluates the chosen action mixture on held-out draws.

$k$	Agreement	Single	Multiple	Mean size	Success
1	0.318	0.008	0.310	2.286	0.699
2	0.204	0.039	0.164	1.913	0.695
4	0.167	0.042	0.125	1.521	0.710

Table 3: Decomposition of set agreement on the original 24 planning checkpoints (Terra). Single and Multiple are disjoint contributions to Agreement by the size of the matching sets. Mean size averages both blocks; Success evaluates the selected action on the held-out block.

	Original Terra panel			Independent Terra panel
$\epsilon$	Agreement	Strict	Public	Agreement	Strict	Public
0.000	0.167	0.710	0.668	0.083	0.762	0.695
0.020	0.167	0.707	0.669	0.208	0.748	0.689
0.050	0.125	0.714	0.670	0.250	0.748	0.689
0.100	0.125	0.714	0.670	0.250	0.748	0.689

Table 4: Empirical tolerance sensitivity at

k=4

on 24 checkpoints per cohort. Strict is selected-action success on the held-out block; Public is normalized feasible benefit.

Association	Direct	Overlap	Discordant	Changed	Mean set size
	Cell relation
Independent permutation	0.042	0.417	0.542	0.125	1.813
Shared permutation	0.042	0.438	0.521	0.083	1.875
Exact keyed	0.042	0.458	0.500	0.000	1.875

Table 5: Permutation controls on 48 archive cells. “Changed” counts cells where the permuted relation differs from the exact keyed reference. Columns report median cell proportions.

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

		Joint readout			Marginal readout
Query	$N$	Correct	FP	Miss	Correct	FP	Miss
Possible image	15	15	0	0	13	2	0
Singleton	15	15	0	0	15	0	0
Hidden pairing	2	2	0	0	1	1	0

Appendix

Table 6: Exact joint-image queries on finite constructions.

N

is the number of queries; Correct, FP, and Miss are counts for each readout.

Policy	ID success			Structural OOD success
	Estimate	Lower	Upper	Estimate	Lower	Upper
Continue	0.000	0.000	0.000	0.000	0.000	0.000
Blind retry	0.268	0.232	0.304	0.250	0.213	0.287
Always verify	0.438	0.366	0.500	0.407	0.343	0.472
Always replace	0.893	0.839	0.938	0.889	0.824	0.944
Status-only	0.670	0.589	0.741	0.611	0.546	0.676

Appendix

Table 7: Central 95% prefix-cluster intervals for score-aware replay success rates. Prefix is the resampling unit; endpoints are rounded outward to three decimals.

	ID			Structural OOD
Policy	$C_{\mathrm{emp}}$	$U_{\mathrm{pol}}$	$c$	$C_{\mathrm{emp}}$	$U_{\mathrm{pol}}$	$c$
Blind retry	0.268	0.145	1.000	0.250	-0.039	1.000
Always verify	0.438	0.343	1.000	0.407	0.205	1.000
Always replace	0.893	0.780	2.000	0.889	0.753	2.000
Status-only	0.670	0.583	1.000	0.611	0.499	1.000
Trace-only	0.857	0.805	1.500	0.861	0.778	1.500

Appendix

Table 8: Frozen replay policies evaluated with

U_{\mathrm{pol}}

C_{\mathrm{emp}}

is strict completion,

U_{\mathrm{pol}}

expected replay utility, and

c

mean intervention cost over 112 ID and 108 OOD cells.

Split	Original	Fixed	Refit	Fixed regret
ID	1.000	0.375	1.000	0.293
Structural OOD	0.880	0.405	0.880	0.361

Appendix

Table 9: Observable-codebook shift on held-out replay. Agreement is with the held-out replay action; fixed and refit selectors receive the same transformed cells.

Split	Obs. rate	Obs. $n$	$B_{\mathrm{acc}}$	$B_{\mathrm{acc}}$ $n$	$B_{U}$	Floor
ID	0.500	56	0.750	84	0.804	0.002
Structural OOD	0.380	41	0.870	94	0.779	0.000

Appendix

Table 10: Collision-class ceilings and observed agreement under the five-key view. Rates and counts use separate columns;

B_{U}

is the best common-action replay-utility ceiling and Floor is its collision-regret floor.

Class	Valid	False	Valid	False	Valid	False	Valid	False	Valid	False
	$n=32$		$n=64$		$n=128$		$n=256$		$n=512$
Near tie	0.0000	0	0.0000	0	0.0000	0	0.0000	0	0.0004	0
Heterogeneous	0.0124	0	0.3084	0	0.9353	0	1.0000	0	1.0000	0
Large margin	0.0174	0	0.4054	0	0.9785	0	1.0000	0	1.0000	0

Appendix

Table 11: Finite-branch preference certification at

\delta=0.05

. Valid is the fraction of 360,000 cells with a correct unique-winner certificate; False counts incorrect certificates. Small nonzero rates may round to 0.0000; exact counts follow below.

	Product						Coupled
	1 branch			32 branches			1 branch			32 branches
Regime	OC	FP	Miss	OC	FP	Miss	OC	FP	Miss	OC	FP	Miss
Discordant	0.372	0.372	0.000	0.002	0.002	0.000	0.427	0.427	0.000	0.000	0.000	0.000
Near tie	0.514	0.269	0.101	0.472	0.255	0.128	0.624	0.311	0.033	0.470	0.220	0.095
Aligned	0.646	0.000	0.354	0.992	0.000	0.008	0.748	0.000	0.252	0.999	0.000	0.001

Appendix

Table 12: Observed compatibility under joint outcome laws with matched action marginals. Within each regime, both families use the same

N=4{,}000

latent-law specifications. The two families share replay actions, costs, marginal success probabilities, and branch budgets per action. Branch outcomes are sampled separately for the two families. All rates are proportions of these specifications. OC denotes observed compatibility; FP and Miss denote false promotion and missed overlap, respectively.

		1 branch				8 branches
Scenario	Truth	Strict	$\tau$	Set	Top-1	Strict	$\tau$	Set	Top-1
Discordant	hold	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Ambiguous	hold	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Provider	hold	0.000	0.000	0.000	0.000	0.000	0.000	0.656	0.553
Codebook	hold	0.000	0.000	0.000	0.000	0.000	0.000	0.654	0.551
Aligned	promote	0.000	0.000	0.000	0.000	0.466	0.622	0.651	0.549

Appendix

Table 13: Known-by-construction gate control. Acceptance rates use the predeclared truth; a displayed zero is

0/20{,}000

Rule	Budget	TP	FP	Miss	Joint TP	Cost	Rounds
Scalar projection	0	0.667	0.285	0.333	0.000	0.000	0.000
Typed-greedy	1	0.826	0.000	0.174	0.000	0.750	0.750
Typed-parallel	2	0.911	0.000	0.089	0.774	1.000	0.750

Appendix

Table 14: Budget feasibility for coordinate-targeted requests. TP, FP, and Miss use eligible, ineligible, and eligible denominators; Joint TP is restricted to eligible states with both coordinates outstanding. Cost and Rounds are means over all 4,000 states.

Rule	Coverage	False promotion	Mean cost
Scalar projection	0.667	0.285	0.000
Joint-coordinate	0.872	0.000	0.825
Local-coordinate	0.906	0.000	0.750
Calibrated typed	0.924	0.000	0.856
Exact typed	0.932	0.000	0.813

Appendix

Table 15: Evidence acquisition under a shared request budget of

B=1.50

. Coverage and false promotion use eligible and ineligible states as denominators, respectively; mean cost is averaged over all states. The audit covers 4,000 states with 64 paired response vectors per state for exact and fixed rules; calibrated selection uses 32 calibration and 32 independent evaluation vectors per state. Informative updates are truthful, coverage and false promotion are proportions, and evaluation cost excludes calibration acquisition.

Rule	$m$	$\|\Pi_{B}\|$	Bound	TP	Oracle TP	Regret	FP
Calibrated coverage	32	7	0.593	0.924	0.933	0.009	0.000

Appendix

Table 16: Finite-sample calibration of evidence selection. The bound uses

\delta=0.05

and the largest nonempty candidate family; TP and FP use eligible and ineligible denominators.

	1 draw			8 draws			32 draws
Estimator	TP	Request	Total	TP	Request	Total	TP	Request	Total
Pooled	0.930	0.912	1.940	0.935	0.974	9.196	0.935	0.812	33.699
By family	0.932	0.902	1.930	0.935	0.812	9.034	0.935	0.812	33.699

Appendix

Table 17: Calibration transfer to separate synthetic cases. TP is over 1,675 eligible cases; Request is mean selected-request cost over all 3,000 evaluation cases. Total adds full calibration cost divided by 3,000, with each response vector costing 4.100; the deployment cap is 1.500 and offline calibration is separately funded.

Split	$N$	Trace $U_{\mathrm{pol}}$	Oracle $U_{\mathrm{pol}}$	Trace regret
ID	112	0.735	0.753	0.019
Structural OOD	108	0.691	0.720	0.029

Appendix

Table 18: Evaluation of a frozen selector under the shifted replay law.

N

counts checkpoint cells; Trace regret is shifted-law oracle utility minus Trace-only utility.

Split	Agreement	Same	Regret
ID	1.000	1.000	0.000
Structural OOD	0.859	1.000	0.001

Appendix

Table 19: Outcome-blind replay control. Answer-, evidence-content-, and score-derived fields are removed before refitting; the reference and blind selector use the same labels and held-out cells.

Condition	Codebook acc.	One-shot FP	Robust FP	Robust miss
Link permuted	0.333	0.297	0.000	0.333
Link preserved	1.000	0.411	0.000	0.332

Appendix

Table 20: Matched-signature mechanism separation.

Key view	Action-set image	Next evidence	Disposition
Key absent	$\{(\{\textsc{retry}\},1),(\{\textsc{verify}\},1)\}$	Verify + Repeat	Hold
Key $\ell_{r}$	$\{(\{\textsc{retry}\},1)\}$	None	Preference-Cert
Key $\ell_{v}$	$\{(\{\textsc{verify}\},1)\}$	None	Preference-Cert

Appendix

Table 21: A separating source coordinate changes the singleton decision. The key-absent row reports the full pre-response request set.

	Overlap			Unique winner
Rule	Correct	Error	Miss	Correct	Error	Miss
Replay-only	4000	6000	0	2000	8000	0
Set membership	2100	3847	1900	1924	4023	76
Serialized top-1	2059	3830	1941	1919	3970	81
Status-aware set	2100	0	1900	1924	176	76
Status-aware top-1	2059	0	1941	1919	140	81

Appendix

Table 22: Post hoc reanalysis of the same 10,000 releases at 32 branches per action: 2,000 per eligible, tied, discordant, provider-gap, and observability-gap regime. Overlap and unique-winner labels have 4,000 and 2,000 positives, respectively, and include the same supplied-status checks. Correct and Error count admitted positive and negative releases; Miss counts withheld positives.

	Observed rates				$\Delta T$			$\Delta E$
Rule	$T_{\mathrm{elig}}$	$E_{\mathrm{tie}}$	$E_{\mathrm{prov}}$	$E_{\mathrm{obs}}$	Estimate	Lower	Upper	Estimate	Lower	Upper
Replay-only	1.000	1.000	1.000	1.000	-0.045	-0.055	-0.035	-0.951	-0.961	-0.940
Set membership	0.962	0.088	0.959	0.965	-0.007	-0.012	-0.003	-0.039	-0.049	-0.030
Serialized top-1	0.960	0.070	0.953	0.962	-0.004	-0.008	-0.001	-0.021	-0.029	-0.015
Status-aware set	0.962	0.088	0.000	0.000	-0.007	-0.012	-0.003	-0.039	-0.049	-0.030
Status-aware top-1	0.960	0.070	0.000	0.000	-0.004	-0.008	-0.001	-0.021	-0.029	-0.015

Appendix

Table 23: Unique-winner release operating characteristics with 32 branches per action and 2,000 independent releases per regime. Rates and paired differences are proportions.

T_{\mathrm{elig}}

denotes eligible promotion;

E_{\mathrm{tie}}

E_{\mathrm{prov}}

, and

E_{\mathrm{obs}}

denote error rates in the tied, provider-gap, and observability-gap regimes.

\Delta T

and

\Delta E

subtract each row from Full gate for eligible promotion and tied-regime error. Pointwise exact 95% interval endpoints are rounded outward.

	Replicate 1 fail	Replicate 1 success	Total
Replicate 0 fail	49	4	53
Replicate 0 success	6	85	91
Total	55	89	144

Appendix

Table 24: Paired local-replicate strict-success contingency on 144 matched branches.

Relation	Cells	Fraction	All success	Partial	All fail
Direct	1	0.021	0	1	0
Overlap	23	0.479	15	0	8
Discordant	24	0.500	13	4	7

Appendix

Table 25: One-shot endpoint relation crossed with strict action quality. Counts use all three actions per cell.

Source	Strict 144	D	O	X	Mean $\|\widehat{L}_{s}^{\mathrm{obs}}\|$	D	O	X	Mean $\|\widehat{L}_{s}^{\mathrm{obs}}\|$
		$b=1$				$b=2$
Luna	1.000	0	24	24	2.000	0	24	24	2.000
Terra	0.958	1	22	25	1.896	1	22	25	1.896
Sol	1.000	0	24	24	2.000	0	24	24	2.000
Qwen3-8B	0.632	1	23	24	1.917	2	22	24	1.875

Appendix

Table 26: Retained executions on 48 checkpoint–fault cells per source. Each source provides two executions per action (288 branches):

b=1

uses the first execution, and

b=2

pools both. Strict 144 is the fraction of the 144

b=1

branches satisfying the strict predicate. D, O, and X count cells with equal singleton sets, other nonempty overlaps, and disjoint sets, respectively.

	Sample	Strict recovery			Structure
Fault	Branches/action	Retry	Verify	Replace	Schema
Transient tool failure	24	0.667	0.667	0.667	0.986
Semantic error	24	0.583	0.583	0.583	0.944
Stale message	24	0.583	0.583	0.583	0.931
Delegation mismatch	24	0.708	0.583	0.708	0.944
All branches	96	0.635	0.604	0.635	0.951

Appendix

Table 27: Score-scrubbed local-model materialization on 12 prefixes. Branches/action is the action-column denominator (24 per fault, 96 overall); Schema uses all action rows.

Readout	Release	Hold	Reject	Agreement	Extra auth.
Replay-only	48	0	0	0.042	0.958
Set membership	24	0	24	0.542	0.458
Serialized top-1	14	0	34	0.542	0.250
Record-erased gate	0	48	0	0.458	0.000
Exact keyed gate	2	22	24	1.000	0.000

Appendix

Table 28: Terminal release decisions on 48 source-complete cells. Agreement is relative to the exact keyed observed-compatibility gate.

Action	Cells	Count	Rate	Lower	Upper
	Sample	Success		95% interval
Blind retry	16	0	0.000	0.000	0.206
Always replace	16	3	0.188	0.040	0.456
Always verify	16	16	1.000	0.794	1.000
Trace-only with verify fallback	16	16	1.000	0.794	1.000

Appendix

Table 29: Held-out non-injected error fallback under semantic cleanliness. Counts and rates use 16 cells per row; interval endpoints are rounded outward.

Quantity	Cells	Count	Rate
Observed live set is singleton	36	5	0.139
Observed live set is tied	36	31	0.861
All actions confidence-compatible	36	36	1.000
Cell has a robust expected winner	36	0	0.000
Expected-transfer rate lower endpoint	1	0	0.000
Expected-transfer rate upper endpoint	1	1	1.000

Appendix

Table 30: One-shot partial-identification audit on strict live cells. Counts and rates use 36 cells unless stated otherwise.

Source	Rule	All- failed	All- passed	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
		Equal-set cells		Exact agreement			Branch success			Selected success
Qwen3-8B	Adjusted	15	26	0.875	0.750	0.979	0.625	0.368	0.858	0.620	0.365	0.870
Terra	Adjusted	0	42	0.896	0.771	1.000	0.979	0.955	1.000	0.974	0.943	1.000
Luna	Adjusted	0	48	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Sol	Adjusted	0	48	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Qwen3-8B	Raw	15	26	0.875	0.771	0.979	0.625	0.378	0.865	0.625	0.377	0.875

Appendix

Table 31: Retained-source diagnostics on 48 cells and 288 branches per source. All-failed and All-passed count equal-set cells with all six branches failing or passing. Lower and Upper delimit descriptive 95% bootstrap intervals; resampling preserves the 12 prefix clusters.

	Luna			Terra			Sol
	Resolved		Positive	Resolved		Positive	Resolved		Positive
Policy	1	2	2	1	2	2	1	2	2
Fixed order	48	48	24	45	48	23	48	48	24
Score range	48	48	24	45	48	23	48	48	24
EC 2	48	48	24	47	48	23	48	48	24
Target-directed	48	48	24	47	48	23	48	48	24

Appendix

Table 32: Checkpoint binding retrieval on 48 cells within 12 prefixes per model. Resolved counts singleton overlap images, including the empty set; Positive counts resolved nonempty targets after two queries.

	Resolved		Positive
Policy	1	2	2
Fixed order	40	44	21
Score range	41	43	20
EC 2	44	47	23
Target-directed	42	44	21

Appendix

Table 33: Additional retrieval comparison on the Qwen3-8B archive, with 48 cells within 12 prefixes. Resolved counts singleton overlap images, including the empty set; Positive counts resolved nonempty targets after two queries.

Query rule	Stop target	1	2	1	2	2
		Image		Decision		Queries
Fixed order	Image	36	44	39	46	28
Minimax, joint	Image	41	44	43	46	23
EC 2 , eligibility first	Image	40	45	44	47	24
EC 2 , joint	Image	41	45	44	47	23
EC 2 , joint	Decision	39	42	44	47	18

Appendix

Table 34: Retrieval on all 48 Qwen3-8B cells. Image counts identified complete

(I_{s}^{\mathrm{obs}},D_{\kappa})

values; Decision counts identified Continue or ineligible dispositions. An unresolved Hold is not an identified disposition. Headers 1 and 2 are per-cell query budgets; Queries totals lookups at budget two. Initially, 32 images and 34 dispositions are identified.

		Strict success			Normalized benefit
Policy	Calls	Estimate	Lower	Upper	Estimate	Lower	Upper
Fixed retry	1	0.677	0.552	0.802	0.848	0.763	0.922
Fixed replacement	1	0.688	0.562	0.802	0.887	0.825	0.945
Uniform	1	0.712	0.608	0.806	0.882	0.841	0.920
Fixed verification	1	0.771	0.656	0.875	0.912	0.853	0.963
Selected, $k=1$	4	0.699	0.591	0.798	0.872	0.824	0.913

Appendix

Table 35: Held-out strict success and normalized public benefit on the original Terra planning cohort. Calls include selection and one final action; each metric has a descriptive 95% checkpoint-bootstrap interval.

	Public			Strict
$k$	Estimate	Lower	Upper	Estimate	Lower	Upper
1	0.318	0.216	0.424	0.378	0.268	0.492
2	0.204	0.119	0.303	0.262	0.161	0.377
4	0.167	0.042	0.333	0.250	0.083	0.417

Appendix

Table 36: Exact best-set agreement across selection and evaluation blocks on the original Terra planning cohort. Public uses the observable score; Strict uses evaluator-only optimality labels.

			Strict-success difference
$k$	Calls	Mean tokens	Estimate	Lower	Upper
1	4	89997.928	-0.072	-0.142	-0.001
2	7	157445.139	-0.076	-0.156	0.001
4	13	292310.155	-0.061	-0.160	0.035

Appendix

Table 37: Selected-policy contrast and expected input-plus-output tokens per decision on the original Terra planning cohort. Differences are relative to fixed verification; intervals are paired strict-success differences.

	Agreement			Success
$k$	Estimate	Lower	Upper	Estimate	Lower	Upper
1	0.404	0.307	0.503	0.765	0.677	0.844
2	0.238	0.148	0.341	0.754	0.665	0.835
4	0.083	0.000	0.208	0.762	0.665	0.847

Appendix

Table 38: Independent replication on 24 new planning checkpoints. Agreement uses public-score sets; Success evaluates uniform selection from those sets on the held-out block. Each metric has a descriptive 95% checkpoint-bootstrap interval.

		$k=1$			$k=2$			$k=4$
Cohort	Model	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
Study	Terra	0.318	0.221	0.424	0.204	0.122	0.306	0.167	0.042	0.333
Study	Luna	0.234	0.164	0.312	0.150	0.109	0.199	0.083	0.000	0.208
Study	Sol	0.424	0.341	0.503	0.331	0.242	0.428	0.167	0.042	0.333
Replication	Terra	0.404	0.307	0.503	0.238	0.148	0.343	0.083	0.000	0.208
Replication	Luna	0.299	0.224	0.388	0.212	0.139	0.303	0.250	0.083	0.417

Appendix

Table 39: GPT-5.6 family exact public-score set agreement. Each entry is Estimate, Lower, or Upper from 10,000 checkpoint-bootstrap resamples stratified by task size. The six panels contain 24 checkpoints each.

		$k=1$			$k=2$			$k=4$			Fixed
Cohort	Model	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
Study	Terra	0.699	0.588	0.795	0.695	0.581	0.794	0.710	0.590	0.816	0.771	0.656	0.865
Study	Luna	0.575	0.469	0.674	0.568	0.460	0.669	0.538	0.417	0.660	0.635	0.510	0.750
Study	Sol	0.722	0.655	0.790	0.729	0.668	0.794	0.750	0.684	0.819	0.729	0.646	0.823
Replication	Terra	0.765	0.678	0.844	0.754	0.666	0.835	0.762	0.665	0.847	0.771	0.667	0.865
Replication	Luna	0.727	0.644	0.803	0.728	0.646	0.804	0.719	0.630	0.802	0.760	0.656	0.854

Appendix

Table 40: GPT-5.6 family selected strict success on held-out draws. Each entry is Estimate, Lower, or Upper from the same checkpoint-bootstrap procedure. Fixed verification is a one-call baseline within the corresponding model–cohort panel.

$k$	Agreement	Single	Multiple	Mean size	Success	Calls
1	0.318	0.008	0.310	2.286	0.699	4
2	0.204	0.039	0.164	1.913	0.695	7
4	0.167	0.042	0.125	1.521	0.710	13

Appendix

Table 41: Post hoc decomposition of observed public-score set agreement on the original 24 planning checkpoints. Single and Multiple are disjoint contributions to Agreement, according to the size of the matching sets. Mean size averages both blocks. Success evaluates the sampled-score action mixture on the held-out block; fixed verification achieves 0.771 with one call. Selection requires

3k+1

calls.

		Agreement			Semantic success			Strict success
Cohort	$k$	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
Original	1	0.359	0.250	0.479	0.725	0.614	0.827	0.697	0.592	0.796
Original	2	0.221	0.126	0.339	0.713	0.600	0.819	0.686	0.577	0.788
Original	4	0.125	0.000	0.250	0.700	0.580	0.812	0.679	0.562	0.786
Independent	1	0.477	0.365	0.594	0.796	0.706	0.877	0.764	0.678	0.843
Independent	2	0.325	0.203	0.459	0.786	0.691	0.871	0.753	0.664	0.834

Appendix

Table 42: Selection using recomputed feasible benefit. Agreement compares complete maximizer sets; Semantic and Strict evaluate the selected action mixture on the held-out block. Each cohort contains 24 checkpoints. Lower and Upper are the endpoints of descriptive 95% checkpoint-bootstrap intervals.

		Original Terra panel						Independent Terra panel
$\epsilon$	$k$	Agree	Common	Jacc.	Size	Strict	Public	Agree	Common	Jacc.	Size	Strict	Public
0.000	1	0.318	0.898	0.619	2.286	0.699	0.662	0.404	0.940	0.693	2.411	0.765	0.699
0.000	2	0.204	0.755	0.472	1.913	0.695	0.661	0.238	0.863	0.557	2.122	0.754	0.695
0.000	4	0.167	0.458	0.299	1.521	0.710	0.668	0.083	0.750	0.375	1.833	0.762	0.695
0.020	1	0.474	0.948	0.727	2.510	0.702	0.665	0.458	0.982	0.759	2.578	0.756	0.695
0.020	2	0.358	0.919	0.660	2.361	0.704	0.663	0.346	0.958	0.680	2.438	0.758	0.695

Appendix

Table 43: Complete empirical-tolerance sensitivity. Agree is exact-set agreement, Common is nonempty-intersection frequency, Jacc. is Jaccard overlap, and Size averages set cardinality across both blocks. Strict and Public evaluate the selected mixture on the held-out block. Each cohort contributes 24 checkpoints.

		Agreement change			Strict-success change			Public-benefit change
Cohort	$\epsilon$	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
Original	0.000	-0.151	-0.281	-0.005	0.011	-0.033	0.057	0.006	-0.021	0.034
Original	0.020	-0.307	-0.451	-0.156	0.005	-0.020	0.028	0.004	-0.014	0.020
Original	0.050	-0.451	-0.573	-0.315	0.004	-0.015	0.023	0.002	-0.009	0.013
Original	0.100	-0.471	-0.589	-0.341	0.004	-0.014	0.023	0.003	-0.008	0.013
Independent	0.000	-0.320	-0.406	-0.237	-0.003	-0.029	0.023	-0.005	-0.025	0.013

Appendix

Table 44: Paired changes with descriptive 95% percentile intervals. Each contrast compares

k=4

with

k=1

at the indicated empirical tolerance.

	Draws per action			Change
Metric	1	2	4	Estimate	Lower	Upper
Exact set	0.318	0.204	0.167	-0.151	-0.281	-0.005
Common action	0.898	0.755	0.458	-0.440	-0.607	-0.271
Jaccard	0.619	0.472	0.299	-0.320	-0.428	-0.197
Uniform choice	0.318	0.300	0.194	-0.123	-0.206	-0.027
Size reference	0.336	0.230	0.264	-0.072	-0.180	0.024

Appendix

Table 45: Alternative agreement measures on 24 checkpoints. Change is

k=4

minus

k=1

; Lower and Upper are descriptive 95% percentile bounds from 10,000 bootstrap samples stratified by checkpoint size. All comparisons remain paired within checkpoint.

	Agreement			Change
Outcome	$k=1$	$k=2$	$k=4$	Estimate	Lower	Upper
Strict	0.840	0.811	0.854	0.014	-0.135	0.167
Strict, cost-adjusted	0.889	0.859	0.896	0.007	-0.117	0.112
Answer	0.840	0.811	0.854	0.014	-0.135	0.167
Answer, cost-adjusted	0.889	0.859	0.896	0.007	-0.117	0.112

Appendix

Table 46: Complete-set agreement on 12 Qwen3-8B checkpoints. Strict requires both the correct answer and a matching reported score; Answer requires only the correct answer. Cost-adjusted rows subtract the fixed penalties 0.05, 0.05, and 0.10 for retry, verify, and replacement. Change is

k=4

minus

k=1

, with separate 95% bootstrap bounds.

		Agreement			All-zero pairs			Held-out success
Model	$k$	Estimate	Lower	Upper	Estimate	Lower	Upper	Estimate	Lower	Upper
Luna	1	0.854	0.729	0.958	0.854	0.729	0.958	0.014	0.000	0.035
Luna	2	0.750	0.583	0.917	0.750	0.583	0.917	0.014	0.000	0.035
Luna	4	0.667	0.417	0.917	0.583	0.333	0.833	0.014	0.000	0.035
Terra	1	0.776	0.583	0.943	0.776	0.583	0.943	0.045	0.007	0.090
Terra	2	0.688	0.458	0.896	0.688	0.458	0.896	0.035	0.007	0.069

Appendix

Table 47: RecoveryBench set-readout intervals. Each entry is Estimate, Lower, or Upper from 10,000 paired checkpoint-bootstrap resamples. Bootstrap resampling is over the 12 checkpoints and preserves all model, action, and repeat observations within a checkpoint. Held-out success is the binary native-reward outcome averaged over the selected action set.

Policy	Estimate	Lower	Upper
Agreement-guided, $k=1$	0.013	0.000	0.028
Agreement-guided, $k=2$	0.030	0.003	0.066
Outcome-mean direct	0.065	0.000	0.171
Fixed verify, Luna	0.000	0.000	0.000
Fixed verify, Terra	0.021	0.000	0.063
Fixed verify, Sol	0.146	0.000	0.313

Appendix

Table 48: Descriptive decision-value analysis on the frozen RecoveryBench panel. Selection uses repeats 0–3 and evaluation uses repeats 4–7. Each entry is the held-out success Estimate, Lower, or Upper from 10,000 task-bootstrap resamples.

Execution status	Episodes	Reward-positive	Positive rate
Native verifier observed	364	46	0.126
Model protocol failure	487	1	0.002
Invalid adapter attempt	13	0	0.000
All episodes	864	47	0.054

Appendix

Table 49: Execution provenance for all 864 RecoveryBench episodes. Positive rate is the number of reward-positive official receipts divided by the number of episodes in the row.

Model	$k$	Estimate	Lower	Upper
Luna	1	-0.443	-1.000	-0.094
Luna	2	-0.443	-1.000	-0.094
Luna	4	-0.632	-1.000	-0.357
Terra	1	-0.783	-1.000	-0.283
Terra	2	-0.741	-1.000	-0.266
Terra	4	-0.529	-1.000	-0.255

Appendix

Table 50: Checkpoint-level rank concordance between exact-set agreement and held-out success on the 12 RecoveryBench checkpoints. Each entry is Estimate, Lower, or Upper from 10,000 checkpoint-bootstrap resamples; resamples with a constant vector are omitted.

Explore similar work

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
Jun 1, 2026Laksh AdvaniAgentic Benchmarks Traces
University of Colorado
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jul 10, 2026Jiale Liu, Huajun Xi, Shaokun Zhang +6Artificial Intelligence Agents Large-Scale Benchmark
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
Jun 4, 2026Dongsheng Zhu, Xuchen Ma, Yucheng Shen +5Large Language Model Agents Fault Tolerance
Shanghai AI Laboratory · East China Normal University · Soochow University +2

Jun 1, 2026, cs.LGJ/K move · Enter open · S save

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

Laksh Advani

University of Colorado

Agentic Benchmarks Traces Closure

Open paper AskPDF arXiv

LLM agents can fail silently by asserting task completion when the environment state shows otherwise. We study this failure mode, false success, across two agent benchmarks: 9,876 tau2-bench trajectories from 8 model families and 1,879 AppWorld trajectories from 4 model families with text-independent ground truth. False success is common but varies by setting: 45--48% of failures in single-control tau2-bench domains, 3% in dual-control telecom, and 75.8% among AppWorld self-assessing coding-agent trajectories with explicit status claims. LLM judges fail reliably: no configuration across 5 judges, 5 prompt strategies, and full task specifications exceeds AUROC 0.65 on tau2-bench, and the same judges reach only 0.54 AUROC on AppWorld API-call traces. Judges rely on surface completion proxies -- confident closing language in tau2-bench and coarse action-sequence volume in AppWorld -- rather than verified state changes. Lightweight TF-IDF detectors achieve task-disjoint AUROC 0.83 on tau2-bench and 0.95 on AppWorld, recovering 4--8x more false successes than the best judge at the same flag rate with 3,300x lower latency. These results suggest that production monitoring should use lightweight, domain-calibrated detectors as triage signals rather than relying on LLM judges as the primary monitor for false success.

Abstract

Explore similar work

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents