Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents

Ashutosh Hathidara

cs.AIOct 13, 2026

Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents

Authors: Ashutosh Hathidara

Organizations: SAP

Abstract

Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human. That decision needs a per-step confidence signal predicting whether the current rollout will eventually succeed. The agent's own self-reported verbalized confidence is the natural candidate, but existing reinforcement learning methods calibrate it against the correctness of a decision at the current step. We propose a method to calibrate it against the eventual outcome of a multi-step episode: under GRPO, the group of rollouts for a task supplies success labels, and a proper scoring rule scores each step's confidence against the realized outcome. In offline RL on recorded trajectories of a Qwen2.5-7B agent in ALFWorld, TextWorld, and WebShop environments, the AUROC of step confidence against episode outcome rises from near chance to 0.68-0.85, and the top-10% of steps the agent is most confident about succeed ~4.3x more often than average. In online RL on the live ALFWorld agent, four calibration rewards improve early-step AUROC, expected calibration error goes from 0.42 to below 0.12, and selective execution improves over both the untrained agent and a format-only RL control, without degrading task success. We also introduce an evaluation protocol that separates episode depth, task difficulty, and attempt-level foresight, which shows where the gain lies: the trained agent recognizes from its first step which tasks it will fail, while telling a succeeding attempt from a failing one at the same task is harder, with some early-window signal emerging on TextWorld.

Figures & tables

Figure 1: Outcome-grounded calibration under GRPO. The

G

attempts at a task end in outcomes

Y

that label every one of their steps. Each step’s confidence

c

is scored against that label by a proper scoring rule (Eq. 3 ), and the group-relative advantage credits the step for both the outcome and the quality of its confidence before updating

\pi_{\theta}

	Pooled AUROC		Tail success lift
Environment	Untrained	Trained	Untrained	Trained
ALFWorld	0.517	0.681	1.55 $\times$	4.33 $\times$
TextWorld	0.578	0.722	1.57 $\times$	4.35 $\times$
WebShop	0.462	0.850	1.11 $\times$	4.35 $\times$

Table 1: Offline training results comparing base instruction tuned vs RL trained Qwen2.5-7B-Instruct model on held-out single-trajectory Qwen2.5-7B-Instruct recordings: pooled step-level AUROC, and the tail success lift (success rate in the top-10% confident samples vs base success rate). Bootstrap CIs in Appendix G .

Reward	Type	AUROC [95% CI]	Format
Spherical	proper	0.882 [.837,.921]	1.00
Brier	proper	0.864 [.822,.903]	1.00
Ranking	order	0.845 [.791,.894]	1.00
L1	improper	0.839 [.788,.887]	1.00
Logarithmic	proper	0.828 [.770,.880]	1.00
Decision	–	0.820 [.773,.863]	1.00

Table 2: Offline training results on WebShop environment across different Calibration reward forms. Pooled AUROC with 95% step-level bootstrap intervals; “Format" is the fraction of samples that correctly predicts the tagged confidence tokens.

Estimator	Seed 0	Seed 1
GRPO	0.613 [.529,.696]	0.676 [.598,.749]
RLOO	0.801 [.739,.856]	0.876 [.825,.922]
PPO, value critic	0.500	0.500

Table 3: Offline training results on WebShop environment across different advantage estimator under a matched ranking + spread reward (

w{=}0.10

). Pooled AUROC with 95% step-level bootstrap intervals; PPO with a learned value critic fails to learn parsable response format across both seeds.

Arm	success	value	$\Delta$	(all steps)	(all steps)	value	$\Delta$
	Episode	Pooled AUROC, $k\leq 2$		ECE	Brier	Top-10% success, $k\leq 2$
Untrained	0.201	0.689 [.640,.733]	–	0.419 [.398,.438]	0.250 [.237,.264]	0.493 [.323,.675]	–
Format-only	0.188	0.708 [.660,.749]	$+0.019$ [-.010,.051]	0.429 [.408,.447]	0.253 [.239,.267]	0.465 [.296,.626]	$-0.024$ [-.094,.037]
Brier+ $\varphi$	0.194	0.761 [.716,.798]	$\mathbf{+0.072}$ [.026,.112]	0.115 [.105,.124]	0.052 [.045,.061]	0.561 [.391,.748]	$+0.080$ [-.000,.185]
Balanced Brier	0.197	0.765 [.721,.805]	$\mathbf{+0.076}$ [.038,.111]	0.114 [.102,.125]	0.054 [.046,.064]	0.609 [.413,.765]	$\mathbf{+0.115}$ [.035,.193]
Brier+dispersion	0.189	0.790 [.742,.828]	$\mathbf{+0.101}$ [.069,.134]	0.054 [.050,.059]	0.044 [.034,.054]	0.635 [.426,.804]	$\mathbf{+0.159}$ [.054,.224]

Table 4: Online training results for Qwen2.5-7B-Instruct on ALFWorld. CIs are 95% task-level cluster bootstraps,

\Delta

= (trained - base instruction tuned) on the shared tasks; early-window ECE, Brier, top-25% success, and the ECE and Brier deltas are in Appendix F .

	Pooled	Between-task	Within-task
Arm	$k{=}0$	$\Delta$ , $k\leq 2$	$k\leq 2$ ( $n$ mixed)	$\Delta$ , $k\leq 2$ ( $n$ paired)	$\Delta$ , all steps	$\Delta$ depth-matched, all
Untrained	0.562 [.530,.600]	–	0.606 (20) [.515,.696]	–	–	–
Format-only	0.568 [.534,.602]	$+0.033$ [-.004,.082]	0.571 (15) [.484,.657]	$-0.054$ (8) [-.118,.018]	$\mathbf{-0.077}$ [-.114,-.045]	$+0.013$ [-.080,.133]
Brier+ $\varphi$	0.642 [.584,.691]	$+0.037$ [-.017,.085]	0.585 (20) [.483,.680]	$-0.059$ (12) [-.184,.059]	$-0.013$ [-.056,.029]	$+0.027$ [-.059,.124]
Balanced Brier	0.697 [.637,.744]	$+0.032$ [-.019,.077]	0.657 (19) [.582,.733]	$-0.026$ (11) [-.128,.055]	$-0.038$ [-.082,.005]	$+0.052$ [-.034,.143]
Brier+dispersion	0.700 [.634,.763]	$\mathbf{+0.072}$ [.029,.115]	0.661 (15) [.568,.757]	$-0.092$ (7) [-.262,.062]	$-0.050$ [-.102,.004]	$+0.035$ [-.060,.140]

Table 5: Decomposition of the online training gain by estimand and step window on ALFWorld. Columns move left to right from no confound removed to difficulty and depth both removed; values are absolute AUROC except where marked

\Delta

(trained minus untrained). Within-task columns use only tasks with both a succeeding and a failing attempt: the absolute cell averages per-task AUROC over that arm’s

n

mixed-outcome tasks (

n

mixed), and each

\Delta

resamples the tasks mixed in both the trained and untrained arm (

n

paired); depth-matched is concordance over same-task, same-step-index pairs across all steps. Bold = paired 95% CI excludes 0.

Figure 2: Comparison of reliability over all steps of untrained 7B agent against two online RL trained agents on live ALFWorld. Marker area

\propto

bin mass; same setting as of Figure 3 ).

Figure 3: Comparison of Risk–Coverage of different live agents on ALFWorld: for the early window (

k\leq 2

, top) and all steps (bottom).

Figure 4: Online training results on TextWorld environment: paired within-task AUROC gain (trained

-

untrained) by step window; 95% CIs over shared held-out tasks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Estimand	7B (128 tasks)	14B (128 tasks)
Pooled, all steps	0.740 [.704,.776]	0.698 [.650,.746]
Depth-only baseline, all steps	0.817 [.777,.857]	0.755 [.722,.792]
Pooled, $k\leq 2$	0.689 [.640,.733]	0.722 [.667,.774]
Depth-only baseline, $k\leq 2$	0.499	0.500
Pooled, step 0	0.562 [.530,.600]	0.599 [.532,.677]
Between-task, $k\leq 2$	0.825 [.758,.881]	0.830 [.748,.896]

Appendix

Table 6: Untrained Qwen2.5-7B-Instruct agents on live ALFWorld: AUROC of step confidence against eventual episode success under every estimand of § 3.2 .

Figure 5: Qwen2.5-7B-Instruct on live ALFWorld: depth-only baseline, untrained agent, and the Brier+

\varphi

RL comparison. Bars are 95% task-level bootstrap intervals.

Arm	Succ.	Pooled all	Pooled $k\leq 2$	Pooled $t{=}0$	Between $k\leq 2$	Within all ( $n$ )	Within $k\leq 2$ ( $n$ )	Within $t{=}0$	Depth-matched
Untrained	0.201	0.740	0.689	0.562	0.825	0.651 (20)	0.606 (20)	0.529	0.633
		[.704,.776]	[.640,.733]	[.530,.600]	[.758,.881]	[.598,.704]	[.515,.696]	[.432,.620]	[.565,.715]
Format-only	0.188	0.754	0.708	0.568	0.858	0.665 (15)	0.571 (15)	0.507	0.646
		[.712,.792]	[.660,.749]	[.534,.602]	[.792,.911]	[.601,.727]	[.484,.657]	[.393,.610]	[.575,.739]
Brier+ $\varphi$	0.194	0.799	0.761	0.642	0.862	0.673 (20)	0.585 (20)	0.500	0.659
		[.753,.838]	[.716,.798]	[.584,.691]	[.798,.911]	[.615,.733]	[.483,.680]	[.378,.626]	[.607,.719]

Appendix

Table 7: Absolute values for every online RL for Qwen2.5-7B-Instruct experiment. Succ. = episode success rate; intervals are 95% task-level cluster bootstraps

Arm	Pooled all	Pooled $k\leq 2$	Between $k\leq 2$	Within all ( $n$ )	Within $k\leq 2$ ( $n$ )	Depth-matched
Format-only	$+0.014$	$+0.019$	$+0.033$	$\mathbf{-0.077}$ (8)	$-0.054$ (8)	$+0.013$
	[-.019,.054]	[-.010,.051]	[-.004,.082]	[-.114,-.045]	[-.118,.018]	[-.080,.133]
Brier+ $\varphi$	$\mathbf{+0.058}$	$\mathbf{+0.072}$	$+0.037$	$-0.013$ (12)	$-0.059$ (12)	$+0.027$
	[.014,.103]	[.026,.112]	[-.017,.085]	[-.056,.029]	[-.184,.059]	[-.059,.124]
Balanced Brier	$+0.041$	$\mathbf{+0.076}$	$+0.032$	$-0.038$ (11)	$-0.026$ (11)	$+0.052$
	[-.008,.084]	[.038,.111]	[-.019,.077]	[-.082,.005]	[-.128,.055]	[-.034,.143]

Appendix

Table 8: Paired deltas (trained - untrained) for every online RL Qwen2.5-7B-Instruct experiment; same setting as Table 7 . Pooled and between-task deltas resample the shared evaluation tasks; within-task deltas resample the

n

tasks mixed in both arms.

Figure 6: Pooled, between-task, and within-task AUROC in the early window (

k\leq 2

) for the untrained Qwen2.5-7B-Instruct agent and four calibration experiments (same setting as Table 4 ). Bars are 95% task-level bootstrap intervals.

Figure 7: Paired deltas of AUROC (trained - untrained) in the early window (

k\leq 2

), pooled (squares) and within-task (circles), with 95% task-level bootstrap intervals.

Arm	rate	rate	$k\leq 2$	all steps
	Win	Emit	Within-task AUROC
Untrained	0.146	0.971	0.476 [.433,.526]	0.697 [.668,.727]
Brier+ $\varphi$	0.262	0.979	0.506 [.464,.548]	0.668 [.633,.703]
Format-only	0.172	0.744	0.517 [.488,.545]	0.684 [.647,.720]

Appendix

Table 9: Online RL training results on TextWorld: win and emission rates over held-out tasks, and within-task AUROC by window. The paired early-window gain is in Figure 4 .

Arm	Mean conf	ECE all	ECE $k\leq 2$	Brier all	Brier $k\leq 2$	Top-10% $k\leq 2$	Top-25% $k\leq 2$	Top-10% all	Top-5% all
	all / $k\leq 2$
Untrained	0.475 / 0.477	0.419	0.277	0.250	0.229	0.493	0.363	0.241	0.241
		[.398,.438]	[.217,.330]	[.237,.264]	[.212,.248]	[.323,.675]	[.241,.475]	[.151,.327]	[.151,.333]
Format-only	0.474 / 0.476	0.429	0.290	0.253	0.229	0.465	0.363	0.193	0.182
		[.408,.447]	[.236,.347]	[.239,.267]	[.212,.248]	[.296,.626]	[.220,.464]	[.117,.272]	[.114,.275]
Brier+ $\varphi$	0.161 / 0.189	0.115	0.051	0.052	0.137	0.561	0.444	0.303	0.491

Appendix

Table 10: Calibration and selective execution for every online training on Qwen2.5-7B-Instruct experiment; same setting as Table 7 . ECE uses 10 equal-width bins, top-

q

% is the success rate among the

q

% most confident steps in the window, and intervals are 95% task-level cluster bootstraps.

Arm	ECE all	ECE $k\leq 2$	Brier all	Brier $k\leq 2$	Top-10% $k\leq 2$	Top-25% $k\leq 2$	Top-10% all	Triage top-50%	Task AUROC
Format-only	$+0.010$	$+0.013$	$+0.003$	$-0.001$	$-0.024$	$-0.012$	$\mathbf{-0.047}$	$-0.008$	$+0.076$
	[-.001,.021]	[-.005,.030]	[-.005,.010]	[-.008,.006]	[-.094,.037]	[-.050,.036]	[-.084,-.012]	[-.034,.018]	[-.013,.176]
Brier+ $\varphi$	$\mathbf{-0.304}$	$\mathbf{-0.226}$	$\mathbf{-0.198}$	$\mathbf{-0.093}$	$+0.080$	$\mathbf{+0.081}$	$\mathbf{+0.061}$	$-0.010$	$-0.008$
	[-.318,-.286]	[-.271,-.147]	[-.213,-.180]	[-.126,-.057]	[-.000,.185]	[.038,.160]	[.018,.128]	[-.026,.008]	[-.127,.111]
Balanced Brier	$\mathbf{-0.305}$	$\mathbf{-0.224}$	$\mathbf{-0.196}$	$\mathbf{-0.090}$	$\mathbf{+0.115}$	$\mathbf{+0.076}$	$\mathbf{+0.050}$	$-0.013$	$+0.015$
	[-.319,-.288]	[-.273,-.126]	[-.213,-.180]	[-.125,-.054]	[.035,.193]	[.042,.155]	[.007,.111]	[-.034,.010]	[-.092,.124]

Appendix

Table 11: Paired deltas (trained - untrained) of the calibration and selective-execution metrics; same setting as Table 7 . Task AUROC = AUROC of mean early confidence against “task solved at least once”.

	Qwen2.5-7B-Instruct	Qwen2.5-14B-Instruct
Mean conf, all / $k\leq 2$	0.475 / 0.477	0.534 / 0.646
Step success, all / $k\leq 2$	0.057 / 0.201	0.112 / 0.289
ECE, all	0.419 [.398,.438]	0.425 [.388,.458]
ECE, $k\leq 2$	0.277 [.217,.330]	0.357 [.296,.417]
Brier, all	0.250 [.237,.264]	0.303 [.281,.328]
Brier, $k\leq 2$	0.229 [.212,.248]	0.310 [.283,.336]

Appendix

Table 12: Calibration and selective execution of the untrained Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct agents on the held-out set. Intervals are 95% task-level cluster bootstraps.

$w$	(per seed)	$[0.2,0.8]$	$\geq 0.8$	(2 seeds)
	AUROC, sampled	Mass in	Mass	AUROC, greedy
0.00	0.746 (0.717, 0.775)	0.64	0.34	0.720 / 0.762
0.05	0.678 (0.679, 0.676)	0.76	0.23	0.690 / 0.698
0.10	0.675 (0.675)	0.74	0.25	0.663 / 0.676
0.15	0.584 (0.594, 0.574)	0.88	0.12	0.593 / 0.597
0.30	0.556 (0.587, 0.525)	0.93	0.08	0.568 / 0.520

Appendix

Table 13: Spread-weight sweep, WebShop offline RL training, ranking reward, evaluated on the held-out prefixes of Table 2 ; sampled columns use

K=8

decodes per prefix at temperature

T=1.0

, top-

p

0.95, seed-averaged (

w=0.10

has one sampled seed), and the interior mass column is the closed interval

[0.2,0.8]

(so

c=0.8

is counted in both mass columns).

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 249 papers

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
Jul 14, 2026Yaopei Zeng, Congchao Wang, JianHang Chen +3AI Agent Reliability LLM Agent Evaluation
Pennsylvania State University · Virginia Tech · Purdue University +1
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
Oct 6, 2026Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti +2Probability Calibration Confidence Estimation in Language Models
University of California, Santa Cruz · Megagon Labs
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside

Jul 14, 2026J/K move · Enter open · S save

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Yaopei Zeng, Congchao Wang, JianHang Chen, Nan Wang+2

Pennsylvania State University · Virginia Tech · Purdue University+1

AI Agent Reliability LLM Agent Evaluation LLM Uncertainty Estimation LLM Agent Reliability Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

LLM agents operate in stateful environments, where a single erroneous step can waste limited interaction budget or cause irreversible effects before task failure becomes apparent. Reliable deployment therefore requires step-level confidence estimation: estimating, before execution, the probability that a proposed action will advance the task. Existing LLM confidence estimators are typically designed for static question answering under a fixed task context and evaluation criterion. For an agent, however, its action productivity depends on an environment transition that is observed only after execution. To address this challenge, we introduce Critic Experience Bank (CEB), a training-free framework that turns feedback from completed trajectories into reusable evidence for future confidence judgments. After each trajectory, an LLM assigns hindsight productivity pseudo-labels to individual actions and stores them with the critic's original pre-execution confidence, task context, action, and observed feedback. For a new action, by retrieving related productive and unproductive experiences, CEB grounds pre-execution confidence in feedback from completed trajectories to condition a fixed LLM critic. CEB thereby adapts over a task stream without parameter updates or ground-truth step labels at deployment. Across four agent benchmarks spanning offline and live web navigation, mobile GUI and shell tasks, and three critic backbones, CEB achieves the best or tied-best ECE, Brier score, and AUC in all twelve benchmark-backbone settings under rule-based step labels, reducing ECE by up to 53.8% relative to the strongest training-free baseline. Its confidence scores also improve downstream utility in selective execution and simulated task success.

Abstract

Confidence Estimation in Language Models

Explore similar work

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents

Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL