Free Lunch for LLMs Confidence: The RL Critic Already Knows

Shujin Wu; Heng Ji

cs.CLOct 22, 2026

Free Lunch for LLMs Confidence: The RL Critic Already Knows

Authors: Shujin Wu, Heng Ji

Organizations: University of Illinois Urbana-Champaign

Abstract

Reliable confidence estimation is essential for trustworthy deployment of large language models (LLMs), but dedicated confidence training stage requires additional data and compute. We ask a fundamental question: what prediction problem does confidence training actually solve? Inspired by prior work on external confidence estimation, we compare self- and external estimators of comparable capacity trained on the same (question, response, correctness) triples. We show that an external model trained solely on a policy's behavioral traces can estimate the policy's confidence as accurately as the policy itself, without accessing the policy's internal representations. This points to a deeper connection with value function learning in reinforcement learning (RL): under a correctness-based outcome, we prove that confidence estimation and critic training are equivalent, both predicting the likelihood that a policy's response will be correct from its trajectory. A simple consequence follows: for LLMs trained with value-based RL, the existing critic can be reused as an off-the-shelf confidence estimator with no additional training or data. The policy remains unchanged, so its task performance is preserved, and the critic provides confidence throughout generation, including intermediate reasoning steps. Across math, code, tool calling, and general reasoning spanning a dozen domains, the reused critic is the best confidence estimator on every in-distribution split and the best-calibrated on 14 of 16, without hurting the task performance.

Figures & tables

Estimator	GSM8K	Numina	ARC-C	APPS	TACO
AUC $\uparrow$
Self	0.71	0.83	0.67	0.84	0.89
Qwen-7B	0.75	0.83	0.63	0.82	0.80
Mistral-7B	0.71	0.68	0.58	0.86	0.90
ECE $\downarrow$
Self	0.21	0.08	0.07	0.05	0.06

Table 1: Confidence estimation on the responses from Llama-3-8B-Instruct using self and external estimators.

		Math		Code		Tool Calling		General
	Method	ID	OOD	ID	OOD	ID	OOD	ID	OOD	Avg.
	R-Tuning	38.5	70.5	49.7	57.5	47.2	59.5	26.1	54.0	50.4
	Original Policy	43.6	73.4	51.6	58.2	52.2	59.1	33.8	62.6	54.3
	Maj@8	43.9	78.3	53.7	60.1	52.3	60.5	35.5	67.1	56.4
	CaC Selection (ours)	44.1	80.3	55.2	63.4	52.3	62.1	38.7	71.2	58.4
Qwen3-8B	Best@8 (oracle)	45.7	86.7	60.7	69.4	52.5	64.4	44.8	74.5	62.3

Table 2: Task accuracy (%) with and without inference-time scaling. Rows above the thin rule use a single response; rows below sample

N{=}8

responses, where CaC Selection returns the one with the highest critic-estimated confidence and Best@8 is the oracle upper bound.

	Math						Code						Tool Calling
	ID			OOD			ID			OOD			ID			OOD
Method	AUC	ECE	Brier	AUC	ECE	Brier	AUC	ECE	Brier	AUC	ECE	Brier	AUC	ECE	Brier	AUC	ECE	Brier
Qwen3-8B
Verbalized	0.66	0.56	0.53	0.73	0.19	0.21	0.60	0.37	0.38	0.61	0.28	0.26	0.47	0.37	0.39	0.57	0.30	0.33
Self-consistency	0.70	0.44	0.42	0.94	0.16	0.06	0.78	0.22	0.23	0.89	0.10	0.10	0.71	0.38	0.36	0.91	0.30	0.30
R-Tuning	0.89	0.05	0.13	0.88	0.24	0.18	0.86	0.05	0.15	0.85	0.08	0.11	0.95	0.07	0.08	0.90	0.19	0.16

Table 3: Confidence estimation performance on math, code and agentic tool calling for two PPO-trained LLMs. AUC is AUROC (higher is better), ECE and Brier are lower-is-better. ID and OOD values are averages over the sub-benchmarks of each split weighted by the number of scored samples (per-benchmark results in Appendix D ).

		In distribution									Out of distribution
	Method	Phys.	Math	Bus.	Chem.	Fin.	Econ.	Hist.	Bio.	All	Phys.	Bio.	Med.	Multi	Chem.	All
Qwen3-8B
AUC $\uparrow$	Verbalized	0.59	0.55	0.52	0.52	0.53	0.53	0.36	0.48	0.55	0.69	0.52	0.60	0.56	0.55	0.59
	Self-consistency	0.79	0.80	0.64	0.76	0.75	0.82	0.77	0.73	0.76	0.84	0.71	0.82	0.81	0.76	0.79
	R-Tuning	0.77	0.75	0.63	0.74	0.68	0.67	0.84	0.77	0.73	0.87	0.81	0.70	0.64	0.61	0.73
	Probe	0.81	0.76	0.63	0.76	0.68	0.66	0.87	0.78	0.75	0.82	0.71	0.67	0.62	0.59	0.68

Table 4: Confidence estimation performance on general reasoning benchmarks spanning multiple domains. LLMs are trained with PPO on WebInstruct-verified.

	ID			OOD
Method	AUC	ECE	Brier	AUC	ECE	Brier
Verbalized	0.53	0.60	0.58	0.55	0.28	0.30
Self-consistency	0.67	0.41	0.41	0.78	0.18	0.21
R-Tuning	0.87	0.05	0.14	0.79	0.25	0.24
Probe	0.86	0.03	0.14	0.74	0.17	0.22
CaC (ours)	0.90	0.02	0.12	0.81	0.14	0.20

Table 5: Confidence estimation on general reasoning for a Qwen3-8B policy trained with GRPO. CaC uses a value model fit post hoc on the GRPO rollouts.

	ID			OOD
Position	AUC	ECE	Brier	AUC	ECE	Brier
0% (prompt)	0.83	0.09	0.14	0.73	0.18	0.22
25%	0.86	0.07	0.13	0.79	0.13	0.18
50%	0.87	0.07	0.13	0.80	0.13	0.17
75%	0.86	0.05	0.12	0.80	0.13	0.17
100%	0.86	0.05	0.12	0.80	0.12	0.17

Table 6: Further analysis of the final general reasoning Qwen3-8B checkpoint. Confidence along the trajectory.

V_{\psi}(s_{t})

is read at the prompt alone (0%) and at 25%, 50%, 75% and 100% of each response’s length.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Algorithm	PPO with GAE
Discount $\gamma$ / GAE $\lambda$	$1.0$ / $1.0$
Reward	Terminal correctness, $\{0,1\}$
Actor / critic learning rate	$10^{-6}$ / $2\times 10^{-5}$
Learning-rate schedule	Constant
Actor clip ratio	$0.2$

Appendix

Table 7: Key PPO hyperparameters. KL regularization affects only the actor objective.

		In distribution						Out of distribution		Average
	Method	synth. math	olympiads	synth. amc	cn_k12	aops	amc_aime	AIME 24	AIME 25	ID	OOD
	Verbalized	0.70	0.63	0.69	0.62	0.63	0.67	0.75	0.70	0.66	0.73
	Self-consistency	0.67	0.73	0.64	0.68	0.77	0.72	0.91	0.96	0.70	0.94
	R-Tuning	0.86	0.89	0.93	0.87	0.87	0.91	0.91	0.85	0.89	0.88
	Probe	0.86	0.90	0.92	0.87	0.87	0.89	0.89	0.89	0.88	0.89
AUC $\uparrow$	CaC (ours)	0.89	0.93	0.95	0.88	0.89	0.89	0.92	0.95	0.91	0.94

Appendix

Table 8: Math, per benchmark. ID columns are the six NuminaMath sources used for RL. OOD is AIME 2024 and 2025. Best per column within each model and metric in bold.

		In distribution			Out of distribution			Average
	Method	TACO	Code- Contests	Code- forces	LCB Easy	LCB Medium	LCB Hard	ID	OOD
	Verbalized	0.53	0.69	0.65	0.52	0.65	0.65	0.60	0.61
	Self-consistency	0.66	0.90	0.90	0.83	0.92	0.92	0.78	0.89
	R-Tuning	0.87	0.86	0.84	0.84	0.89	0.84	0.86	0.85
	Probe	0.90	0.87	0.85	0.81	0.89	0.85	0.88	0.84
AUC $\uparrow$	CaC (ours)	0.93	0.93	0.91	0.84	0.94	0.95	0.93	0.91

Appendix

Table 9: Code, per benchmark. ID columns are the three sources used for RL. OOD is LiveCodeBench (LCB) by difficulty. Best per column within each model and metric in bold.

		In distribution	Out of distribution			Average
	Method	ToolRL	API-Bank L1	API-Bank L2	API-Bank L3	ID	OOD
	Verbalized	0.47	0.52	0.65	0.68	0.47	0.57
	Self-consistency	0.71	0.92	0.92	0.85	0.71	0.91
	R-Tuning	0.95	0.91	0.92	0.86	0.95	0.90
	Probe	0.93	0.84	0.92	0.85	0.93	0.85
AUC $\uparrow$	CaC (ours)	0.97	0.93	0.95	0.89	0.97	0.92

Appendix

Table 10: Agentic tool calling, per benchmark. ID is held-out ToolRL. OOD is API-Bank levels 1 to 3. Best per column within each model and metric in bold.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 249 papers

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
Jul 14, 2026Yaopei Zeng, Congchao Wang, JianHang Chen +3AI Agent Reliability LLM Agent Evaluation
Pennsylvania State University · Virginia Tech · Purdue University +1
When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
Sep 28, 2026Yekun Xu, Ante Wang, Jingyi Ren +3Confidence Estimation in Language Models LLM Reliability
College of AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs
Sep 27, 2026Shiyu Ni, Keping Bi, Jiafeng Guo +4Confidence Estimation in Language Models Language Model Calibration
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Jul 14, 2026J/K move · Enter open · S save

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Yaopei Zeng, Congchao Wang, JianHang Chen, Nan Wang+2

Pennsylvania State University · Virginia Tech · Purdue University+1

AI Agent Reliability LLM Agent Evaluation LLM Uncertainty Estimation LLM Agent Reliability Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

LLM agents operate in stateful environments, where a single erroneous step can waste limited interaction budget or cause irreversible effects before task failure becomes apparent. Reliable deployment therefore requires step-level confidence estimation: estimating, before execution, the probability that a proposed action will advance the task. Existing LLM confidence estimators are typically designed for static question answering under a fixed task context and evaluation criterion. For an agent, however, its action productivity depends on an environment transition that is observed only after execution. To address this challenge, we introduce Critic Experience Bank (CEB), a training-free framework that turns feedback from completed trajectories into reusable evidence for future confidence judgments. After each trajectory, an LLM assigns hindsight productivity pseudo-labels to individual actions and stores them with the critic's original pre-execution confidence, task context, action, and observed feedback. For a new action, by retrieving related productive and unproductive experiences, CEB grounds pre-execution confidence in feedback from completed trajectories to condition a fixed LLM critic. CEB thereby adapts over a task stream without parameter updates or ground-truth step labels at deployment. Across four agent benchmarks spanning offline and live web navigation, mobile GUI and shell tasks, and three critic backbones, CEB achieves the best or tied-best ECE, Brier score, and AUC in all twelve benchmark-backbone settings under rule-based step labels, reducing ECE by up to 53.8% relative to the strongest training-free baseline. Its confidence scores also improve downstream utility in selective execution and simulated task success.

Abstract

Confidence Estimation in Language Models

Explore similar work

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation

Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs