On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models

Shuoyuan Wang; Beier Luo; Hao Zeng; Chengyao Yu; Songxin Zhang; Zejian Xie; Bingyi Jing; Hongxin Wei

cs.LGSep 26, 2026

On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models

Authors: Shuoyuan Wang, Beier Luo, Hao Zeng, Chengyao Yu, Songxin Zhang, Zejian Xie, Bingyi Jing, Hongxin Wei

Organizations: Southern University of Science and Technology · Nanyang Technological University · University of Electronic Science and Technology of China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

Abstract

Large reasoning models (LRMs) often suffer from overconfidence when expressing their uncertainty. Confidence-aware reinforcement learning (RL) offers a promising way to optimize calibration. However, it relies on on-policy rollouts and is thus constrained by the model's pre-RL confidence distribution, which we term confidence prior. In this work, we reveal that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL. Theoretically, we prove that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk. To overcome this exploration bottleneck, we propose CalibSFT, a plug-and-play supervised fine-tuning stage that shapes a calibrated confidence prior with broad support before RL. For each question, CalibSFT constructs confidence targets combining its success rate with response-level correctness, which provably preserves proper-scoring optimality, and then balances training responses across the confidence spectrum to enable diverse confidence exploration during RL. To learn from incorrect responses without imitating their reasoning, CalibSFT introduces correctness-conditional supervision, guiding confidence across all responses while supervising reasoning only on correct ones. Across 16 mathematical and general reasoning benchmarks, incorporating CalibSFT reduces calibration errors and improves discrimination across five representative RL algorithms while preserving comparable accuracy. Furthermore, CalibSFT delivers practical benefits for downstream selective prediction and model routing. Our code is available at https://github.com/ml-stat-Sustech/verbalized-confidence-training.

Figures & tables

Figure 1: Confidence distributions from three base models on DeepScaleR-Train. Confidence concentrates on a few high values for both correct and incorrect responses.

	In-distribution				Out-of-distribution
Method	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$
Base	59.32	71.90	33.20	34.36	45.36	65.71	45.15	46.94
RLVR	64.16	73.16	31.10	31.52	43.88	64.19	50.29	51.52
CalibSFT	58.51	79.93	17.30	14.23	46.37	61.20	29.15	25.49
RLCR	65.33	82.36	14.43	14.46	43.96	69.38	31.14	31.17
+ CalibSFT	67.20	86.32	12.77	8.59	45.71	71.31	23.24	18.20

Table 1: Performance comparison on in-distribution and out-of-distribution benchmarks. Each entry averages the datasets within the corresponding split. CalibSFT initialization improves calibration and discrimination for all RL baselines while also raising Pass@1.

		RLCR		ReDoubt		CoCA		DCPO		C3RL
Dataset	Method	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$	$r$	$\rho$
	Baseline	0.768	0.619	0.829	0.762	0.750	0.643	0.669	0.619	0.635	0.595
ID	+CalibSFT	0.963	0.952	0.976	0.976	0.887	0.976	0.996	0.976	0.896	0.833
	Baseline	0.505	0.548	0.667	0.762	0.447	0.571	0.467	0.548	0.681	0.714
OOD	+CalibSFT	0.729	0.619	0.738	0.881	0.605	0.619	0.611	0.643	0.848	0.929

Table 2: Pearson

r

and Spearman

\rho

correlation between accuracy and mean confidence across ID and OOD datasets. Both correlations increase for all RL methods when incorporating CalibSFT, indicating that confidence can better reflect task-level accuracy.

Figure 4: Confidence distributions on MinervaMath for RLCR, CoCA, and DCPO. CalibSFT reduces concentration at dominant confidence values and lowers ECE for all three methods.

Figure 5: Accuracy (

\uparrow

) and ECE (

\downarrow

) on AIME 2026 across token budgets for RLCR. CalibSFT initialization yields lower ECE across all budgets while maintaining accuracy.

	ID				OOD
	ECE $\downarrow$		AUROC $\uparrow$		ECE $\downarrow$		AUROC $\uparrow$
Confidence Format	Base	+Ours	Base	+Ours	Base	+Ours	Base	+Ours
Linguistic (low / medium / high)	17.71	15.90	73.56	79.58	24.06	20.31	64.20	67.73
Integer (0–9)	14.69	14.06	79.00	84.14	26.31	16.70	67.00	69.36
Probability $\{0.00,0.05,\ldots,1.00\}$	14.05	12.13	81.81	86.05	32.81	19.30	68.72	71.23
Probability $\{0.00,0.01,\ldots,1.00\}$	14.05	8.59	81.81	86.32	32.81	18.20	68.72	71.31

Table 3: AUROC (

\uparrow

) and ECE (

\downarrow

) of RLCR on Qwen3-8B across confidence formats. “Base” denotes standard RLCR, and “+Ours” denotes RLCR initialized from CalibSFT. Across all four formats, CalibSFT consistently improves AUROC and reduces ECE across the 8 ID and 8 OOD benchmarks.

Metric	Method	Math500	Olympiad	Minerva	DROP	HotpotQA	NQOpen	Average
		In-distribution			Out-of-distribution
	RLCR	7.84	17.69	30.34	19.92	26.41	38.25	23.41
ECE $\downarrow$	+CalibSFT	5.34	7.76	12.74	13.30	13.04	22.77	12.49
	RLCR	75.47	83.24	66.29	60.60	60.48	76.85	70.49
AUROC $\uparrow$	+CalibSFT	80.34	87.35	70.75	65.45	62.66	77.13	73.95

Table 4: AUROC (

\uparrow

) and ECE (

\downarrow

) of RLCR on Gemma4-E2B-Instruct across benchmarks of varying difficulty. CalibSFT noticeably improves calibration across all difficulty levels.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Format	System prompt
Probability	A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence level. The confidence level indicates how certain the Assistant is about its answer, expressed as a decimal between 0 and 1 with exactly two decimal places, enclosed within <confidence> </confidence> tags. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> 0.xx </confidence> . The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.
Integer	A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence score. The confidence score indicates how certain the Assistant is about its answer, expressed as a single integer from 0 to 9, enclosed within <confidence> </confidence> tags. A score of 0 means very low confidence and the answer is likely incorrect. A score of 9 means very high confidence and the answer is very likely correct. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> X </confidence> , where X is one of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.
Linguistic	A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence level. The confidence level indicates how certain the Assistant is about its answer, expressed as exactly one of low, medium, or high, enclosed within <confidence> </confidence> tags. Low means the answer is likely incorrect, medium means the answer is uncertain, and high means the answer is likely correct. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> label </confidence> . The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.

Appendix

Table 5: System prompts for the three confidence formats.

Figure 6: Training dynamics of RLCR, CoCA, and DCPO with and without CalibSFT initialization. Standard RL improves Brier score and ECE but degrades AUROC over training. In contrast, CalibSFT consistently achieves markedly better Brier score, ECE, and AUROC while preserving task accuracy.

Table 6: Per-dataset results on eight in-domain mathematical benchmarks spanning a range of difficulty. Each entry reports Pass@1, AUROC, Brier score, and ECE. CalibSFT initialization improves calibration and discrimination on most datasets, with the largest gains on benchmarks where baseline RL is heavily overconfident.

Table 7: Per-dataset results on eight out-of-domain general reasoning benchmarks spanning a range of difficulty. Each entry reports Pass@1, AUROC, Brier score, and ECE. CalibSFT initialization improves calibration and discrimination on most datasets, with the largest gains on benchmarks where baseline RL is heavily overconfident.

	In-distribution				Out-of-distribution
Method	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$
RLCR	65.50 ±0.66	82.08 ±0.38	14.64 ±0.29	14.26 ±0.29	43.53 ±0.61	69.05 ±0.47	31.67 ±0.75	31.99 ±1.15
+ CalibSFT	67.15 ±0.92	86.29 ±0.19	12.98 ±0.47	9.36 ±1.51	45.69 ±0.12	71.77 ±0.65	22.70 ±0.77	17.75 ±0.64
DCPO	64.69 ±0.21	78.40 ±0.41	14.63 ±0.19	11.78 ±0.33	43.77 ±0.39	68.37 ±0.54	33.22 ±1.15	32.69 ±1.09
+ CalibSFT	64.90 ±0.19	85.22 ±0.15	12.45 ±0.29	10.15 ±0.60	44.89 ±0.12	72.91 ±0.42	21.60 ±0.58	17.39 ±0.36

Appendix

Table 8: Performance stability across random seeds on Qwen3-8B. Each entry reports the mean and standard deviation over 3 random seeds (43, 44, and 45), following the evaluation protocol of Table 1 . The variation remains small across all metrics, and CalibSFT consistently outperforms RL baselines while maintaining task accuracy.

	In-distribution				Out-of-distribution
Method	PCE $\downarrow$	Cap. Brier $\downarrow$	E-AURC $\downarrow$	RCE $\downarrow$	PCE $\downarrow$	Cap. Brier $\downarrow$	E-AURC $\downarrow$	RCE $\downarrow$
RLCR	9.71	10.63	7.75	13.50	30.99	27.53	19.74	11.45
+ CalibSFT	7.30	8.66	5.61	5.30	14.77	18.72	17.14	4.93
ReDoubt	6.71	11.98	8.02	15.23	22.51	21.71	18.34	14.77
+ CalibSFT	5.54	9.33	4.77	7.69	15.58	18.80	15.85	4.70
CoCA	10.96	12.82	8.88	19.49	22.64	22.84	21.52	15.13

Appendix

Table 9: Calibration performance on additional metrics, on the in-distribution and out-of-distribution splits and under the protocol of Table 1 . PCE is the part of ECE contributed by the overconfident bins, Cap. Brier scores each question’s mean confidence against its rollout success rate, E-AURC is the excess of the risk–coverage area over a perfect ranking, and RCE measures the deviation from a monotone correspondence between confidence and expected correctness. Across all baseline RL methods, incorporating CalibSFT reduces overconfidence and misranking errors (lower PCE, E-AURC, and RCE) while improving question-level confidence alignment (lower Cap. Brier).

	In-distribution				Out-of-distribution
Method	Support $\uparrow$	Top-1 $\downarrow$	Top-3 $\downarrow$	Eff. Supp. $\uparrow$	Support $\uparrow$	Top-1 $\downarrow$	Top-3 $\downarrow$	Eff. Supp. $\uparrow$
RLCR	15.75	33.30	67.74	6.90	18.62	33.18	66.22	7.03
+ CalibSFT	86.88	12.59	20.57	49.42	94.50	3.73	9.90	72.83
ReDoubt	7.62	43.17	92.47	3.50	10.88	44.84	86.31	4.20
+ CalibSFT	58.00	9.43	23.64	32.03	59.50	7.83	20.70	31.62
CoCA	12.75	55.50	77.38	4.85	13.62	46.41	67.06	6.06

Appendix

Table 10: Confidence diversity on the in-distribution and out-of-distribution splits, under the protocol of Table 1 . Support counts the unique confidence values a model produces, Top-1 and Top-3 are the share of responses taking its most frequent one and three values, and Effective Support (Eff. Supp.) is the exponential of the entropy of the value distribution. Baseline RL methods collapse into narrow confidence supports with heavy concentration on top values, whereas CalibSFT broadens both support size and effective support for all objectives.

Target	$[0.0,0.1]$	$(0.1,0.2]$	$(0.2,0.3]$	$(0.3,0.4]$	$(0.4,0.5]$	$(0.5,0.6]$	$(0.6,0.7]$	$(0.7,0.8]$	$(0.8,0.9]$	$(0.9,1.0]$
CSFT	11.11	11.11	11.11	11.11	11.11	11.11	11.11	11.11	11.11	0.00
UaIT	0.00	0.00	0.00	0.00	0.00	0.00	6.48	56.06	35.15	2.31
SaySelf	22.07	14.41	11.93	10.22	9.04	8.26	8.89	6.63	5.56	3.00
LoVeC	27.33	3.78	8.17	0.20	1.37	0.63	0.22	1.54	15.80	40.96
CalibSFT	11.11	11.11	11.11	11.11	5.56	11.11	11.11	11.11	11.11	5.56

Appendix

Table 11: Distribution (%) of training targets across ten confidence intervals. Target scores derived from prior methods (UaIT, SaySelf, LoVeC) are concentrated or skewed, whereas CalibSFT provides a balanced, near-uniform distribution across the entire range.

Table 17

Table 14: Ablation of target-balanced sampling on DeepScaleR-Eval. Balancing substantially broadens confidence support and prevents concentration at extremes, while maintaining comparable task performance (Pass@1) and calibration.

Variant	Correct	Incorrect	Correct	Incorrect	balanced	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$
	Reasoning/answer		Confidence		Segment	ID performance
Confidence-only	–	–	$\checkmark$	$\checkmark$	–	43.56	74.62	20.97	17.43
Full-CE	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	58.49	61.66	29.29	27.65
Full-Balanced	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	58.46	76.67	19.04	16.87
Correct-only	$\checkmark$	–	$\checkmark$	–	$\checkmark$	59.89	77.58	18.95	16.98
CalibSFT	$\checkmark$	–	$\checkmark$	$\checkmark$	$\checkmark$	58.51	79.93	17.30	14.23

Appendix

Table 15: Ablation of the CalibSFT supervision design on eight ID benchmarks. The variants differ in the supervised responses for each segment and in loss balancing. CalibSFT achieves the best calibration while keeping Pass@1 close to the best.

		In-distribution				Out-of-distribution
Estimator	Method	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Pass@1 $\uparrow$	AUROC $\uparrow$	Brier $\downarrow$	ECE $\downarrow$
	RLCR	65.33	82.36	14.43	14.46	43.96	69.38	31.14	31.17
Verbalized	+ CalibSFT	67.20	86.32	12.77	8.59	45.71	71.31	23.24	18.20
	RLCR	65.33	83.60	14.35	15.03	43.96	69.70	31.22	30.54
Token probability	+ CalibSFT	67.20	87.66	12.75	10.13	45.71	71.51	22.80	19.29
	RLCR	70.86	85.36	13.09	17.62	44.83	70.83	30.61	30.53

Appendix

Table 16: Confidence estimators for RLCR with and without CalibSFT initialization, averaged over the 8 in-distribution and 8 out-of-distribution benchmarks. Self-consistency aggregates the sampled responses by majority vote, so its Pass@1 column reports majority-vote accuracy. CalibSFT initialization improves calibration and discrimination under all three estimators.

Figure 9: Reliability diagrams of RLCR with and without CalibSFT across three in-distribution mathematical benchmarks. CalibSFT effectively reduces both underconfidence on easier tasks (MATH-500) and overconfidence on harder tasks (MinervaMath and AIME 2026), aligning confidence closer to actual accuracy.

Stage	Minutes per step	Wall-clock (h)	GPU-hours
CalibSFT	0.33	1.1	8.8
RLCR	4.77	15.9	127.3
RLCR + CalibSFT	4.58	15.3	122.2

Appendix

Table 17: Training cost comparison between CalibSFT and confidence-aware RL under a 200-step training budget on eight NVIDIA B20Z GPUs. CalibSFT takes substantially less training time and consumes only 6.9% of the GPU-hours of RLCR.

	In-distribution		Out-of-distribution
Method	$r_{\max}=10\%$	$r_{\max}=20\%$	$r_{\max}=20\%$
RLCR	31.4 ±0.6	86.4 ±1.1	0.0 ±0.0
+ CalibSFT	62.6 ±2.5	88.6 ±1.3	20.3 ±0.8
ReDoubt	28.8 ±0.5	70.2 ±12.3	0.0 ±0.0
+ CalibSFT	66.1 ±0.9	88.6 ±1.2	18.0 ±2.5
CoCA	0.0 ±0.0	82.3 ±1.4	0.0 ±0.0

Appendix

Table 18: Certified coverage (%) at fixed risk budgets for confidence-aware RL with and without CalibSFT initialization. Values report the mean and standard deviation on held-out questions over ten random splits (

\delta=0.05

). Perfect ranking serves as an oracle reference that ranks all correct responses above incorrect ones. CalibSFT initialization consistently increases certified coverage across all methods and risk budgets.

Figure 11: Certified coverage across risk budgets on ID (top) and OOD (bottom) benchmarks. The results are averaged over ten random splits. Initializing with CalibSFT consistently yields higher certified coverage across budgets and tracks the perfect-ranking reference curve more closely, particularly under strict risk tolerances.

	Tolerance $\varepsilon=1\%$			Tolerance $\varepsilon=2\%$
Method	Deferred (%) $\downarrow$	Tokens saved (%) $\uparrow$	Accuracy	Deferred (%) $\downarrow$	Tokens saved (%) $\uparrow$	Accuracy
RLCR	65.0 ±1.1	10.1	82.97	45.9 ±1.0	23.3	83.33
+ CalibSFT	60.3 ±3.6	17.9	82.66	39.2 ±1.0	36.1	82.47
ReDoubt	67.4 ±0.8	10.1	82.99	43.1 ±1.0	27.1	83.24
+ CalibSFT	55.3 ±2.7	23.8	82.81	36.3 ±3.7	42.1	82.60
CoCA	98.7 ±0.1	0.6	82.61	98.7 ±0.1	0.6	82.61

Appendix

Table 19: Confidence-guided model routing to Qwen3.7-Flash across two accuracy-loss tolerances

\varepsilon

, where local policies are trained on Qwen3-8B. Values are averaged over ten random splits. CalibSFT initialization substantially reduces deferral rates and increases token savings while maintaining task accuracy close to the full-deferral baseline (

82.58\%

Explore similar work

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration
Jul 15, 2026Shuhao Li, Guodong Du, Anhao Zhao +3Confidence Estimation in Language Models Language Model Calibration
Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
Jun 23, 2026Conor Finlay, Joshua Kurien, Saurabh Dash +2Confidence Estimation in Language Models Language Model Calibration
1Cohere Labs
Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University

Jul 15, 2026J/K move · Enter open · S save

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration

Shuhao Li, Guodong Du, Anhao Zhao, Wanyu Lin+2

Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University

Confidence Estimation in Language Models Language Model Calibration On-Policy Distillation RL Post-Training LLM Post-Training LLM Reasoning

Open paper AskPDF arXiv

Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that confidence in reasoning models should be used both stage-wise and position-awarely. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.

Abstract

Explore similar work

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

Process Supervision of Confidence Margin for Calibrated LLM Reasoning