Organizations: Southern University of Science and Technology · Nanyang Technological University · University of Electronic Science and Technology of China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Large reasoning models (LRMs) often suffer from overconfidence when expressing their uncertainty. Confidence-aware reinforcement learning (RL) offers a promising way to optimize calibration. However, it relies on on-policy rollouts and is thus constrained by the model's pre-RL confidence distribution, which we term confidence prior. In this work, we reveal that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL. Theoretically, we prove that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk. To overcome this exploration bottleneck, we propose CalibSFT, a plug-and-play supervised fine-tuning stage that shapes a calibrated confidence prior with broad support before RL. For each question, CalibSFT constructs confidence targets combining its success rate with response-level correctness, which provably preserves proper-scoring optimality, and then balances training responses across the confidence spectrum to enable diverse confidence exploration during RL. To learn from incorrect responses without imitating their reasoning, CalibSFT introduces correctness-conditional supervision, guiding confidence across all responses while supervising reasoning only on correct ones. Across 16 mathematical and general reasoning benchmarks, incorporating CalibSFT reduces calibration errors and improves discrimination across five representative RL algorithms while preserving comparable accuracy. Furthermore, CalibSFT delivers practical benefits for downstream selective prediction and model routing. Our code is available at https://github.com/ml-stat-Sustech/verbalized-confidence-training.
Figures & tables
Figure 1: Confidence distributions from three base models on DeepScaleR-Train. Confidence concentrates on a few high values for both correct and incorrect responses.
Figure 2
In-distribution
Out-of-distribution
Method
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
Base
59.32
71.90
33.20
34.36
45.36
65.71
45.15
46.94
RLVR
64.16
73.16
31.10
31.52
43.88
64.19
50.29
51.52
CalibSFT
58.51
79.93
17.30
14.23
46.37
61.20
29.15
25.49
RLCR
65.33
82.36
14.43
14.46
43.96
69.38
31.14
31.17
+ CalibSFT
67.20
86.32
12.77
8.59
45.71
71.31
23.24
18.20
Table 1: Performance comparison on in-distribution and out-of-distribution benchmarks. Each entry averages the datasets within the corresponding split. CalibSFT initialization improves calibration and discrimination for all RL baselines while also raising Pass@1.
RLCR
ReDoubt
CoCA
DCPO
C3RL
Dataset
Method
r
ρ
r
ρ
r
ρ
r
ρ
r
ρ
Baseline
0.768
0.619
0.829
0.762
0.750
0.643
0.669
0.619
0.635
0.595
ID
+CalibSFT
0.963
0.952
0.976
0.976
0.887
0.976
0.996
0.976
0.896
0.833
Baseline
0.505
0.548
0.667
0.762
0.447
0.571
0.467
0.548
0.681
0.714
OOD
+CalibSFT
0.729
0.619
0.738
0.881
0.605
0.619
0.611
0.643
0.848
0.929
Table 2: Pearson r and Spearman ρ correlation between accuracy and mean confidence across ID and OOD datasets. Both correlations increase for all RL methods when incorporating CalibSFT, indicating that confidence can better reflect task-level accuracy.
Figure 4: Confidence distributions on MinervaMath for RLCR, CoCA, and DCPO. CalibSFT reduces concentration at dominant confidence values and lowers ECE for all three methods.
Figure 5: Accuracy ( ↑ ) and ECE ( ↓ ) on AIME 2026 across token budgets for RLCR. CalibSFT initialization yields lower ECE across all budgets while maintaining accuracy.
ID
OOD
ECE ↓
AUROC ↑
ECE ↓
AUROC ↑
Confidence Format
Base
+Ours
Base
+Ours
Base
+Ours
Base
+Ours
Linguistic (low / medium / high)
17.71
15.90
73.56
79.58
24.06
20.31
64.20
67.73
Integer (0–9)
14.69
14.06
79.00
84.14
26.31
16.70
67.00
69.36
Probability {0.00,0.05,…,1.00}
14.05
12.13
81.81
86.05
32.81
19.30
68.72
71.23
Probability {0.00,0.01,…,1.00}
14.05
8.59
81.81
86.32
32.81
18.20
68.72
71.31
Table 3: AUROC ( ↑ ) and ECE ( ↓ ) of RLCR on Qwen3-8B across confidence formats. “Base” denotes standard RLCR, and “+Ours” denotes RLCR initialized from CalibSFT. Across all four formats, CalibSFT consistently improves AUROC and reduces ECE across the 8 ID and 8 OOD benchmarks.
In-distribution
Out-of-distribution
Metric
Method
Math500
Olympiad
Minerva
DROP
HotpotQA
NQOpen
Average
RLCR
7.84
17.69
30.34
19.92
26.41
38.25
23.41
ECE ↓
+CalibSFT
5.34
7.76
12.74
13.30
13.04
22.77
12.49
RLCR
75.47
83.24
66.29
60.60
60.48
76.85
70.49
AUROC ↑
+CalibSFT
80.34
87.35
70.75
65.45
62.66
77.13
73.95
Table 4: AUROC ( ↑ ) and ECE ( ↓ ) of RLCR on Gemma4-E2B-Instruct across benchmarks of varying difficulty. CalibSFT noticeably improves calibration across all difficulty levels.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Format
System prompt
Probability
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence level. The confidence level indicates how certain the Assistant is about its answer, expressed as a decimal between 0 and 1 with exactly two decimal places, enclosed within <confidence> </confidence> tags. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> 0.xx </confidence> . The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.
Integer
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence score. The confidence score indicates how certain the Assistant is about its answer, expressed as a single integer from 0 to 9, enclosed within <confidence> </confidence> tags. A score of 0 means very low confidence and the answer is likely incorrect. A score of 9 means very high confidence and the answer is very likely correct. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> X </confidence> , where X is one of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.
Linguistic
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence level. The confidence level indicates how certain the Assistant is about its answer, expressed as exactly one of low, medium, or high, enclosed within <confidence> </confidence> tags. Low means the answer is likely incorrect, medium means the answer is uncertain, and high means the answer is likely correct. The response must strictly follow this format: <think> reasoning process here </think> <answer> final short answer only </answer> <confidence> label </confidence> . The <answer> tag must contain only the final answer string needed for exact-match evaluation, not a full sentence, explanation, or reasoning.
Appendix
Table 5: System prompts for the three confidence formats.
Figure 6: Training dynamics of RLCR, CoCA, and DCPO with and without CalibSFT initialization. Standard RL improves Brier score and ECE but degrades AUROC over training. In contrast, CalibSFT consistently achieves markedly better Brier score, ECE, and AUROC while preserving task accuracy.
Table 6: Per-dataset results on eight in-domain mathematical benchmarks spanning a range of difficulty. Each entry reports Pass@1, AUROC, Brier score, and ECE. CalibSFT initialization improves calibration and discrimination on most datasets, with the largest gains on benchmarks where baseline RL is heavily overconfident.
Table 7: Per-dataset results on eight out-of-domain general reasoning benchmarks spanning a range of difficulty. Each entry reports Pass@1, AUROC, Brier score, and ECE. CalibSFT initialization improves calibration and discrimination on most datasets, with the largest gains on benchmarks where baseline RL is heavily overconfident.
In-distribution
Out-of-distribution
Method
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
RLCR
65.50 ±0.66
82.08 ±0.38
14.64 ±0.29
14.26 ±0.29
43.53 ±0.61
69.05 ±0.47
31.67 ±0.75
31.99 ±1.15
+ CalibSFT
67.15 ±0.92
86.29 ±0.19
12.98 ±0.47
9.36 ±1.51
45.69 ±0.12
71.77 ±0.65
22.70 ±0.77
17.75 ±0.64
DCPO
64.69 ±0.21
78.40 ±0.41
14.63 ±0.19
11.78 ±0.33
43.77 ±0.39
68.37 ±0.54
33.22 ±1.15
32.69 ±1.09
+ CalibSFT
64.90 ±0.19
85.22 ±0.15
12.45 ±0.29
10.15 ±0.60
44.89 ±0.12
72.91 ±0.42
21.60 ±0.58
17.39 ±0.36
Appendix
Table 8: Performance stability across random seeds on Qwen3-8B. Each entry reports the mean and standard deviation over 3 random seeds (43, 44, and 45), following the evaluation protocol of Table 1 . The variation remains small across all metrics, and CalibSFT consistently outperforms RL baselines while maintaining task accuracy.
In-distribution
Out-of-distribution
Method
PCE ↓
Cap. Brier ↓
E-AURC ↓
RCE ↓
PCE ↓
Cap. Brier ↓
E-AURC ↓
RCE ↓
RLCR
9.71
10.63
7.75
13.50
30.99
27.53
19.74
11.45
+ CalibSFT
7.30
8.66
5.61
5.30
14.77
18.72
17.14
4.93
ReDoubt
6.71
11.98
8.02
15.23
22.51
21.71
18.34
14.77
+ CalibSFT
5.54
9.33
4.77
7.69
15.58
18.80
15.85
4.70
CoCA
10.96
12.82
8.88
19.49
22.64
22.84
21.52
15.13
Appendix
Table 9: Calibration performance on additional metrics, on the in-distribution and out-of-distribution splits and under the protocol of Table 1 . PCE is the part of ECE contributed by the overconfident bins, Cap. Brier scores each question’s mean confidence against its rollout success rate, E-AURC is the excess of the risk–coverage area over a perfect ranking, and RCE measures the deviation from a monotone correspondence between confidence and expected correctness. Across all baseline RL methods, incorporating CalibSFT reduces overconfidence and misranking errors (lower PCE, E-AURC, and RCE) while improving question-level confidence alignment (lower Cap. Brier).
In-distribution
Out-of-distribution
Method
Support ↑
Top-1 ↓
Top-3 ↓
Eff. Supp. ↑
Support ↑
Top-1 ↓
Top-3 ↓
Eff. Supp. ↑
RLCR
15.75
33.30
67.74
6.90
18.62
33.18
66.22
7.03
+ CalibSFT
86.88
12.59
20.57
49.42
94.50
3.73
9.90
72.83
ReDoubt
7.62
43.17
92.47
3.50
10.88
44.84
86.31
4.20
+ CalibSFT
58.00
9.43
23.64
32.03
59.50
7.83
20.70
31.62
CoCA
12.75
55.50
77.38
4.85
13.62
46.41
67.06
6.06
Appendix
Table 10: Confidence diversity on the in-distribution and out-of-distribution splits, under the protocol of Table 1 . Support counts the unique confidence values a model produces, Top-1 and Top-3 are the share of responses taking its most frequent one and three values, and Effective Support (Eff. Supp.) is the exponential of the entropy of the value distribution. Baseline RL methods collapse into narrow confidence supports with heavy concentration on top values, whereas CalibSFT broadens both support size and effective support for all objectives.
Target
[0.0,0.1]
(0.1,0.2]
(0.2,0.3]
(0.3,0.4]
(0.4,0.5]
(0.5,0.6]
(0.6,0.7]
(0.7,0.8]
(0.8,0.9]
(0.9,1.0]
CSFT
11.11
11.11
11.11
11.11
11.11
11.11
11.11
11.11
11.11
0.00
UaIT
0.00
0.00
0.00
0.00
0.00
0.00
6.48
56.06
35.15
2.31
SaySelf
22.07
14.41
11.93
10.22
9.04
8.26
8.89
6.63
5.56
3.00
LoVeC
27.33
3.78
8.17
0.20
1.37
0.63
0.22
1.54
15.80
40.96
CalibSFT
11.11
11.11
11.11
11.11
5.56
11.11
11.11
11.11
11.11
5.56
Appendix
Table 11: Distribution (%) of training targets across ten confidence intervals. Target scores derived from prior methods (UaIT, SaySelf, LoVeC) are concentrated or skewed, whereas CalibSFT provides a balanced, near-uniform distribution across the entire range.
Table 17
Table 14: Ablation of target-balanced sampling on DeepScaleR-Eval. Balancing substantially broadens confidence support and prevents concentration at extremes, while maintaining comparable task performance (Pass@1) and calibration.
Reasoning/answer
Confidence
Segment
ID performance
Variant
Correct
Incorrect
Correct
Incorrect
balanced
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
Confidence-only
–
–
✓
✓
–
43.56
74.62
20.97
17.43
Full-CE
✓
✓
✓
✓
–
58.49
61.66
29.29
27.65
Full-Balanced
✓
✓
✓
✓
✓
58.46
76.67
19.04
16.87
Correct-only
✓
–
✓
–
✓
59.89
77.58
18.95
16.98
CalibSFT
✓
–
✓
✓
✓
58.51
79.93
17.30
14.23
Appendix
Table 15: Ablation of the CalibSFT supervision design on eight ID benchmarks. The variants differ in the supervised responses for each segment and in loss balancing. CalibSFT achieves the best calibration while keeping Pass@1 close to the best.
In-distribution
Out-of-distribution
Estimator
Method
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
Pass@1 ↑
AUROC ↑
Brier ↓
ECE ↓
RLCR
65.33
82.36
14.43
14.46
43.96
69.38
31.14
31.17
Verbalized
+ CalibSFT
67.20
86.32
12.77
8.59
45.71
71.31
23.24
18.20
RLCR
65.33
83.60
14.35
15.03
43.96
69.70
31.22
30.54
Token probability
+ CalibSFT
67.20
87.66
12.75
10.13
45.71
71.51
22.80
19.29
RLCR
70.86
85.36
13.09
17.62
44.83
70.83
30.61
30.53
Appendix
Table 16: Confidence estimators for RLCR with and without CalibSFT initialization, averaged over the 8 in-distribution and 8 out-of-distribution benchmarks. Self-consistency aggregates the sampled responses by majority vote, so its Pass@1 column reports majority-vote accuracy. CalibSFT initialization improves calibration and discrimination under all three estimators.
Figure 9: Reliability diagrams of RLCR with and without CalibSFT across three in-distribution mathematical benchmarks. CalibSFT effectively reduces both underconfidence on easier tasks (MATH-500) and overconfidence on harder tasks (MinervaMath and AIME 2026), aligning confidence closer to actual accuracy.
Stage
Minutes per step
Wall-clock (h)
GPU-hours
CalibSFT
0.33
1.1
8.8
RLCR
4.77
15.9
127.3
RLCR + CalibSFT
4.58
15.3
122.2
Appendix
Table 17: Training cost comparison between CalibSFT and confidence-aware RL under a 200-step training budget on eight NVIDIA B20Z GPUs. CalibSFT takes substantially less training time and consumes only 6.9% of the GPU-hours of RLCR.
In-distribution
Out-of-distribution
Method
rmax=10%
rmax=20%
rmax=20%
RLCR
31.4 ±0.6
86.4 ±1.1
0.0 ±0.0
+ CalibSFT
62.6 ±2.5
88.6 ±1.3
20.3 ±0.8
ReDoubt
28.8 ±0.5
70.2 ±12.3
0.0 ±0.0
+ CalibSFT
66.1 ±0.9
88.6 ±1.2
18.0 ±2.5
CoCA
0.0 ±0.0
82.3 ±1.4
0.0 ±0.0
Appendix
Table 18: Certified coverage (%) at fixed risk budgets for confidence-aware RL with and without CalibSFT initialization. Values report the mean and standard deviation on held-out questions over ten random splits ( δ=0.05 ). Perfect ranking serves as an oracle reference that ranks all correct responses above incorrect ones. CalibSFT initialization consistently increases certified coverage across all methods and risk budgets.
Figure 11: Certified coverage across risk budgets on ID (top) and OOD (bottom) benchmarks. The results are averaged over ten random splits. Initializing with CalibSFT consistently yields higher certified coverage across budgets and tracks the perfect-ranking reference curve more closely, particularly under strict risk tolerances.
Tolerance ε=1%
Tolerance ε=2%
Method
Deferred (%) ↓
Tokens saved (%) ↑
Accuracy
Deferred (%) ↓
Tokens saved (%) ↑
Accuracy
RLCR
65.0 ±1.1
10.1
82.97
45.9 ±1.0
23.3
83.33
+ CalibSFT
60.3 ±3.6
17.9
82.66
39.2 ±1.0
36.1
82.47
ReDoubt
67.4 ±0.8
10.1
82.99
43.1 ±1.0
27.1
83.24
+ CalibSFT
55.3 ±2.7
23.8
82.81
36.3 ±3.7
42.1
82.60
CoCA
98.7 ±0.1
0.6
82.61
98.7 ±0.1
0.6
82.61
Appendix
Table 19: Confidence-guided model routing to Qwen3.7-Flash across two accuracy-loss tolerances ε , where local policies are trained on Qwen3-8B. Values are averaged over ten random splits. CalibSFT initialization substantially reduces deferral rates and increases token savings while maintaining task accuracy close to the full-deferral baseline ( 82.58% ).
Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that confidence in reasoning models should be used both stage-wise and position-awarely. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.
Shuhao Li, Guodong Du, Anhao Zhao +3
Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University
Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confidence only once: either before thinking or after answering. We argue that confidence in reasoning models is state-dependent: before thinking, confidence should estimate the chance of the model correctly solving the prompt, while after thinking it should predict whether the realized answer is likely to be correct. This distinction determines the appropriate supervision target: prompt-level success should supervise confidence estimates made after seeing the prompt, while individual answer-level correctness should supervise confidence estimates made after answering. We introduce CALIBER (Calibration Before and After Reasoning), which elicits both estimates and supervises each with the target matched to its information state. Under this unified protocol, CALIBER reduces Expected Calibration Error (ECE) by 52.5% over the strongest single-confidence baseline on BigMathDigits for the 7B model, while achieving the best Brier score and AUROC, and remains within 2.1 points of the best accuracy. Further, on a larger 30B model, CALIBER achieves the best ECE on BigMathDigits while remaining competitive in Brier score and AUROC. Out of distribution, it achieves the best ECE and Brier score on GPQA and TriviaQA, and remains competitive on SimpleQA. Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.
Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.