Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

Yadong Xi; Rongsheng Zhang; Tangjie Lv; Ziyang Luo; Ruochen Zhao

cs.CLSep 27, 2026

Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

Authors: Yadong Xi, Rongsheng Zhang, Tangjie Lv, Ziyang Luo, Ruochen Zhao

Organizations: NetEase · Amazon · Singapore University of Technology and Design

Abstract

Reinforcement learning with binary correctness rewards trains correctness, not calibrated confidence. The confidence that reasoning models verbalize is systematically overconfident, and the problem is not merely one of scale: verbalized confidence tracks how willing a model is to commit to an answer, not how likely the answer is to be right. Post-hoc rescaling therefore fits one distribution but rarely transfers. We look inside the model instead. On factual question answering, a linear probe on the hidden state between the chain of thought and the answer is substantially better calibrated: its expected calibration error is 5 to 38 times lower than that of the verbalized score across four benchmarks and two model families. However, when used to pick among N sampled answers, that same probe nearly ties majority voting yet falls far short of the oracle. Internal states answer "how certain am I" well and "which answer is right" poorly, so the signal should be reported as a confidence rather than used to select answers. As a result, we introduce probe-guided self-distillation (Probe-SD): score a model's own sampled traces with the probe, overwrite the confidence each trace states, and finetune the base checkpoint of the same family, so nothing but the model itself remains at test time. On Qwen3-14B, Probe-SD cuts ECE from 0.178 to 0.024 in-domain and from 0.542 to 0.113 out-of-domain, where it also beats post-hoc recalibration and self-consistency distillation. The resulting confidence is well-calibrated and useful for weighted voting, behaviors previously attributed to online RL, here obtained with supervised finetuning alone.

Figures & tables

		Qwen3-14B					Ministral-3-8B
Dataset		Acc	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Cov. $\uparrow$	Acc	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Cov. $\uparrow$
TriviaQA (ID)	V	.729	.178 $\pm$ .020	.825 $\pm$ .017	.184 $\pm$ .017	.980	.664	.247 $\pm$ .024	.669 $\pm$ .015	.263 $\pm$ .021	.957
TriviaQA (ID)	P	.729	.015 $\pm$ .009	.948 $\pm$ .009	.081 $\pm$ .008	1.000	.664	.014 $\pm$ .008	.917 $\pm$ .012	.105 $\pm$ .009	.977
EntityQ	V	.267	.529 $\pm$ .022	.812 $\pm$ .017	.455 $\pm$ .018	.903	.207	.627 $\pm$ .022	.734 $\pm$ .016	.564 $\pm$ .019	.950
EntityQ	P	.267	.014 $\pm$ .009	.913 $\pm$ .014	.097 $\pm$ .010	1.000	.207	.024 $\pm$ .012	.909 $\pm$ .015	.091 $\pm$ .009	.999
NQ-open	V	.440	.436 $\pm$ .029	.705 $\pm$ .021	.416 $\pm$ .025	.723	.370	.513 $\pm$ .026	.643 $\pm$ .018	.490 $\pm$ .022	.798

Table 1: Verbalized score (V) versus hidden-state probe (P). All calibration metrics are computed on the subset that the source can score. Brier decomposition and missing-score sensitivity analysis are in Appendix E .

		In-domain (TriviaQA)				Out-of-domain (macro of 3)
Model	Method	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
Qwen3-14B	Verbal	.729 $\pm$ .024	.178 $\pm$ .020	.825 $\pm$ .017	.184 $\pm$ .017	.258 $\pm$ .013	.542 $\pm$ .013	.715 $\pm$ .017	.468 $\pm$ .011
	Verbal-Iso	—	.013 $\pm$ .009	.825 $\pm$ .017	.128 $\pm$ .010	—	.274 $\pm$ .012	.714 $\pm$ .017	.236 $\pm$ .007
	Verbal-TS+bias	—	.047 $\pm$ .011	.825 $\pm$ .017	.131 $\pm$ .010	—	.294 $\pm$ .011	.715 $\pm$ .017	.245 $\pm$ .009
	Answer Probability	—	.251 $\pm$ .024	.636 $\pm$ .029	.253 $\pm$ .023	—	.695 $\pm$ .013	.586 $\pm$ .020	.686 $\pm$ .013
	Surface-Feat	—	.042 $\pm$ .010	.858 $\pm$ .016	.128 $\pm$ .010	—	.332 $\pm$ .012	.697 $\pm$ .017	.285 $\pm$ .007

Table 2: Main results. TriviaQA is in-domain. Out-of-domain is the macro average of the other three benchmarks. Readout and rescaling methods reuse the Verbal answers and therefore have no separate accuracy.

	Components		In-domain				Out-of-domain
Variant	relabel	CW	Acc	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
Probe-SD (full)	yes	yes	.749	.024	.928	.092	.279	.113	.783	.147
w/o CW	yes	no	.748	.033	.919	.097	.280	.129	.768	.157
w/o relabel	no	no	.747	.159	.814	.173	.281	.522	.723	.446

Table 3: Component ablation on Qwen3-14B. “relabel” is probe relabeling of the verbalized confidence and CW the confidence-weighted loss.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

qid	idx	Gold	Model answer	Graded	Verbal $\to$ new	$p_{\mathrm{probe}}$
qb_8463	10	Times Square	Times Square	correct	95 $\to$ 73	.727
qw_10780	17	USA	United States of America	correct	95 $\to$ 74	.742
jp_2478	17	Elizabeth	Elizabeth	correct	99 $\to$ 81	.806
sfq_19580	11	Chris Huhne	Chris Huhne	correct	95 $\to$ 75	.749
sfq_4960	0	Glasgow	York	incorrect	85 $\to$ 66	.665

Appendix

Table 4: Targeted audit of mechanical relabeling. Candidates were selected by gap, so the table verifies the replacement mechanism and the direction of the label, not the average size of the correction.

Dimension	Qwen3-14B	Ministral-3-8B
Layers	40	34
Regularization $C$	$\{0.0025,0.005,0.01,0.025,0.05,0.1,0.25,0.5,1.0,2.0,5.0\}$
Readout	$\{\text{residual},\ \text{MLP}\}$
Total configurations	1,760	1,496

Appendix

Table 5: Probe grid search space.

Dataset	Readout	sel. Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
Qwen3-14B
TriviaQA	(i) residual	.772	.015	.948	.081
	(ii) MLP	.779	.019	.948	.081
EntityQ	(i) residual	.310	.014	.913	.097
	(ii) MLP	.302	.026	.909	.100
NQ-open	(i) residual	.463	.078	.826	.177

Appendix

Table 6: Residual versus MLP readout, per dataset and macro-averaged over the four test sets. (i) is the validation-selected winner for that model, (ii) the best configuration of the other readout type (Qwen: (i) residual L24

C{=}0.005

, (ii) MLP L24

C{=}0.01

, Ministral: (i) MLP L21

C{=}2.0

, (ii) residual L21

C{=}1.0

). These test-set numbers describe configurations already fixed on validation data. They are not used for selection. Bold marks the better of the two within a dataset.

Model	Dataset	Oracle (pass@ $N$ )	Max-Probe	Majority vote
Qwen3-14B	TriviaQA	.879	.778	.771
	EntityQ	.447	.315	.306
	NQ-open	.695	.458	.492
	SimpleQA	.246	.074	.077
Ministral-3-8B	TriviaQA	.844	.717	.728
	EntityQ	.363	.231	.237

Appendix

Table 7: Best-of-

N

accuracy at

N{=}32

. Bold marks the better of Max-Probe and majority voting. Oracle is the theoretical ceiling.

Metric	String pre-screen	After human review
Extraction accuracy	92.0% (184/200)	99.5% (199/200)
Grading accuracy (answerable)	93.8% (181/193)	99.5% (198/199)
Grading precision	83.1%	98.7% (77/78)
Grading recall	100%	100% (77/77)
True grading errors	—	1/200 (0.5%)

Appendix

Table 8: Extraction and grading reliability under a 200-question human audit.

Extraction outcome	$n$	Verdict	Flagged grading case	$n$	Verdict
Exact or prefix match	175	faithful	More specific answer contains gold	5	correct
Substring match	9	correct distillation	Alias not in gold list	3	correct
Surface mismatch, valid	5	valid synthesis	Functional description $\equiv$ entity	2	correct
Empty: explicit abstention	10	correct empty	Output contains gold substring	3	correct
Empty: answer was given	1	true error	Subset or containment relation	4	correct
			Diacritic difference	1	correct

Appendix

Table 9: Human classification of the audit sample. Left: extraction outcomes over all 200 cases. Right: attribution of the cases the string pre-screen flagged as grading anomalies.

Source	ECE		AUROC		Brier
	Full	Subset	Full	Subset	Full	Subset
Probe	.0147	.0363	.9477	.9621	.0809	.0718
Verbal	.1782	.1905	.8243	.8511	.1839	.1877
Token	.2514	.2602	.6360	.6181	.2530	.2586

Appendix

Table 10: Full-pool versus 500-row-subset metrics. The subset is used to audit labels, not to re-estimate the final ECE.

Dataset (Acc)	Source	Brier $\downarrow$	Rel $\downarrow$	Res $\uparrow$	Unc
Qwen3-14B
TriviaQA (.729)	Verbal	.184	.052	.059	.194
	Probe	.081	.001	.116	.197
EntityQ (.267)	Verbal	.455	.297	.040	.207
	Probe	.097	.000	.098	.196
NQ-open (.440)	Verbal	.416	.207	.023	.249

Appendix

Table 11: Brier score and its decomposition into Reliability (

\downarrow

), Resolution (

\uparrow

) and Uncertainty. Uncertainty depends on the base accuracy of the scorable subset and is therefore comparable only within a (model, dataset) cell. Bold marks the better value per cell.

Model	Method	TriviaQA	EntityQ	NQ-open	SimpleQA	Macro
ECE $\downarrow$
Qwen	Probe-SD	.023 $\pm$ .006	.090 $\pm$ .008	.124 $\pm$ .012	.114 $\pm$ .007	.088 $\pm$ .004
Qwen	Verbal ∗	.177 $\pm$ .021	.531 $\pm$ .022	.423 $\pm$ .032	.665 $\pm$ .015	.449 $\pm$ .012
Qwen	Probe-SD-LRM	.030 $\pm$ .007	.102 $\pm$ .008	.163 $\pm$ .013	.136 $\pm$ .007	.108 $\pm$ .004
Qwen	Probe-SD w/o CW	.032 $\pm$ .006	.105 $\pm$ .008	.146 $\pm$ .012	.126 $\pm$ .007	.102 $\pm$ .004
Qwen	SC-SD	.063 $\pm$ .008	.224 $\pm$ .009	.241 $\pm$ .013	.237 $\pm$ .009	.191 $\pm$ .005

Appendix

Table 12: Convention 2 (common subset). Common question counts are: EntityQ 982/998, NQ-open 690/893, SimpleQA 846/929, TriviaQA 989/981 for Qwen/Ministral. Bold marks the best method per column within a model.

Model	Method	TriviaQA	EntityQ	NQ-open	SimpleQA	Macro
ECE $\downarrow$
Qwen	Probe-SD	.024 $\pm$ .009	.089 $\pm$ .013	.162 $\pm$ .018	.104 $\pm$ .010	.095 $\pm$ .006
Qwen	Verbal ∗	.182 $\pm$ .020	.493 $\pm$ .020	.410 $\pm$ .022	.545 $\pm$ .018	.408 $\pm$ .010
Qwen	Probe-SD-LRM	.031 $\pm$ .010	.101 $\pm$ .014	.205 $\pm$ .018	.122 $\pm$ .010	.115 $\pm$ .007
Qwen	Probe-SD w/o CW	.033 $\pm$ .010	.105 $\pm$ .013	.182 $\pm$ .017	.114 $\pm$ .010	.109 $\pm$ .006
Qwen	SC-SD	.064 $\pm$ .013	.222 $\pm$ .016	.266 $\pm$ .019	.208 $\pm$ .012	.190 $\pm$ .007

Appendix

Table 13: Convention 3 (full-sample attribution, missing scores set to 0). AUROC is not reported because of the tie mass at 0.

Dataset	Convention	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
TriviaQA	scorable subset	.014 [.010,.026]	.948 [.938,.957]	.080 [.072,.089]
	common subset ( $n_{q}{=}989$ )	.013 [.010,.025]	.948 [.938,.957]	.080 [.071,.089]
	full-sample attribution	.015 [.010,.027]	—	.080 [.072,.088]
EntityQ	scorable subset	.022 [.015,.038]	.898 [.882,.914]	.109 [.099,.120]
	common subset ( $n_{q}{=}982$ )	.023 [.015,.039]	.899 [.883,.915]	.109 [.099,.120]
	full-sample attribution	.020 [.014,.035]	—	.099 [.090,.109]

Appendix

Table 14: Qwen readout-probe baseline from the SFT evaluation suite under the three denominator conventions. Brackets are 95% bootstrap intervals.

Hyperparameter	Qwen3-14B	Ministral-3-8B	Note
Training set size	10,000	10,000	identical across methods
Learning rate	$5\times 10^{-6}$	$3\times 10^{-6}$
Epochs	2	2	${\approx}625$ steps
LR schedule	cosine to 0	cosine to 0	50 warmup steps
Weight decay	0.1	0.1
Adam $\beta_{1}/\beta_{2}$	0.9 / 0.95	0.9 / 0.95

Appendix

Table 15: Supervised finetuning hyperparameters.

Purpose	$T$	Qwen3-14B	Ministral-3-8B
Evaluation probe scoring	0.6	L24 residual, $C{=}0.005$	L21 MLP, $C{=}2.0$
Probe-SD training labels	1.5	L24 residual, $C{=}0.005$	L22 residual, $C{=}0.25$

Appendix

Table 16: Probe configurations selected independently for each purpose and sampling temperature. Selection uses validation data only (Appendix C ).

Training pool	TriviaQA	NQ-open	EntityQ	SimpleQA
$T{=}0.6$	.739 $\pm$ .024	.470 $\pm$ .027	.289 $\pm$ .025	.077 $\pm$ .013
$T{=}1.0$	.738 $\pm$ .024	.467 $\pm$ .026	.288 $\pm$ .025	.078 $\pm$ .013
$T{=}1.5$	.744 $\pm$ .024	.470 $\pm$ .026	.291 $\pm$ .025	.076 $\pm$ .012
$T{=}1.5$ , no top- $k$ /top- $p$	.745 $\pm$ .024	.471 $\pm$ .026	.286 $\pm$ .024	.078 $\pm$ .012

Appendix

Table 17: Training-data generation temperature ablation for Qwen3-14B (§ 4.2 ). All resulting models are evaluated directly at

T{=}0.6

. Errors are 95% question-level bootstrap half-widths.

Dataset	Method	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
TriviaQA	Verbal	.729 $\pm$ .024 ∗	.178 $\pm$ .020	.825 $\pm$ .017	.184 $\pm$ .017
	Frozen Probe ∗	—	.014 $\pm$ .008	.948 $\pm$ .010	.080 $\pm$ .009
	Verbal-Iso	—	.013 $\pm$ .009	.825 $\pm$ .017	.128 $\pm$ .010
	Verbal-TS+bias	—	.047 $\pm$ .011	.825 $\pm$ .017	.131 $\pm$ .010
	Answer Prob.	—	.251 $\pm$ .024	.636 $\pm$ .029	.253 $\pm$ .023
	Surface-Feat	—	.042 $\pm$ .010	.858 $\pm$ .016	.128 $\pm$ .010

Appendix

Table 18: Qwen3-14B, complete per-dataset results. TriviaQA is in-domain. The other three are held-out test datasets under distribution shift. Bold marks the best value per column within a dataset.

Dataset	Method	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
TriviaQA	Verbal	.664 $\pm$ .025 ∗	.247 $\pm$ .024	.669 $\pm$ .015	.263 $\pm$ .021
	Frozen Probe ∗	—	.019 $\pm$ .008	.915 $\pm$ .013	.106 $\pm$ .009
	Verbal-Iso	—	.020 $\pm$ .014	.669 $\pm$ .015	.188 $\pm$ .009
	Verbal-TS+bias	—	.067 $\pm$ .016	.669 $\pm$ .015	.193 $\pm$ .008
	Answer Prob.	—	.302 $\pm$ .025	.569 $\pm$ .027	.306 $\pm$ .025
	Surface-Feat	—	.042 $\pm$ .010	.770 $\pm$ .016	.174 $\pm$ .009

Appendix

Table 19: Ministral-3-8B, complete per-dataset results. Conventions as in Table 18 .

Model	Set	$n$	mean	p25	p50	p75
Ministral-3-8B	Pool (random 15k)	15,000	0.636	0.317	0.777	0.943
Ministral-3-8B	Retained	13,840	0.672	0.495	0.782	0.930
Qwen3-14B	Pool (random 15k)	15,000	0.722	0.460	0.936	0.992
Qwen3-14B	Retained	14,951	0.723	0.485	0.915	0.989

Appendix

Table 20: Probe-score distribution of the candidate pool and the coherently selected set.

Model	Downsampling	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Sample Acc	$\Delta$ pp	Train Acc
Qwen3-14B	calibrated (Probe-SD)	0.091	0.819	0.134	0.397	—	—
Qwen3-14B	random ( rand_sub )	0.094	0.820	0.134	0.394	—	—
Ministral-3-8B	random	0.123	0.808	0.150	0.346	4.26	70.0%
	diffdist_rp	0.112	0.813	0.144	0.347	0.01	65.5%
	calibrated	0.105	0.815	0.142	0.348	1.17	61.8%
	probe_align	0.118	0.807	0.149	0.348	0.01	66.2%

Appendix

Table 21: Downsampling policies across models, 4-dataset macro (3 seeds).

\Delta

pp is the largest difficulty-decile deviation from the pool and Train Acc the aligned training accuracy, both from the Ministral variant study.

Decile	$q_{src}$	Probe	Observed acc	Probe-SD	SC-SD	Verbal-SD
1	.675	.191	.141	.295	.374	.756
2	.803	.337	.331	.424	.522	.817
3	.866	.461	.423	.537	.637	.867
4	.907	.601	.546	.645	.752	.906
5	.938	.836	.891	.865	.878	.936
6	.952	.900	.916	.904	.926	.945

Appendix

Table 22: Per-decile values behind Figure 3 (a): each source’s mean, aggregated by decile of the source verbalized score

q_{src}

on TriviaQA (

n{=}100

per decile). Deciles are equal-frequency in

q_{src}

, so that column is monotone by construction.

Quartile	Gap range	Probe-SD	SC-SD	Verbal-SD	$q_{src}$ itself
Q0	$[-.560,-.011)$	.018	.034	.037	.030
Q1	$[-.011,+.044)$	.025	.030	.019	.013
Q2	$[+.044,+.373)$	.120	.134	.202	.188
Q3	$[+.373,+.832]$	.164	.263	.588	.575

Appendix

Table 23: Per-quartile values behind Figure 3 (b): mean absolute error of each student relative to Probe, by quartile of the overconfidence gap

q_{src}-p_{\mathrm{probe}}

on TriviaQA (

n{=}250

per quartile). The last column is

\lvert q_{src}-p_{\mathrm{probe}}\rvert

, the distance between the verbalized score and Probe with no distillation at all.

	Probe-SD		SC-SD		Verbal-SD
Dataset	Probe	$q_{src}$	Probe	$q_{src}$	Probe	$q_{src}$
TriviaQA	.913	.771	.872	.791	.767	.822
EntityQ	.895	.676	.810	.738	.658	.803
NQ-open	.823	.550	.700	.590	.526	.725
SimpleQA	.762	.494	.642	.641	.468	.757

Appendix

Table 24: Question-level Pearson correlation between each student’s verbalized score and the two candidate teachers, Probe and the source verbalized score

q_{src}

. Bold marks, for each student, the teacher it correlates with more strongly.

Bin	$n$	$\mathrm{acc}_{src}$	Probe	SC	Probe-SD	SC-SD
Probe $\ll$ SC	23	.635	$-.165$ [ $-.309,-.012$ ]	$+.219$ [ $.070,.386$ ]	$-.100$ [ $-.232,.032$ ]	$+.051$ [ $-.116,.222$ ]
Probe $<$ SC	86	.700	$-.123$ [ $-.171,-.071$ ]	$+.041$ [ $-.009,.098$ ]	$-.090$ [ $-.138,-.042$ ]	$-.000$ [ $-.047,.050$ ]
$\approx$	579	.739	$-.003$ [ $-.014,.009$ ]	$+.013$ [ $.002,.026$ ]	$+.018$ [ $.003,.033$ ]	$+.045$ [ $.028,.062$ ]
Probe $>$ SC	178	.610	$+.052$ [ $.027,.078$ ]	$-.133$ [ $-.160,-.106$ ]	$+.028$ [ $-.008,.064$ ]	$+.087$ [ $.049,.127$ ]
Probe $\gg$ SC	134	.766	$+.078$ [ $.044,.111$ ]	$-.381$ [ $-.424,-.337$ ]	$+.023$ [ $-.011,.058$ ]	$+.041$ [ $.003,.082$ ]

Appendix

Table 25: Per-bin values behind Figure 7 : calibration residuals on TriviaQA (

N{=}1{,}000

questions), binned by the question-level disagreement

\Delta=p_{\mathrm{probe}}-f_{\mathrm{SC}}

. Residual

=

score

-

corresponding accuracy, so positive values are overconfidence. Probe and SC are referenced to the source accuracy and each student to its own. Brackets give 95% question-level bootstrap intervals (

B{=}5000

Quintile	$\lvert\mathrm{Probe}-\mathrm{SC}\rvert$	mean $\Delta$	Probe-SD	SC-SD	Verbal-SD
Q0	$[.000,.013)$	$-.004$	.030	.055	.115
Q1	$[.013,.044)$	$-.018$	.053	.079	.173
Q2	$[.044,.107)$	$-.023$	.123	.171	.327
Q3	$[.107,.242)$	$+.060$	.103	.146	.263
Q4	$[.242,.855]$	$+.295$	.100	.126	.179

Appendix

Table 26: Mean absolute error of each student relative to Probe, by quintile of

\lvert\mathrm{Probe}-\mathrm{SC}\rvert

on TriviaQA (

n{=}200

per quintile). This binning is by the magnitude of teacher disagreement and is not the same convention as Table 23 .

	Qwen3-14B: SFT vs Instruct	Ministral-3-8B: SFT vs Reasoning
Sample accuracy	.7508 vs .7272 ( $+.0236$ )	.6995 vs .6639 ( $+.0356$ )
Retrieval term	$+.0221$ (93.7%)	$+.0260$ (73.0%)
Usage term	$+.0002$ (0.8%)	$+.0071$ (19.9%)
Miss-branch term	$+.0013$ (5.5%)	$+.0025$ (7.0%)

Appendix

Table 27: Sample-level accuracy difference and its decomposition into a retrieval term, a usage (conditional accuracy) term, and a miss-branch term, for both families.

	Qwen3-14B		Ministral-3-8B
Metric	SFT-Base	Instruct	SFT-Base	Reasoning
Facts per trace	9.24	9.02	9.33	9.29
truth: true	80.1%	78.0%	69.7%	67.8%
truth: false (hallucination)	13.5%	15.2%	26.7%	27.6%
relevance: bridge	5.83	5.97	6.13	5.57
true $\wedge$ bridge	4.95	4.98	4.81	4.32

Appendix

Table 28: Chain-of-thought quality audit, 200 traces per family. Facts are counted per trace.

	Qwen3-14B			Ministral-3-8B
Metric	SFT-Base	Instruct	$\Delta$	SFT-Base	Reasoning	$\Delta$
Gold recall hit rate $r$	.7888	.7640	$+2.48$ pp	.7294	.6998	$+2.96$ pp
Accuracy $\mid$ gold recalled	.8915	.8912	$+0.03$ pp	.8889	.8791	$+0.98$ pp
Accuracy $\mid$ gold not recalled	.2254	.1962	$+2.92$ pp	.1888	.1622	$+2.66$ pp
Sample accuracy	.7508	.7272	$+2.36$ pp	.6995	.6639	$+3.56$ pp

Appendix

Table 29: Gold recall hit rate and conditional accuracy.

	Qwen3-14B		Ministral-3-8B			$\to$ SFT-Base
Difficulty	SFT	Instr.	SFT	Reas.	Source side	easy	med.	hard
easy	687	663	582	563	Qwen Instruct easy (663)	635	27	1
medium	179	190	255	245	Qwen medium (190)	51	130	9
hard	134	147	163	192	Qwen hard (147)	1	22	124
					Min. Reasoning easy (563)	506	56	1
					Min. medium (245)	73	151	21

Appendix

Table 30: Retrieval difficulty over 1,000 questions (left) and the difficulty transition matrix across the two checkpoints of the same family (right). Difficulty is defined by the number of gold-recall hits over the 10 samples of a question: easy = all 10, medium = 1–9, hard = never.

Explore similar work

Jun 23, 2026cs.CL

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confidence only once: either before thinking or after answering. We argue that confidence in reasoning models is state-dependent: before thinking, confidence should estimate the chance of the model correctly solving the prompt, while after thinking it should predict whether the realized answer is likely to be correct. This distinction determines the appropriate supervision target: prompt-level success should supervise confidence estimates made after seeing the prompt, while individual answer-level correctness should supervise confidence estimates made after answering. We introduce CALIBER (Calibration Before and After Reasoning), which elicits both estimates and supervises each with the target matched to its information state. Under this unified protocol, CALIBER reduces Expected Calibration Error (ECE) by 52.5% over the strongest single-confidence baseline on BigMathDigits for the 7B model, while achieving the best Brier score and AUROC, and remains within 2.1 points of the best accuracy. Further, on a larger 30B model, CALIBER achieves the best ECE on BigMathDigits while remaining competitive in Brier score and AUROC. Out of distribution, it achieves the best ECE and Brier score on GPQA and TriviaQA, and remains competitive on SimpleQA. Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.

Conor Finlay, Joshua Kurien, Saurabh Dash +2

1Cohere Labs

Confidence Estimation Large Reasoning Models LLM Reasoning Strategies

Ask PDF

Sep 28, 2026cs.LG

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose CREDO (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction.

Chenxiao Fan, Chongming Gao, Gangyi Zhang +8

University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore

Reinforcement Learning With Verifiable Reward Verifiable Rewards Readout

Ask PDF

Jul 15, 2026cs.CL

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration

Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.

Shuhao Li, Guodong Du, Anhao Zhao +3

Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University

Post-Training Supervised Finetuning Offline Reinforcement Learning

Ask PDF