Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Chenxiao Fan; Chongming Gao; Gangyi Zhang; Leyang Shen; Yaxin Gong; Jiamin Wang; Jiakai Wang; Dong Wang

cs.LGSep 28, 2026

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Authors: Chenxiao Fan, Chongming Gao, Gangyi Zhang, Leyang Shen, Yaxin Gong, Jiamin Wang, Jiakai Wang, Dong Wang, +3 more

Organizations: University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore

Abstract

Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose CREDO (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction.

Figures & tables

Figure 1: (a) The calibration task: a model answers and reports a confidence; calibration means the reported probability matches observed correctness. (b) Verbalized methods sample the confidence as text and train it through a scalar reward. CREDO reads the relative probability of a pair of reserved tokens and trains it by direct regression from the calibration loss. (c) Confidence distributions on code. Verbalized reports (GRPO, RLCR, DCPO) concentrate on a few values shared by correct and incorrect answers, even when calibration is trained explicitly (RLCR, DCPO). The readout (CREDO) spreads across the full range.

Figure 2: Collapse of the verbalized channel, with CREDO for contrast. Effective levels

=1/\sum_{v}p_{v}^{2}

(equiprobable-level equivalent); top-value share = pool fraction of the most frequent value; tie tax = AUROC lost to exactly equal reports.

Figure 3: Overview of CREDO. Each rollout contains an answer, a confidence analysis, and a confidence slot. The readout $c$ is the relative probability of two reserved tokens at the slot. The calibration loss trains $c$ by differentiable regression toward a hybrid target $t$ . The policy gradient scores the answer and analysis segments through their text. Discrepancy weighting gives more weight to rollouts where the outcome and the readout disagree.

	Mathematics				Code
Method	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
Base	.439 $\pm$ .001	.472 $\pm$ .002	.644 $\pm$ .004	.453 $\pm$ .002	.455 $\pm$ .001	.418 $\pm$ .001	.690 $\pm$ .004	.395 $\pm$ .002
GRPO	.706 $\pm$ .009	.199 $\pm$ .027	.775 $\pm$ .062	.204 $\pm$ .029	.581 $\pm$ .007	.182 $\pm$ .070	.753 $\pm$ .061	.205 $\pm$ .066
RLCR	.682 $\pm$ .010	.159 $\pm$ .012	.831 $\pm$ .009	.171 $\pm$ .009	.553 $\pm$ .013	.075 $\pm$ .005	.815 $\pm$ .014	.146 $\pm$ .006
DCPO	.722 $\pm$ .008	.217 $\pm$ .008	.848 $\pm$ .012	.198 $\pm$ .003	.576 $\pm$ .003	.153 $\pm$ .024	.776 $\pm$ .025	.153 $\pm$ .024
Base-d	.439 $\pm$ .001	.442 $\pm$ .001	.765 $\pm$ .010	.424 $\pm$ .001	.455 $\pm$ .001	.389 $\pm$ .002	.782 $\pm$ .003	.373 $\pm$ .002

Table 1: Accuracy and calibration in and out of domain, each entry macro-averaged over evaluation sets and training seeds. Column groups are the evaluation domain. Baselines are read through their own verbalized numeral and again through a digit expectation (Appendix B.3 ), in the rows named -d ; CREDO is read through its readout. Bold marks the best mean in each column and any within one joint standard error of it.

	Mathematics				Code
	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
CREDO	.753	.094	.933	.088	.614	.052	.869	.117
w/o differentiable regression ( $\alpha{=}0$ )	.696	.241	.880	.232	.611	.172	.857	.172
w/o uncertainty-analysis segment	.709	.096	.902	.107	.587	.048	.868	.117
w/o confidence-segment advantage ( $\hat{A}^{\mathrm{conf}}{\equiv}0$ )	.696	.099	.909	.102	.576	.047	.867	.126
w/o discrepancy weighting ( $\kappa{=}0$ )	.721	.122	.904	.102	.572	.051	.843	.132

Table 2: Leave-one-out ablation of CREDO (seed 43).

Figure 4: Accuracy and Brier for the two evaluation domains as each coefficient (

\kappa

\alpha

\gamma

) of § 3 is varied on seed 43, the others held fixed. Exact values are in Table D.2 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Shared
Base model	Qwen3-8B	Precision	bf16
Seeds	$43$ , $44$ , $45$	Rollouts per question $G$	$8$
Learning rate	$2\times 10^{-6}$	Per-device batch size	$2$
Schedule	constant, no warmup	Gradient accumulation	$64$
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )	Epochs	$2$
Rollout temperature	$1.0$	Rollout top- $k$	$50$

Appendix

Table 4: Training configuration. Values that differ by domain are given as mathematics / code. Rollouts are generated with vLLM ( Kwon et al., 2023 ) .

Mathematics			Code
Set	Problems	$n$	Set	Problems	$n$
DeepScaleR (train)	$9{,}500$	—	DeepCoder (train)	$9{,}500$	—
DeepScaleR test	$500$	4	DeepCoder test	$500$	4
MATH-500	$500$	4	HumanEval+	$163$	4
AIME 2024	$30$	8	LiveCodeBench v5	$279$	4
AIME 2025	$30$	8	LiveCodeBench v6	$131$	4

Appendix

Table 5: Datasets.

n

is the number of samples drawn per problem at evaluation. AIME and AMC are the corresponding years’ competition problems; HumanEval+ strengthens the tests of HumanEval ( Chen et al., 2021 ) .

	Qwen3-1.7B				Qwen3-4B				Qwen3-8B
	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
Base	.268	.531	.691	.477	.425	.468	.641	.447	.440	.474	.640	.455
GRPO	.460	.391	.782	.357	.660	.221	.813	.220	.717	.174	.820	.179
RLCR	.494	.145	.853	.155	.663	.186	.836	.185	.682	.167	.837	.181
DCPO	.501	.203	.823	.189	.651	.220	.797	.196	.715	.222	.846	.195
Base-d	.268	.557	.717	.495	.425	.446	.765	.428	.440	.443	.772	.425

Appendix

Table 6: Mathematics results at three model scales (seed 43). All methods follow the same protocol as Table 1 . -d rows evaluate through the digit expectation; CREDO through its logit readout. Bold marks the best in each column.

	Score-function reward	Differentiable regression
Verbalized numeral	RLCR / DCPO (Table 1 )	Digit-exp. emission
$\{H,L\}$ readout	Readout + reward only	CREDO (Table 1 )

Appendix

Table 7: Design space for the confidence channel.

	Mathematics				Code
	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
CREDO (full)	.753	.094	.933	.088	.614	.052	.869	.117
Digit-exp. emission	.676	.251	.671	.218	.587	.180	.724	.229
Readout + reward only	.695	.262	.869	.252	.588	.187	.818	.186
GRPO + auxiliary head	.702	.106	.902	.111	.599	.125	.832	.155

Appendix

Table 8: Emission and training-path decomposition (seed 43). Bold marks the best in each column.

		Mathematics				Code
	Eval	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
RLCR	verbalized	.682	.167	.837	.181	.539	.070	.820	.144
RLCR + weighting (parsed)	verbalized	.701	.142	.837	.152	.545	.104	.818	.155
RLCR + weighting (digit-exp)	verbalized	.697	.141	.847	.148	.542	.085	.810	.149
RLCR	digit-exp	.682	.164	.855	.180	.539	.078	.822	.144
RLCR + weighting (parsed)	digit-exp	.701	.151	.857	.152	.545	.108	.823	.153

Appendix

Table 9: Verbalized baselines with discrepancy weighting (seed 43). The Eval column indicates the confidence source at evaluation. Shaded rows evaluate through the digit-probability expectation; CREDO through its logit readout; unshaded rows through the parsed verbalized numeral. Bold marks the best in each column.

	Mathematics				Code
$r^{\mathrm{ans}}$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
$\beta F+a$	.753	.094	.933	.088	.614	.052	.869	.117
$F\,(\beta+a)$	.714	.105	.910	.110	.589	.038	.871	.113
$a$	.723	.113	.924	.091	.583	.042	.889	.102

Appendix

Table 10: Answer-reward shapes on seed 43; all else follows Table B.2 .

Figure 6: Calibration curves, pooled over the three seeds and over each domain’s evaluation sets. Baselines report through their verbalized numeral, CREDO through its readout. Equal-mass bins; bars are Wilson 95% intervals ( Wilson, 1927 ) .

Accuracy $\uparrow$
	DeepScaleR	MATH-500	AIME-24	AIME-25	AIME-26	AMC-23	AMC-24	Average
Base	.603 $\pm$ .005	.821 $\pm$ .004	.240 $\pm$ .013	.172 $\pm$ .009	.136 $\pm$ .002	.599 $\pm$ .010	.503 $\pm$ .008	.439 $\pm$ .001
GRPO	.836 $\pm$ .006	.939 $\pm$ .001	.610 $\pm$ .021	.451 $\pm$ .021	.475 $\pm$ .030	.856 $\pm$ .014	.779 $\pm$ .014	.706 $\pm$ .009
RLCR	.821 $\pm$ .011	.932 $\pm$ .006	.564 $\pm$ .021	.382 $\pm$ .010	.465 $\pm$ .039	.851 $\pm$ .021	.762 $\pm$ .008	.682 $\pm$ .010
DCPO	.845 $\pm$ .003	.944 $\pm$ .007	.624 $\pm$ .035	.478 $\pm$ .009	.521 $\pm$ .019	.864 $\pm$ .018	.780 $\pm$ .029	.722 $\pm$ .008
Base-d	.603 $\pm$ .005	.821 $\pm$ .004	.240 $\pm$ .013	.172 $\pm$ .009	.136 $\pm$ .002	.599 $\pm$ .010	.503 $\pm$ .008	.439 $\pm$ .001

Appendix

Table 11: Per-set results on mathematics.

	Accuracy $\uparrow$					ECE $\downarrow$
	DeepCoder	HumanEval+	LCB v5	LCB v6	Average	DeepCoder	HumanEval+	LCB v5	LCB v6	Average
GRPO	.389 $\pm$ .015	.852 $\pm$ .003	.400 $\pm$ .002	.338 $\pm$ .007	.495 $\pm$ .004	.418 $\pm$ .058	.086 $\pm$ .011	.360 $\pm$ .042	.396 $\pm$ .057	.315 $\pm$ .041
RLCR	.388 $\pm$ .006	.858 $\pm$ .003	.400 $\pm$ .009	.365 $\pm$ .007	.502 $\pm$ .002	.311 $\pm$ .027	.054 $\pm$ .017	.271 $\pm$ .017	.276 $\pm$ .022	.228 $\pm$ .014
DCPO	.394 $\pm$ .022	.861 $\pm$ .004	.415 $\pm$ .025	.363 $\pm$ .019	.508 $\pm$ .014	.436 $\pm$ .059	.153 $\pm$ .017	.344 $\pm$ .047	.387 $\pm$ .049	.330 $\pm$ .040
GRPO-d	.389 $\pm$ .015	.852 $\pm$ .003	.400 $\pm$ .002	.338 $\pm$ .007	.495 $\pm$ .004	.405 $\pm$ .040	.050 $\pm$ .002	.345 $\pm$ .031	.391 $\pm$ .042	.298 $\pm$ .028
RLCR-d	.388 $\pm$ .006	.858 $\pm$ .003	.400 $\pm$ .009	.365 $\pm$ .007	.502 $\pm$ .002	.315 $\pm$ .022	.056 $\pm$ .011	.270 $\pm$ .016	.285 $\pm$ .023	.232 $\pm$ .013

Appendix

Table 12: Per-set results on code, models trained on mathematics.

	Accuracy $\uparrow$					ECE $\downarrow$
	DeepCoder	HumanEval+	LCB v5	LCB v6	Average	DeepCoder	HumanEval+	LCB v5	LCB v6	Average
Base	.318 $\pm$ .004	.829 $\pm$ .007	.345 $\pm$ .012	.330 $\pm$ .007	.455 $\pm$ .001	.558 $\pm$ .005	.105 $\pm$ .008	.503 $\pm$ .012	.506 $\pm$ .001	.418 $\pm$ .001
GRPO	.475 $\pm$ .006	.882 $\pm$ .011	.522 $\pm$ .008	.443 $\pm$ .007	.581 $\pm$ .007	.229 $\pm$ .070	.113 $\pm$ .098	.187 $\pm$ .083	.199 $\pm$ .036	.182 $\pm$ .070
RLCR	.449 $\pm$ .008	.872 $\pm$ .019	.480 $\pm$ .017	.412 $\pm$ .016	.553 $\pm$ .013	.049 $\pm$ .014	.111 $\pm$ .017	.068 $\pm$ .010	.071 $\pm$ .016	.075 $\pm$ .005
DCPO	.477 $\pm$ .016	.878 $\pm$ .016	.525 $\pm$ .013	.422 $\pm$ .010	.576 $\pm$ .003	.167 $\pm$ .031	.112 $\pm$ .015	.178 $\pm$ .041	.156 $\pm$ .013	.153 $\pm$ .024
Base-d	.318 $\pm$ .004	.829 $\pm$ .007	.345 $\pm$ .012	.330 $\pm$ .007	.455 $\pm$ .001	.528 $\pm$ .004	.074 $\pm$ .017	.474 $\pm$ .012	.481 $\pm$ .003	.389 $\pm$ .002

Appendix

Table 13: Per-set results on code.

Accuracy $\uparrow$
	DeepScaleR	MATH-500	AIME-24	AIME-25	AIME-26	AMC-23	AMC-24	Average
GRPO	.743 $\pm$ .004	.892 $\pm$ .005	.374 $\pm$ .016	.276 $\pm$ .009	.288 $\pm$ .004	.728 $\pm$ .003	.644 $\pm$ .010	.564 $\pm$ .003
RLCR	.701 $\pm$ .016	.873 $\pm$ .009	.324 $\pm$ .009	.233 $\pm$ .026	.207 $\pm$ .016	.688 $\pm$ .024	.579 $\pm$ .025	.515 $\pm$ .011
DCPO	.721 $\pm$ .009	.894 $\pm$ .002	.357 $\pm$ .010	.240 $\pm$ .024	.253 $\pm$ .013	.727 $\pm$ .004	.644 $\pm$ .017	.548 $\pm$ .003
GRPO-d	.743 $\pm$ .004	.892 $\pm$ .005	.374 $\pm$ .016	.276 $\pm$ .009	.288 $\pm$ .004	.728 $\pm$ .003	.644 $\pm$ .010	.564 $\pm$ .003
RLCR-d	.701 $\pm$ .016	.873 $\pm$ .009	.324 $\pm$ .009	.233 $\pm$ .026	.207 $\pm$ .016	.688 $\pm$ .024	.579 $\pm$ .025	.515 $\pm$ .011

Appendix

Table 14: Per-set results on mathematics, models trained on code.

Weighting strength $\kappa$
	Mathematics				Code
	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$	Acc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$	Brier $\downarrow$
$0$	.721	.122	.904	.102	.572	.051	.843	.132
$0.25$	.753	.094	.933	.088	.584	.044	.854	.124
$0.5$	.749	.093	.924	.101	.614	.052	.869	.117
$1$	.730	.108	.924	.100	.591	.049	.858	.124

Appendix

Table 15: Every evaluated setting of the three coefficients of § 3 , all on seed 43.

Explore similar work

Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation Large Language Model Reinforcement Learning
Johns Hopkins University
Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models
Sep 27, 2026Yadong Xi, Rongsheng Zhang, Tangjie Lv +2Confidence Calibration Large Reasoning Models
NetEase · Amazon · Singapore University of Technology and Design
OpenJev-RLCD: A Working RLCD Implementation
Sep 30, 2026Zhimin Gao, Pichao WangReinforcement Learning With Verifiable Reward Large Reasoning Models

Apr 25, 2026, cs.LGJ/K move · Enter open · S save

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme+1

Johns Hopkins University

Confidence Estimation Large Language Model Reinforcement Learning LLM Reasoning Strategies Process Reward Model

Open paper AskPDF arXiv

Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (\textbf{RLCM}), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.

Abstract

Explore similar work

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

OpenJev-RLCD: A Working RLCD Implementation