Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

Zhengzhao Ma; Xueru Wen; Boxi Cao; Yaojie Lu; Hongyu Lin; Jinglin Yang; Min He; Xianpei Han

cs.LGMar 10, 2026

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

Authors: Zhengzhao Ma, Xueru Wen, Boxi Cao, Yaojie Lu, Hongyu Lin, Jinglin Yang, Min He, Xianpei Han,

Organizations: Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China ·

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.

Figures & tables

Figure 1: Illustration of gradient conflict between policy accuracy maximization and calibration error minimization.

Figure 2: The overall framework of DCPO, which leverages block-wise verbalized confidence rollout and decoupled advantage estimation to decouple the optimization objectives of accuracy and calibration, and further integrates instance-level and group-level signals for more stable calibration optimization.

Figure 3: Reliability diagrams for different LLMs. The dashed line denotes perfect calibration; bar height indicates empirical accuracy per confidence bin, and color intensity reflects sample frequency. The Expected Calibration Error (ECE) is reported above each subplot, revealing prevalent over-confidence across models.

Panel of Figure 4 — Figure 4: The impact of RLVR on LLM calibration, which demonstrate that model confidence increases during RLVR training and RLVR exacerbates the models’ over-confidence.

Figure 5: The accuracy and calibration performance of QWEN3-8B trained with different RL methods.The figures illustrate that while existing calibration optimization methods can improve model calibration, their accuracy decreases.

Method	Confidence	MATH-500				AIME24				AIME25
Method	Confidence	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$
Base	logtis	84.7	0.075	0.078	0.743	27.5	0.461	0.431	0.750	20.0	0.486	0.483	0.796
Base	verbal	82.8	0.115	0.114	0.566	25.8	0.534	0.550	0.674	18.3	0.665	0.624	0.612
GRPO	logtis	90.4	0.054	0.104	0.726	42.5	0.370	0.505	0.736	29.2	0.469	0.556	0.742
GRPO	verbal	88.0	0.105	0.104	0.475	40.0	0.515	0.510	0.556	28.3	0.593	0.593	0.568
ConfClass	-	90.4	0.102	0.079	0.524	42.5	0.363	0.297	0.642	29.2	0.401	0.352	0.523

Table 1: Accuracy and calibration evaluation on 6 mathematical reasoning benchmarks. We report Accuracy (Acc) for reasoning performance, ECE, PCE and AUROC for calibration estimation. DCPO-I and DCPO-G denote variants that use only instance-level accuracy and only group-level accuracy, respectively, as calibration optimization signals. Bold and underlined values indicate the best and second-best results in each column, respectively.

Method	LCB-v5				LCB-v6				HE+
Method	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$	AUROC $\uparrow$
Base	0.228	0.482	0.478	0.634	0.196	0.458	0.397	0.659	0.902	0.077	0.068	0.802
GRPO	0.320	0.393	0.522	0.693	0.278	0.386	0.411	0.788	0.944	0.083	0.083	0.821
ConfClass	0.320	0.462	0.401	0.685	0.278	0.397	0.373	0.772	0.944	0.102	0.113	0.839
RLCR	0.267	0.231	0.227	0.785	0.248	0.290	0.285	0.829	0.931	0.036	0.035	0.907
DCPO	0.314	0.223	0.205	0.796	0.283	0.267	0.250	0.845	0.947	0.034	0.032	0.914

Table 2: Accuracy and calibration evaluation over 3 code generation benchmarks.

Figure 6: Accuracy and PCE on AIME25 dataset at different training steps for GRPO and DCPO. The figures illustrate that during the training process, our method can significantly reduce over-confidence while preserving accuracy.

Variant	Acc $\uparrow$	ECE $\downarrow$	PCE $\downarrow$
DCPO	60.8	0.128	0.126
w/o Instance-level Labels	60.5	0.209	0.229
w/o Group-level Labels	58.7	0.138	0.122
w/o Decoupled Optimzation	57.3	0.258	0.247
w/o On-policy Training	56.3	0.223	0.210

Table 3: Ablation results averaged over 5 mathematical benchmarks.

Figure 7: The gradient-norm dynamics across different training methods, which demonstrates that DCPO achieves more stable optimization dynamics than other methods.

Figure 8: Distribution of verbalized confidence predictions across 5 mathematical benchmarks. The y-axis is log-scaled to better visualize the highly concentrated confidence distributions.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Base model	Qwen3-8B
Max prompt length	1024
Max response length	3000
Total epochs	5
Learning rate	$1\times 10^{-6}$
Training batch size	256

Appendix

Table 4: Key hyperparameters used in reinforcement learning training.

Model	Method	MATH		AIME24		AIME25		AMC23		AMC24
Model	Method	Acc	ECE	Acc	ECE	Acc	ECE	Acc	ECE	Acc	ECE
Llama3.1-8B-Instruct	Base	0.214	0.481	0.033	0.710	0.000	0.645	0.089	0.635	0.078	0.670
Llama3.1-8B-Instruct	GRPO	0.363	0.351	0.050	0.731	0.044	0.582	0.133	0.593	0.115	0.492
Llama3.1-8B-Instruct	RLCR	0.288	0.214	0.033	0.372	0.000	0.387	0.100	0.301	0.107	0.398
Llama3.1-8B-Instruct	DCPO	0.365	0.203	0.050	0.367	0.033	0.335	0.133	0.297	0.122	0.371
Qwen3-14B	Base	0.893	0.068	0.325	0.476	0.250	0.452	0.657	0.189	0.578	0.267

Appendix

Table 5: Mathematical reasoning performance across different model families and sizes.

Step	0	20	40	60	80	100	120
Violation Rate	1.60%	0.58%	0.20%	0.21%	0.17%	0.20%	0.15%

Appendix

Table 6: Format violation rate during training steps.

$\lambda$	Acc	ECE
0	0.587	0.138
0.25	0.594	0.130
0.5	0.608	0.128
0.75	0.611	0.167
1	0.605	0.209

Appendix

Table 7: Sensitivity to

\lambda

in DCPO.

Method	Base	GRPO	DCPO
Length	1033.63	1299.87	1292.21

Appendix

Table 8: Average generation length.

Figure 9: Generation length during training.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University
Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts
Sep 28, 2026Chenxiao Fan, Chongming Gao, Gangyi Zhang +8Confidence Estimation in Language Models RL for Language Model Reasoning
University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore
RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
Oct 8, 2026Gukhyeon Lee, SangKeun LeeLanguage Model Calibration LLM Uncertainty Estimation
Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea

Apr 25, 2026J/K move · Enter open · S save

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme+1

Johns Hopkins University

Confidence Estimation in Language Models RL for Language Model Reasoning Language Model Calibration LLM Reliability Process Supervision LLM Uncertainty Estimation

Open paper AskPDF arXiv

Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin (RLCM), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation.

Abstract

Reinforcement Learning

Explore similar work

Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty