Organizations: National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing National Research Center for Information Science and Technology, Tsinghua University
Group Relative Policy Optimization (GRPO) enhances LLM reasoning but often induces overconfidence, where incorrect responses yield lower perplexity than correct ones, degrading relative calibration as described by the Area Under the Curve (AUC). Existing approaches either yield limited improvements in calibration or sacrifice gains in reasoning accuracy. We first prove that this degradation in GRPO-style algorithms stems from their uncertainty-agnostic advantage estimation, which inevitably misaligns optimization gradients with calibration. This leads to improved accuracy at the expense of degraded calibration. We then propose Calibration-Aware Policy Optimization (CAPO). It adopts a logistic AUC surrogate loss that is theoretically consistent and admits regret bound, enabling uncertainty-aware advantage estimation. By further incorporating a noise masking mechanism, CAPO achieves stable learning dynamics that jointly optimize calibration and accuracy. Experiments on multiple mathematical reasoning benchmarks show that CAPO-1.5B significantly improves calibration by up to 15% while achieving accuracy comparable to or better than GRPO, and further boosts accuracy on downstream inference-time scaling tasks by up to 5%. Moreover, when allowed to abstain under low-confidence conditions, CAPO achieves a Pareto-optimal precision-coverage trade-off, highlighting its practical value for hallucination mitigation.
Figures & tables
Figure 1: (a) Comparison of average calibration (measured by AUC-mean) and accuracy (measured by mean@16) across six test benchmarks for our method and all baselines on Qwen2.5-Math-7B; (b)(c) Comparison of calibration dynamics on the validation set during training between our method and GRPO (left) / GSPO (right).
Figure 2: The relationship between advantage and the confidence gap of a correct–wrong response pair.
Figure 3: Results of calibration (measured by AUC-mean) and accuracy (measured by mean@16) for our method and all baselines on Qwen2.5-Math-7B across six test benchmarks.
Model
Ours
GRPO
GSPO
CoDaPO
CDE
SimKO
Qwen2.5-Math-1.5B
25.33
20.33
20.00
21.67
16.67
11.67
Qwen2.5-Math-7B
38.33
33.33
32.21
31.66
31.66
23.33
Table 1: Inference-time scaling accuracy (%) of different methods on the AIME 2024 and AIME 2025 datasets.
Figure 4: Precision-Coverage curves of our method and all baselines on six test benchmarks for the Qwen2.5-Math-1.5B (left) and Qwen2.5-Math-7B models (right).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
train_batch_size
128
ppo_mini_batch_size
64
max_prompt_length
1050
max_response_length
3046
ref-high
2.5
ref-low
1.05
Appendix
Table 2: Key hyperparameter settings used in our experiments.
Figure 5: Results of calibration (measured by AUC-mean) and accuracy (measured by mean@16) for our method and all baselines on Qwen2.5-Math-1.5B across six test benchmarks.
Figure 6: Accuracy trajectories on the validation set over training steps for our method and all baselines on the Qwen2.5-Math-1.5B (left) and Qwen2.5-Math-7B models (right).
Figure 7: Ablation studies on the effectiveness of applying the masking mechanism alone to GRPO. The results show that the mask by itself does not improve calibration, highlighting the necessity of the calibration-aware advantage estimator.
Figure 8: Ablation studies on the sensitivity of accuracy improvement curves (a) and calibration metrics (b) to the hyperparameter τ .
Figure 9: Ablation studies on the impact of the noise-masking mechanism on training stability (a) and the sensitivity to the ref-high and ref-low hyperparameters (b).
Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.
Zhengzhao Ma, Xueru Wen, Boxi Cao +6
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney
Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO.
Junshu Pan, Zhizhang Fu, Shulin Huang +5
Zhejiang University · Westlake University · Shanghai Innovation Institute +1