Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Qiannian Zhao; Chen Yang; Jinhao Jing; Yunke Zhang; Xuhui Ren; Lu Yu; Shijie Zhang; Hongzhi Yin

cs.AIFeb 26, 2026

Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Authors: Qiannian Zhao, Chen Yang, Jinhao Jing, Yunke Zhang, Xuhui Ren, Lu Yu, Shijie Zhang, Hongzhi Yin

Abstract

Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks. In practice, these models are predominantly trained via Reinforcement Learning with Verifiable Rewards (RLVR), yet most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty. We term this discrepancy the uncertainty-reward mismatch, under which high- and low-uncertainty solutions are treated equivalently, preventing the policy from "Know What You Know" and impeding the shift from optimizing for correct answers to optimizing effective reasoning paths. This limitation is especially critical in reasoning-centric tasks such as mathematics and question answering, where performance hinges on the quality of the model's internal reasoning process rather than mere memorization of final answers. To address this, we propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs. EGPO estimates per-sample uncertainty using a zero-overhead entropy proxy derived from token-level likelihoods and aligns it with extrinsic correctness through an asymmetric calibration mechanism that preserves correct reasoning while selectively regulating overconfident failures, thereby enabling stable and uncertainty-aware policy optimization. Moreover, EGPO recovers informative learning signals from otherwise degenerate group-based rollouts without modifying the verifier or reward definition. Extensive experiments across multiple benchmarks demonstrate that the proposed EGPO leads to substantial and consistent improvements in reasoning performance, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

Figures & tables

Figure 1. Overview of EGPO: sample-level metacognitive calibration (four quadrants) and group-level rollout triage (entirely-correct / mixed (contains both correct and incorrect outcomes) / entirely-incorrect).

Figure 2. Asymmetric calibration in EGPO: correct responses are never down-weighted and incorrect responses are never up-weighted.

1.5B Models
Model	MATH-500	AIME24	MMLU-STEM	Minerva	GSM8K
Qwen2.5-Math-1.5B (Base)	23.40	6.67	17.44	19.90	58.15
Qwen2.5-Math-1.5B + GRPO ( Yu et al., 2025 )	67.20 (+43.80)	13.33 (+6.66)	20.84 (+3.40)	32.26 (+12.36)	71.65 (+13.50)
Qwen2.5-Math-1.5B + DAPO ( Yu et al., 2025 )	69.80 (+46.40)	13.33 (+6.66)	22.52 (+5.08)	33.09 (+13.19)	72.33 (+14.18)
Qwen2.5-Math-1.5B + EDGE-GRPO ( Zhang et al., 2025 )	72.00 (+48.60)	16.67 (+10.00)	23.78 (+6.34)	33.82 (+13.92)	74.68 (+16.53)
Qwen2.5-Math-1.5B + EGPO (ours)	74.00 (+50.60)	13.33 (+6.66)	25.69 (+8.25)	37.13 (+17.23)	79.83 (+21.68)

Table 1. The Pass@1 Accuracy (%) on 1.5B and 7B models. Abbreviate DeepSeek-R1-Distill-Qwen-* as DeepSeek-R1-* .

Variant	Constraint on $w_{i}$	Entirely-incorrect groups	Renorm	MATH-500	Minerva	GSM8K	Avg.
Base	–	–	–	23.4	19.90	58.15	33.82
C1 (symmetric)	$\lambda_{\min}\leq w_{i}\leq\lambda_{\max}$	NSR + $w_{i}$	none	66.0	33.09	80.97	60.02
C2 (neg $\leq 1$ only)	$w_{i}\leq 1$ when $r_{i}=-1$ only	NSR + $w_{i}$	none	70.6	30.51	78.01	59.71
C3 (pos $\geq 1$ only)	$w_{i}\geq 1$ when $r_{i}=+1$ only	NSR + $w_{i}$	none	72.6	33.04	80.59	62.08
C4 (neg locked to 1)	asymmetric clamp	NSR with $w_{i}\equiv 1$	none	72.0	34.61	79.68	62.10
C5 (clamp $\rightarrow$ renorm)	asymmetric clamp	NSR + $w_{i}$	after clamp	81.05	35.29	80.29	65.54

Table 2. Ablations on Qwen2.5-Math-1.5B under the default binary outcome reward and

w_{i}

denotes EGPO’s group-relative weight.

Figure 3. Density comparison of the entropy on the response for Qwen2.5-Math-1.5B, comparing Base/GRPO/DAPO/EDGE-GRPO/EGPO on correct vs. incorrect rollouts.

Figure 4. A qualitative case on Qwen2.5-Math-1.5B comparing Base vs. EGPO, including the entropy on the response.

Backbone	TE	AE
DeepSeek-R1-Distill-Qwen-1.5B	0.745	0.778
DeepSeek-R1-Distill-Qwen-7B	0.700	0.738

Table 3. ROC-AUC for predicting incorrectness using entropy (higher is better).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Pool size / Val size	10,000 / 128
Subset ratio	default 0.85, extended 0.15
Bucket ratio	mixed 0.60, entirely_correct 0.40 (entirely_incorrect retained for robustness)
Min generations	$\geq 2$
Length filters	prompt $\leq 4096$ ; response $\leq 4096$ (tokenizer-based)
Answer constraints	single-value GT enforced; empty GT dropped

Appendix

Table 4. Construction recipe for the OpenR1-Stable-10k training dataset.

Panel of Figure 5 — Figure 5. ROC diagnostics for predicting incorrect rollouts using entropy on DeepSeek-R1-Distill-Qwen backbones, comparing thinking-side vs. answer-side entropy.

Figure 6. Entropy distributions for DeepSeek-R1-Distill-Qwen-1.5B (thinking-side vs. answer-side; correct vs. incorrect).

Panel of Figure 7 — Figure 7. Supplementary scatter plots for DeepSeek-R1-Distill-Qwen-1.5B.

Figure 8. Entropy distributions for DeepSeek-R1-Distill-Qwen-7B (thinking-side vs. answer-side; correct vs. incorrect).

Panel of Figure 9 — Figure 9. Supplementary scatter plots for DeepSeek-R1-Distill-Qwen-7B.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Oct 8, 2026

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

Gukhyeon Lee, SangKeun Lee

Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea

Language Model Calibration LLM Uncertainty Estimation Large Reasoning Models RL for Language Model Reasoning

Ask PDF

Mar 10, 2026

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.

Zhengzhao Ma, Xueru Wen, Boxi Cao +6

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2

Reinforcement Learning RL for Language Model Reasoning Language Model Calibration Reinforcement Learning with Verifiable Rewards

Ask PDF

Jun 7, 2026

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR

Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.

Shumeng Yang, Yisu Liu, Jiayi Zheng +2

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2

RL for Language Model Reasoning Maximum Entropy RL Reinforcement Learning with Verifiable Rewards LLM Reasoning

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR