Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Qiannian Zhao; Chen Yang; Jinhao Jing; Yunke Zhang; Xuhui Ren; Lu Yu; Shijie Zhang; Hongzhi Yin

cs.AIFeb 26, 2026

Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Authors: Qiannian Zhao, Chen Yang, Jinhao Jing, Yunke Zhang, Xuhui Ren, Lu Yu, Shijie Zhang, Hongzhi Yin

Abstract

Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks. In practice, these models are predominantly trained via Reinforcement Learning with Verifiable Rewards (RLVR), yet most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty. We term this discrepancy the uncertainty-reward mismatch, under which high- and low-uncertainty solutions are treated equivalently, preventing the policy from "Know What You Know" and impeding the shift from optimizing for correct answers to optimizing effective reasoning paths. This limitation is especially critical in reasoning-centric tasks such as mathematics and question answering, where performance hinges on the quality of the model's internal reasoning process rather than mere memorization of final answers. To address this, we propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs. EGPO estimates per-sample uncertainty using a zero-overhead entropy proxy derived from token-level likelihoods and aligns it with extrinsic correctness through an asymmetric calibration mechanism that preserves correct reasoning while selectively regulating overconfident failures, thereby enabling stable and uncertainty-aware policy optimization. Moreover, EGPO recovers informative learning signals from otherwise degenerate group-based rollouts without modifying the verifier or reward definition. Extensive experiments across multiple benchmarks demonstrate that the proposed EGPO leads to substantial and consistent improvements in reasoning performance, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

Figures & tables

Figure 1. Overview of EGPO: sample-level metacognitive calibration (four quadrants) and group-level rollout triage (entirely-correct / mixed (contains both correct and incorrect outcomes) / entirely-incorrect).

Figure 2. Asymmetric calibration in EGPO: correct responses are never down-weighted and incorrect responses are never up-weighted.

1.5B Models
Model	MATH-500	AIME24	MMLU-STEM	Minerva	GSM8K
Qwen2.5-Math-1.5B (Base)	23.40	6.67	17.44	19.90	58.15
Qwen2.5-Math-1.5B + GRPO ( Yu et al., 2025 )	67.20 (+43.80)	13.33 (+6.66)	20.84 (+3.40)	32.26 (+12.36)	71.65 (+13.50)
Qwen2.5-Math-1.5B + DAPO ( Yu et al., 2025 )	69.80 (+46.40)	13.33 (+6.66)	22.52 (+5.08)	33.09 (+13.19)	72.33 (+14.18)
Qwen2.5-Math-1.5B + EDGE-GRPO ( Zhang et al., 2025 )	72.00 (+48.60)	16.67 (+10.00)	23.78 (+6.34)	33.82 (+13.92)	74.68 (+16.53)
Qwen2.5-Math-1.5B + EGPO (ours)	74.00 (+50.60)	13.33 (+6.66)	25.69 (+8.25)	37.13 (+17.23)	79.83 (+21.68)

Table 1. The Pass@1 Accuracy (%) on 1.5B and 7B models. Abbreviate DeepSeek-R1-Distill-Qwen-* as DeepSeek-R1-* .

Variant	Constraint on $w_{i}$	Entirely-incorrect groups	Renorm	MATH-500	Minerva	GSM8K	Avg.
Base	–	–	–	23.4	19.90	58.15	33.82
C1 (symmetric)	$\lambda_{\min}\leq w_{i}\leq\lambda_{\max}$	NSR + $w_{i}$	none	66.0	33.09	80.97	60.02
C2 (neg $\leq 1$ only)	$w_{i}\leq 1$ when $r_{i}=-1$ only	NSR + $w_{i}$	none	70.6	30.51	78.01	59.71
C3 (pos $\geq 1$ only)	$w_{i}\geq 1$ when $r_{i}=+1$ only	NSR + $w_{i}$	none	72.6	33.04	80.59	62.08
C4 (neg locked to 1)	asymmetric clamp	NSR with $w_{i}\equiv 1$	none	72.0	34.61	79.68	62.10
C5 (clamp $\rightarrow$ renorm)	asymmetric clamp	NSR + $w_{i}$	after clamp	81.05	35.29	80.29	65.54

Table 2. Ablations on Qwen2.5-Math-1.5B under the default binary outcome reward and

w_{i}

denotes EGPO’s group-relative weight.

Figure 3. Density comparison of the entropy on the response for Qwen2.5-Math-1.5B, comparing Base/GRPO/DAPO/EDGE-GRPO/EGPO on correct vs. incorrect rollouts.

Figure 4. A qualitative case on Qwen2.5-Math-1.5B comparing Base vs. EGPO, including the entropy on the response.

Backbone	TE	AE
DeepSeek-R1-Distill-Qwen-1.5B	0.745	0.778
DeepSeek-R1-Distill-Qwen-7B	0.700	0.738

Table 3. ROC-AUC for predicting incorrectness using entropy (higher is better).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Pool size / Val size	10,000 / 128
Subset ratio	default 0.85, extended 0.15
Bucket ratio	mixed 0.60, entirely_correct 0.40 (entirely_incorrect retained for robustness)
Min generations	$\geq 2$
Length filters	prompt $\leq 4096$ ; response $\leq 4096$ (tokenizer-based)
Answer constraints	single-value GT enforced; empty GT dropped

Appendix

Table 4. Construction recipe for the OpenR1-Stable-10k training dataset.

Panel of Figure 5 — Figure 5. ROC diagnostics for predicting incorrect rollouts using entropy on DeepSeek-R1-Distill-Qwen backbones, comparing thinking-side vs. answer-side entropy.

Figure 6. Entropy distributions for DeepSeek-R1-Distill-Qwen-1.5B (thinking-side vs. answer-side; correct vs. incorrect).

Panel of Figure 7 — Figure 7. Supplementary scatter plots for DeepSeek-R1-Distill-Qwen-1.5B.

Figure 8. Entropy distributions for DeepSeek-R1-Distill-Qwen-7B (thinking-side vs. answer-side; correct vs. incorrect).

Panel of Figure 9 — Figure 9. Supplementary scatter plots for DeepSeek-R1-Distill-Qwen-7B.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
Oct 8, 2026Gukhyeon Lee, SangKeun LeeLanguage Model Calibration LLM Uncertainty Estimation
Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Mar 10, 2026Zhengzhao Ma, Xueru Wen, Boxi Cao +6Reinforcement Learning RL for Language Model Reasoning
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2
PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
Jun 7, 2026Shumeng Yang, Yisu Liu, Jiayi Zheng +2RL for Language Model Reasoning Maximum Entropy RL
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2

Oct 8, 2026J/K move · Enter open · S save

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Gukhyeon Lee, SangKeun Lee

Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea

Language Model Calibration LLM Uncertainty Estimation Large Reasoning Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.

Abstract

RL for Language Model Reasoning

Explore similar work

RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR