Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards

Song Yu; Li Li; Zhenming Wu

cs.LGNov 6, 2026

Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards

Authors: Song Yu, Li Li, Zhenming Wu

Organizations: School of Computer and Information Science, Southwest University, Chongqing 400715, China

Abstract

Policy entropy is widely used to diagnose exploration and collapse in reinforcement learning with verifiable rewards (RLVR), yet its magnitude does not consistently track reasoning performance across optimization objectives. We argue that this ambiguity arises because global token entropy conflates uncertainty in successful and failed trajectories, diversity across correct solution strategies, and variation among realizations of the same strategy. We introduce Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that separates complete-trajectory entropy from average token entropy, conditions uncertainty on verifier outcome and process validity, measures correct-solution-mode retention, and connects these population-level quantities to local covariance-driven updates. In controlled sweeps over six RLVR objectives at two model scales, entropy trajectories do not induce a stable ordering of reasoning accuracy. Token entropy is higher for verifier-failed than successful trajectories under both broad separately aggregated estimates and a common-question paired analysis, whose confidence intervals exclude equality at 10 of 12 displayed endpoints. A local covariance approximation predicts the direction of adjacent-checkpoint entropy change in 95.5% of nonfailed transitions; a reward-stratified endpoint proxy further shows that positive chosen-token covariance is more tail-concentrated within failed prefixes. Every reportable trained endpoint exhibits some contraction in independently audited correct-solution-mode breadth, whose positive association with large- $k$ coverage remains exploratory. These results show that entropy collapse should be diagnosed by which uncertainty is removed, rather than by global entropy alone.

Figures & tables

Figure 1: Global entropy does not monotonically track reasoning performance in the formal Qwen2.5-3B sweep.

Figure 2: Conceptual overview of RCEA. The same direction of global entropy change can reflect error removal with successful-mode preservation or correct-solution mode contraction. Here $R$ is verifier reward, $V$ is process validity, and $Z$ is a semantic solution mode; the panels illustrate limiting cases rather than empirically assigned collapse labels.

Method	AIME’24	AIME’25	MATH 500	AMC’23	Minerva	Average	Pass@64	Mode Retention
Qwen2.5-3B	13.3	3.3	48.6	35.0	15.0	23.05	92.8	100.0
+ GRPO ( Shao et al. 2024 )	3.3	0.0	63.8	35.0	20.0	24.43	93.0	94.2
+ DAPO ( Yu et al. 2025 )	10.0	0.0	64.4	27.5	20.0	24.38	91.2	91.1
+ Clip-Cov ( Cui et al. 2025 )	10.0	3.3	65.4	42.5	17.5	27.75	93.2	96.0
+ KL-Cov ( Cui et al. 2025 )	3.3	3.3	62.6	32.5	15.0	23.35	93.6	94.8
+ EP-GRPO ( Yu et al. 2026 )	0.0	0.0	61.4	37.5	15.0	22.78	93.2	99.9

Table 1: Reasoning accuracy, MATH 500 pass@64, and audited correct-solution-mode retention (%). Macro Average weights the five benchmarks equally. Bold and underline mark the best and second-best reportable results within each scale. Clip-Cov-7B: step 3,600 (pre-failure).

Figure 3: Training token-entropy trajectories. Thin lines are raw traces, thick lines are rolling summaries.

Figure 4: Failed-to-successful token-entropy ratios. Values above one indicate higher entropy in verifier-failed trajectories; outcome strata are aggregated separately.

Figure 5: Common-question paired token-entropy ratios. Points show geometric means and bars denote paired-bootstrap 95% confidence intervals.

Figure 6: Predicted versus observed entropy changes across adjacent checkpoints.

y=x

indicates perfect prediction.

Method	Qwen2.5-3B [-1pt]successful $\rightarrow$ failed	Qwen2.5-7B [-1pt]successful $\rightarrow$ failed
Base	2.4 $\rightarrow$ 61.2 [-1pt] $+58.8$ pp	2.7 $\rightarrow$ 43.2 [-1pt] $+40.4$ pp
GRPO	9.5 $\rightarrow$ 82.7 [-1pt] $+73.2$ pp	9.9 $\rightarrow$ 91.8 [-1pt] $+81.9$ pp
DAPO	12.5 $\rightarrow$ 83.9 [-1pt] $+71.4$ pp	10.8 $\rightarrow$ 93.2 [-1pt] $+82.4$ pp
Clip-Cov	10.0 $\rightarrow$ 84.5 [-1pt] $+74.5$ pp	1.5 $\rightarrow$ 83.0 [-1pt] $+81.4$ pp
KL-Cov	4.6 $\rightarrow$ 70.4 [-1pt] $+65.8$ pp	10.2 $\rightarrow$ 86.2 [-1pt] $+76.0$ pp
EP-GRPO	7.1 $\rightarrow$ 82.9 [-1pt] $+75.8$ pp	7.6 $\rightarrow$ 81.3 [-1pt] $+73.6$ pp

Table 2: Within-stratum q99 share of positive chosen-token covariance (%), successful

\rightarrow

failed.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Role	GPU	CPU	System memory	Execution pattern
Training	1 $\times$ RTX PRO 6000 (96 GB)	25 vCPU, Xeon Platinum 8470Q	120 GB	One single-GPU training job
Inference	8 $\times$ RTX 5090 D (32 GB each)	200 vCPU, Xeon Platinum 8470Q	496 GB	Up to eight independent workers

Appendix

Table A1: Compute infrastructure used for the reported experiments. The eight inference GPUs execute independent single-GPU tasks; Qwen2.5-3B/7B do not use tensor parallelism.

Dataset	Benchmark	Probe	Main use
AMC23	40	20	Greedy, probe
AIME24	30	20	Greedy, probe
AIME25	30	20	Greedy, probe
MATH500	500	100	Greedy, pass@ $k$ , probe
Minerva Math	40 subset	40	Greedy, probe

Appendix

Table A2: Evaluation inventory. “Probe” is the number of fixed questions used for entropy and semantic-mode diagnostics.

Main-paper artifact	Ordinary reporting point	Clip-Cov-7B treatment
Training-entropy trace	Complete one-epoch trace	Complete trace; failure marked at step 4,000
Main benchmark table	Final checkpoint	Step 2,000 pre-collapse milestone
Broad/paired reward ratios	Final diagnostic checkpoint	Step 2,000 pre-collapse probe
Exact covariance fit	Four adjacent 50-step transitions	All four transitions excluded
Reward-stratified q99 table	Final endpoint proxy	Step 8,000 retained as failed diagnostic

Appendix

Table A3: Checkpoint policy. Different Clip-Cov-7B entries answer different questions and must not be interpreted as measurements of one common endpoint.

Scale	Method	Broad $\rho_{H}$	Success/failure coverage	$n_{\mathrm{paired}}$	Paired ratio [95% CI]
3B	GRPO	1.642	43.5/82.5	52	1.216 [1.105, 1.345]
3B	DAPO	1.699	42.0/80.5	45	1.189 [1.048, 1.332]
3B	Clip-Cov	1.526	44.5/83.5	56	1.225 [1.143, 1.321]
3B	KL-Cov	1.457	43.5/85.0	57	1.209 [1.098, 1.329]
3B	EP-GRPO	1.467	40.5/85.5	52	1.267 [1.079, 1.487]
3B	GTPO	1.244	41.5/79.5	42	1.031 [0.969, 1.099]

Appendix

Table A4: Endpoint failed-to-successful token-entropy ratios. Coverage is the percentage of the 200 probe questions satisfying

K_{r}\geq 8

. Clip-Cov-7B uses the pre-collapse step-2,000 probe; all other trained runs use their final probe.

Scale	Method	$n$	Failure higher	$p$
3B	GRPO	52	71.2%	0.0016
3B	DAPO	45	73.3%	0.0012
3B	Clip-Cov	56	80.4%	$2.7{\times}10^{-6}$
3B	KL-Cov	57	70.2%	0.0016
3B	EP-GRPO	52	73.1%	0.0006
3B	GTPO	42	50.0%	0.5612

Appendix

Table A5: Question-level direction check on the common-outcome subset. “Failure higher” is the fraction with

\widehat{\mathcal{H}}_{\mathrm{tok},-}(x)>\widehat{\mathcal{H}}_{\mathrm{tok},+}(x)

;

p

is the one-sided exact sign-test probability under an equal-direction null.

Scale	Method	Ret.	95% CI	$n$
3B	GRPO	94.2%	[89.2, 99.9]	72
3B	DAPO	91.1%	[85.6, 96.7]	72
3B	Clip-Cov	96.0%	[90.1, 102.5]	73
3B	KL-Cov	94.8%	[89.1, 100.6]	72
3B	EP-GRPO	99.9%	[92.3, 108.1]	72
3B	GTPO	83.7%	[77.4, 89.9]	67

Appendix

Table A6: Question-bootstrap uncertainty for the published rarefied effective-mode retention. Intervals use 10,000 resamples and condition on the frozen rarefaction draws and the single training seed. Clip-Cov-7B uses step 2,000; other rows use final checkpoints.

Scope	$n$	Sign	$\rho_{s}$	$r$
All nonfailed	44	95.5%	0.986	0.968
3B	24	95.8%	0.990	0.994
7B nonfailed	20	95.0%	0.937	0.962

Appendix

Table A7: Finite-difference covariance fit by scale.

n

counts adjacent 50-raw-step transitions;

\rho_{s}

and

r

are Spearman and Pearson correlations.

Upper tokens	Success mass	Failure mass	Gap	Direction
5.0%	25.3%	94.6%	69.3 pp	12/12
2.5%	13.9%	94.2%	80.3 pp	12/12
1.0%	5.6%	86.6%	81.0 pp	12/12

Appendix

Table A8: Sensitivity to the rank-defined upper-tail budget. Mass columns macro-average the fraction of positive chosen-token covariance in each reward stratum; “Direction” counts endpoints with greater failed-stratum concentration.

Raw step	Accuracy	Boxed rate	Mean length
2,400	76.6%	99.6%	598.5
3,600	60.8%	95.2%	431.5
4,000	0.2%	1.0%	1,999.8
8,000	0.2%	0.4%	2,040.0

Appendix

Table A9: Dense MATH500 diagnostics for Clip-Cov-7B. The generation limit is 2,048 tokens.

Entropy Collapse in RL

See all 13 papers

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
Jun 7, 2026Shumeng Yang, Yisu Liu, Jiayi Zheng +2RL for Language Model Reasoning Maximum Entropy RL
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
May 12, 2026Huimin Xu, Shuai Zhao, Xiaobao Wu +1Entropy Collapse in RL Reinforcement Learning
Nanyang Technological University, Singapore · Shanghai Jiao Tong University, China · 3VinUniversity, Vietnam
Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR
Apr 13, 2026Yuhang He, Haodong Wu, Siyi Liu +7RL for Language Model Reasoning Token-Level Credit Assignment
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.

Jun 7, 2026J/K move · Enter open · S save

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR

Shumeng Yang, Yisu Liu, Jiayi Zheng, Zhaohui Yang+1

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications+2

RL for Language Model Reasoning Maximum Entropy RL Reinforcement Learning with Verifiable Rewards LLM Reasoning

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.

Abstract

Entropy Collapse in RL

Explore similar work

PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR

Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR