Policy entropy is widely used to diagnose exploration and collapse in reinforcement learning with verifiable rewards (RLVR), yet its magnitude does not consistently track reasoning performance across optimization objectives. We argue that this ambiguity arises because global token entropy conflates uncertainty in successful and failed trajectories, diversity across correct solution strategies, and variation among realizations of the same strategy. We introduce Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that separates complete-trajectory entropy from average token entropy, conditions uncertainty on verifier outcome and process validity, measures correct-solution-mode retention, and connects these population-level quantities to local covariance-driven updates. In controlled sweeps over six RLVR objectives at two model scales, entropy trajectories do not induce a stable ordering of reasoning accuracy. Token entropy is higher for verifier-failed than successful trajectories under both broad separately aggregated estimates and a common-question paired analysis, whose confidence intervals exclude equality at 10 of 12 displayed endpoints. A local covariance approximation predicts the direction of adjacent-checkpoint entropy change in 95.5% of nonfailed transitions; a reward-stratified endpoint proxy further shows that positive chosen-token covariance is more tail-concentrated within failed prefixes. Every reportable trained endpoint exhibits some contraction in independently audited correct-solution-mode breadth, whose positive association with large-k coverage remains exploratory. These results show that entropy collapse should be diagnosed by which uncertainty is removed, rather than by global entropy alone.
Figures & tables
Figure 1: Global entropy does not monotonically track reasoning performance in the formal Qwen2.5-3B sweep.
Figure 2: Conceptual overview of RCEA. The same direction of global entropy change can reflect error removal with successful-mode preservation or correct-solution mode contraction. Here R is verifier reward, V is process validity, and Z is a semantic solution mode; the panels illustrate limiting cases rather than empirically assigned collapse labels.
Method
AIME’24
AIME’25
MATH 500
AMC’23
Minerva
Average
Pass@64
Mode Retention
Qwen2.5-3B
13.3
3.3
48.6
35.0
15.0
23.05
92.8
100.0
+ GRPO ( Shao et al. 2024 )
3.3
0.0
63.8
35.0
20.0
24.43
93.0
94.2
+ DAPO ( Yu et al. 2025 )
10.0
0.0
64.4
27.5
20.0
24.38
91.2
91.1
+ Clip-Cov ( Cui et al. 2025 )
10.0
3.3
65.4
42.5
17.5
27.75
93.2
96.0
+ KL-Cov ( Cui et al. 2025 )
3.3
3.3
62.6
32.5
15.0
23.35
93.6
94.8
+ EP-GRPO ( Yu et al. 2026 )
0.0
0.0
61.4
37.5
15.0
22.78
93.2
99.9
Table 1: Reasoning accuracy, MATH 500 pass@64, and audited correct-solution-mode retention (%). Macro Average weights the five benchmarks equally. Bold and underline mark the best and second-best reportable results within each scale. Clip-Cov-7B: step 3,600 (pre-failure).
Figure 3: Training token-entropy trajectories. Thin lines are raw traces, thick lines are rolling summaries.
Figure 4: Failed-to-successful token-entropy ratios. Values above one indicate higher entropy in verifier-failed trajectories; outcome strata are aggregated separately.
Figure 5: Common-question paired token-entropy ratios. Points show geometric means and bars denote paired-bootstrap 95% confidence intervals.
Figure 6: Predicted versus observed entropy changes across adjacent checkpoints. y=x indicates perfect prediction.
Table A1: Compute infrastructure used for the reported experiments. The eight inference GPUs execute independent single-GPU tasks; Qwen2.5-3B/7B do not use tensor parallelism.
Dataset
Benchmark
Probe
Main use
AMC23
40
20
Greedy, probe
AIME24
30
20
Greedy, probe
AIME25
30
20
Greedy, probe
MATH500
500
100
Greedy, pass@ k , probe
Minerva Math
40 subset
40
Greedy, probe
Appendix
Table A2: Evaluation inventory. “Probe” is the number of fixed questions used for entropy and semantic-mode diagnostics.
Main-paper artifact
Ordinary reporting point
Clip-Cov-7B treatment
Training-entropy trace
Complete one-epoch trace
Complete trace; failure marked at step 4,000
Main benchmark table
Final checkpoint
Step 2,000 pre-collapse milestone
Broad/paired reward ratios
Final diagnostic checkpoint
Step 2,000 pre-collapse probe
Exact covariance fit
Four adjacent 50-step transitions
All four transitions excluded
Reward-stratified q99 table
Final endpoint proxy
Step 8,000 retained as failed diagnostic
Appendix
Table A3: Checkpoint policy. Different Clip-Cov-7B entries answer different questions and must not be interpreted as measurements of one common endpoint.
Scale
Method
Broad ρH
Success/failure coverage
npaired
Paired ratio [95% CI]
3B
GRPO
1.642
43.5/82.5
52
1.216 [1.105, 1.345]
3B
DAPO
1.699
42.0/80.5
45
1.189 [1.048, 1.332]
3B
Clip-Cov
1.526
44.5/83.5
56
1.225 [1.143, 1.321]
3B
KL-Cov
1.457
43.5/85.0
57
1.209 [1.098, 1.329]
3B
EP-GRPO
1.467
40.5/85.5
52
1.267 [1.079, 1.487]
3B
GTPO
1.244
41.5/79.5
42
1.031 [0.969, 1.099]
Appendix
Table A4: Endpoint failed-to-successful token-entropy ratios. Coverage is the percentage of the 200 probe questions satisfying Kr≥8 . Clip-Cov-7B uses the pre-collapse step-2,000 probe; all other trained runs use their final probe.
Scale
Method
n
Failure higher
p
3B
GRPO
52
71.2%
0.0016
3B
DAPO
45
73.3%
0.0012
3B
Clip-Cov
56
80.4%
2.7×10−6
3B
KL-Cov
57
70.2%
0.0016
3B
EP-GRPO
52
73.1%
0.0006
3B
GTPO
42
50.0%
0.5612
Appendix
Table A5: Question-level direction check on the common-outcome subset. “Failure higher” is the fraction with Htok,−(x)>Htok,+(x) ; p is the one-sided exact sign-test probability under an equal-direction null.
Scale
Method
Ret.
95% CI
n
3B
GRPO
94.2%
[89.2, 99.9]
72
3B
DAPO
91.1%
[85.6, 96.7]
72
3B
Clip-Cov
96.0%
[90.1, 102.5]
73
3B
KL-Cov
94.8%
[89.1, 100.6]
72
3B
EP-GRPO
99.9%
[92.3, 108.1]
72
3B
GTPO
83.7%
[77.4, 89.9]
67
Appendix
Table A6: Question-bootstrap uncertainty for the published rarefied effective-mode retention. Intervals use 10,000 resamples and condition on the frozen rarefaction draws and the single training seed. Clip-Cov-7B uses step 2,000; other rows use final checkpoints.
Scope
n
Sign
ρs
r
All nonfailed
44
95.5%
0.986
0.968
3B
24
95.8%
0.990
0.994
7B nonfailed
20
95.0%
0.937
0.962
Appendix
Table A7: Finite-difference covariance fit by scale. n counts adjacent 50-raw-step transitions; ρs and r are Spearman and Pearson correlations.
Upper tokens
Success mass
Failure mass
Gap
Direction
5.0%
25.3%
94.6%
69.3 pp
12/12
2.5%
13.9%
94.2%
80.3 pp
12/12
1.0%
5.6%
86.6%
81.0 pp
12/12
Appendix
Table A8: Sensitivity to the rank-defined upper-tail budget. Mass columns macro-average the fraction of positive chosen-token covariance in each reward stratum; “Direction” counts endpoints with greater failed-stratum concentration.
Raw step
Accuracy
Boxed rate
Mean length
2,400
76.6%
99.6%
598.5
3,600
60.8%
95.2%
431.5
4,000
0.2%
1.0%
1,999.8
8,000
0.2%
0.4%
2,040.0
Appendix
Table A9: Dense MATH500 diagnostics for Clip-Cov-7B. The generation limit is 2,048 tokens.
Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.
Shumeng Yang, Yisu Liu, Jiayi Zheng +2
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2
Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Existing remedies, including entropy regularization and ratio-based clipping heuristics, either control entropy in a coarse-grained manner or rely on approximate on-policy training. In this paper, we revisit entropy collapse from a token-level entropy flow perspective. Our analysis reveals that entropy-decreasing tokens consistently outweigh entropy-increasing ones, resulting in a severely imbalanced entropy flow. This perspective provides a unified explanation of entropy collapse in existing RLVR algorithms and highlights the importance of balancing entropy dynamics. Motivated by this analysis, we propose On-Policy Entropy Flow Optimization (OPEFO), an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change, while remaining strict on-policy. Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training stability and final performance. We will release the code and models upon publication.
Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning ability of Large Language Models (LLMs), but sparse outcome rewards make token-level credit assignment difficult. We study token-level credit as a reward-conditioned shift from the behavior policy to a hindsight posterior. In autoregressive RLVR, this shift can be expressed through Conditional Mutual Information (CMI), which shows that token entropy upper-bounds possible hindsight credit. Entropy, however, indicates capacity rather than update direction, so we introduce the Four Quadrant Decomposition to separate updates by reward polarity and token entropy. Controlled interventions show that these two factors jointly shape token updates. Sustained reasoning gains concentrate in signed high-entropy quadrants, whereas low-entropy updates saturate quickly. Based on this analysis, we propose Hindsight-Aware Policy Optimization (HAPO), a sign-preserving modification to GRPO that performs capacity-guided advantage reallocation. Experiments on mathematical reasoning benchmarks in two model settings show that HAPO achieves competitive performance among entropy-aware baselines.
Yuhang He, Haodong Wu, Siyi Liu +7
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Ltd.