RL for LLM reasoning often plateaus early as its policy sharpens. Policy entropy is studied as both a diagnostic and a control signal, but increasing it alone does not reliably change the eventual plateau in our experiments. We analyze RL's implicit target policy using a sharpness factor κ and effective counterpart κeff that describe how reward or advantage gaps translate into probability-ratio gaps. We introduce reward budgeting, a plug-in advantage construction assigning a fixed total reward budget to each prompt. For binary rewards and independent on-policy rollouts, its expected unclipped score-function update is the exact gradient of a finite-G concave objective with diminishing returns; its advantage gap decreases as more rollouts become correct. For general rewards, a shifted log-mean interpretation holds asymptotically under the stated regularity conditions. On Qwen3-8B and Llama3.2-3B-Instruct, reward budgeting consistently improves RL baselines, with gains up to +12.50 on AMC23 and +10.00 on AIME24 and AIME25. These results support an update-dynamics view of premature convergence and substantial gains without explicitly preserving entropy. Code is available at https://github.com/Qwen-Applications/RewardBudget.
Figures & tables
Figure 1: Average training entropy and validation accuracy across 12 Llama3.2-3B-Instruct runs.
Figure 2: Entropy-intervention test: each method uses its own setting for the first 500 steps and then switches back to the same baseline for steps 500–1000.
Entropy View
Supported?
What it suggests
A Sufficient Causal Lever
✗
The tested early entropy interventions did not improve the final observed plateau.
A Diagnostic Correlate
✓
Entropy can indicate over-converging dynamics.
Table 3
Figure 3: Validation accuracy as training progresses. The three vertical dashed lines mark the convergence step tconv . Overall, smaller κ results in later convergence but also slows training.
Figure 4: Training dynamics for DRGRPO, KLCov, CLIPCov, GSPO, and DAPO. The left axes show entropy in (a) and κeff in (b); the right axes show averaged validation accuracy in (a) and DAPO validation accuracy in (b).
Reward Budget
DrGRPO
GRPO (Most)
k/G
+
−
κ
+
−
κ
+
−
κ
1/8
0.88
-0.13
1000
0.88
-0.13
1000
2.65
-0.38
3024
2/8
0.38
-0.13
500
0.75
-0.25
1000
1.73
-0.58
2309
3/8
0.21
-0.13
333
0.63
-0.38
1000
1.29
-0.78
2066
4/8
0.13
-0.13
250
0.50
-0.50
1000
1.00
-1.00
2000
5/8
0.08
-0.13
200
0.38
-0.63
1000
0.78
-1.29
2066
Table 1: Correct/incorrect samples’ advantages in 2 decimal places across different constructions: Reward Budget, DrGRPO and GRPO (most RL methods fall into this type). G is the group size and k is the number of correct rollouts.
Method
AMC23
AIME24
AIME25
MATH500
Olympiad (Physics)
Average
Qwen3-8B
GRPO
82.50
26.67
43.33
95.60
51.27
59.87
+ RB
+10.00
+10.00
+10.00
+1.20
+2.12
+6.66
DrGRPO
82.50
30.00
43.33
97.20
47.46
60.10
+ RB
+5.00
+6.67
+3.33
-0.20
+1.27
+3.21
GSPO
90.00
26.67
36.67
96.80
47.04
59.44
Table 2: Benchmark scores (%) from eight responses per prompt at temperature 0.7 : mean accuracy for Qwen3-8B and pass@8 for Llama3.2-3B-Instruct. Shaded rows show each baseline’s percentage-point change with reward budgeting (RB); GRPO retains reference-policy KL.
Figure 5: Reward budgeting on Qwen3-8B: (a) training-entropy differences from each baseline, mostly below zero; (b) effective sharpness and mean validation accuracy across methods (black curve). These descriptive trajectories do not establish the cause of the accuracy gains.
KL
RB
AMC23
AIME24
AIME25
MATH500
Olympiad (Physics)
Average
✓
×
82.50
26.67
43.33
95.60
51.27
59.87
✓
✓
92.50
36.67
53.33
96.80
53.39
66.53
×
×
85.00
30.00
40.00
96.00
50.35
60.27
×
✓
95.00
36.67
46.67
97.00
52.89
65.65
Table 3: Qwen3-8B GRPO ablation of reference-policy KL and reward budgeting (RB); the first two rows give the absolute scores in Table 2 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Accuracy evaluated at predictive horizon 10 is plotted against the entropy for three settings κ=200,2000,5000 . Different marker groups indicate different κ . The inset describes the absolute regression coefficients ∣βH∣ from OLS linear fits that relate the target metric “Validation Accuracy” V to (i) “Entropy” H alone and (ii) H together with κ , showing the relative importance of H vs. κ .
Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.
Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but is suffer from advantage collapse: when all rollouts of a query receive identical rewards, the group variance vanishes, most damagingly on hard samples, where scaling rollout budgets yields little. We introduce Joint Policy and Prompt Optimization (P O) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. P O mines hard samples with a success-rate threshold, evolves reasoning prompts for them with GEPA, and internalizes the elicited trajectories via context distillation, which optimizes each trajectory under the original query and thus removes inference-time prompting, with a Context Ratio Mask (CRM) filtering out extreme likelihood ratios. P O restores critical advantage signals and surpasses the GRPO baseline by up to 8.2 points in average accuracy on six held-out benchmarks across all training datasets and backbones, while also outperforming DAPO and other baselines. The gains are especially pronounced on hard benchmarks, reaching up to 16.3 points above GRPO on average across AIME24 and AIME25. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates provides a self-reinforcing framework that facilitates more effective LLM alignment.
Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2
Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.
Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1