Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning
Organizations: University of Cambridge · Qwen Business Unit, Alibaba
Abstract
RL for LLM reasoning often plateaus early as its policy sharpens. Policy entropy is studied as both a diagnostic and a control signal, but increasing it alone does not reliably change the eventual plateau in our experiments. We analyze RL's implicit target policy using a sharpness factor and effective counterpart that describe how reward or advantage gaps translate into probability-ratio gaps. We introduce reward budgeting, a plug-in advantage construction assigning a fixed total reward budget to each prompt. For binary rewards and independent on-policy rollouts, its expected unclipped score-function update is the exact gradient of a finite- concave objective with diminishing returns; its advantage gap decreases as more rollouts become correct. For general rewards, a shifted log-mean interpretation holds asymptotically under the stated regularity conditions. On Qwen3-8B and Llama3.2-3B-Instruct, reward budgeting consistently improves RL baselines, with gains up to +12.50 on AMC23 and +10.00 on AIME24 and AIME25. These results support an update-dynamics view of premature convergence and substantial gains without explicitly preserving entropy. Code is available at https://github.com/Qwen-Applications/RewardBudget.
Figures & tables
| Entropy View | Supported? | What it suggests |
| A Sufficient Causal Lever | ✗ | The tested early entropy interventions did not improve the final observed plateau. |
| A Diagnostic Correlate | ✓ | Entropy can indicate over-converging dynamics. |
| Reward Budget | DrGRPO | GRPO (Most) | |||||||
| 1/8 | 0.88 | -0.13 | 1000 | 0.88 | -0.13 | 1000 | 2.65 | -0.38 | 3024 |
| 2/8 | 0.38 | -0.13 | 500 | 0.75 | -0.25 | 1000 | 1.73 | -0.58 | 2309 |
| 3/8 | 0.21 | -0.13 | 333 | 0.63 | -0.38 | 1000 | 1.29 | -0.78 | 2066 |
| 4/8 | 0.13 | -0.13 | 250 | 0.50 | -0.50 | 1000 | 1.00 | -1.00 | 2000 |
| 5/8 | 0.08 | -0.13 | 200 | 0.38 | -0.63 | 1000 | 0.78 | -1.29 | 2066 |
| Method | AMC23 | AIME24 | AIME25 | MATH500 | Olympiad (Physics) | Average |
| Qwen3-8B | ||||||
| GRPO | 82.50 | 26.67 | 43.33 | 95.60 | 51.27 | 59.87 |
| + RB | +10.00 | +10.00 | +10.00 | +1.20 | +2.12 | +6.66 |
| DrGRPO | 82.50 | 30.00 | 43.33 | 97.20 | 47.46 | 60.10 |
| + RB | +5.00 | +6.67 | +3.33 | -0.20 | +1.27 | +3.21 |
| GSPO | 90.00 | 26.67 | 36.67 | 96.80 | 47.04 | 59.44 |
| KL | RB | AMC23 | AIME24 | AIME25 | MATH500 | Olympiad (Physics) | Average |
| 82.50 | 26.67 | 43.33 | 95.60 | 51.27 | 59.87 | ||
| 92.50 | 36.67 | 53.33 | 96.80 | 53.39 | 66.53 | ||
| 85.00 | 30.00 | 40.00 | 96.00 | 50.35 | 60.27 | ||
| 95.00 | 36.67 | 46.67 | 97.00 | 52.89 | 65.65 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,162 papers