Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
Organizations: KRAFTON
Abstract
Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Countdown | Mathematical reasoning |
|---|---|---|
| Models | Qwen2.5-3B-Instruct | Qwen2.5-3B-Instruct |
| Qwen2.5-Math-1.5B | ||
| Framework | verl [ Sheng et al., 2025 ] | |
| Optimizer | AdamW | |
| Learning rate | ||
| LR warmup steps | 10 | |
| Variant | Final reward |
|---|---|
| Dr. GRPO | 0.74 |
| Dr. GRPO w/o correction | 0.71 |
| Dr. GRPO w/o clipping | 0.73 |
| Method | Peak reward | Final reward | Final entropy | |
|---|---|---|---|---|
| GRPO | 8 | 0.74 | 0.72 | – |
| LGPO | 8 | 0.77 | 0.76 | – |
| GRPO (clipping only) | 16 | 0.68 | 0.66 | 2.21 |
| LGPO (gating only) | 16 | 0.78 | 0.76 | 0.80 |
| LGPO + clipping | 16 | 0.70 | 0.70 | 1.64 |
| Advantage of gated tokens | Share of gated tokens | Share of all training tokens |
|---|---|---|
| Zero ( ) | 48.9% | 12.1% |
| Negative ( ) | 45.1% | 11.1% |
| Positive ( ) | 6.1% | 1.5% |
| Total gated by LGPO | 100% | 24.7% |
| Total clipped by GSPO (reference) | – | 1.6% |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers