cs.LGNov 11, 2026

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

Authors: Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho

Organizations: KRAFTON

Abstract

Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio

    Jan 30, 2026Shiye Lei, Zhihao Cheng, Dacheng TaoOff-Policy RLRL for Language Model Reasoning

  2. Towards Better Training Signal: Advantage Clipped Policy Optimization

    Sep 29, 2026Ruichuan Huang, Jinghan Liu, Congliang ChenPolicy OptimizationRL for Language Model Reasoning