KL-Regularized RL

RL: Reinforcement Learning

Overview

KL-regularized reinforcement learning studies how to improve a policy while limiting how far it moves from a reference or behavior policy, often to stabilize learning or preserve useful behavior. Methods include trust-region policy updates, soft Bellman and Q-learning algorithms, and mirror-descent or Gibbs-style updates; forward and reverse KL penalties can produce different trade-offs in coverage and concentration. The approach is used in online and offline control, bandits and games, as well as language-model alignment with preference or verifiable rewards. Theory now gives fast-rate and convergence guarantees in specific settings, while a central practical tension is choosing the KL strength and direction to balance stability and capability retention against exploration, diversity, and distribution shift.

Latest papers

CardsList

64 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

    Feb 25, 2026Kenton Tang, Yuzhu Chen, Fengxiang HeRL from Human FeedbackGeneralization in RL

  2. Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing

    Dec 30, 2025Lars van der Laan, Nathan KallusReinforcement LearningKL-Regularized RL

  3. GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

    Apr 28, 2025Kaichen Zhang, Yuzhong Hong, Junwei Bao +4LLM Post-TrainingLLM Alignment