KL-Regularized RL

RL: Reinforcement Learning

Overview

KL-regularized reinforcement learning studies how to improve a policy while limiting how far it moves from a reference or behavior policy, often to stabilize learning or preserve useful behavior. Methods include trust-region policy updates, soft Bellman and Q-learning algorithms, and mirror-descent or Gibbs-style updates; forward and reverse KL penalties can produce different trade-offs in coverage and concentration. The approach is used in online and offline control, bandits and games, as well as language-model alignment with preference or verifiable rewards. Theory now gives fast-rate and convergence guarantees in specific settings, while a central practical tension is choosing the KL strength and direction to balance stability and capability retention against exploration, diversity, and distribution shift.

Latest papers

CardsList

64 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. Regularized Score Centering

    Nov 16, 2026Hao Bai, Chenlu Ye, Xuanchang Zhang +2Agentic RLAsynchronous RL

  2. A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

    Nov 10, 2026Audrey Huang, Nan Jiang, Sham M. Kakade +3Policy OptimizationRL Post-Training

  3. GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

    Nov 4, 2026Xuliang Huang, Tianxiang Zhao, Chengwei Qin +1Offline RLRL for Language Model Reasoning

  4. Forecasting How RLHF Responds to Reward-Model Edits Before Training

    Nov 2, 2026Zixiao Wang, Duzhen Zhang, Xingbo Du +2Reinforcement LearningRL Post-Training

  5. Controlling Sharpening in LLM Post-Training

    Nov 2, 2026Simon Matrenok, Nicolas Menet, Nicolas Flammarion +1RL Post-TrainingKL-Regularized RL

  6. ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL

    Nov 1, 2026Geonwoo Cho, Jaegyun Im, Doyoon Kim +1Offline-To-Online RLRL Stability

  7. Fast Rates for Offline Q-Learning with KL Regularization

    Oct 13, 2026Yize Hao, Guangxin Jiang, Shuang QiuOffline RLQ-Learning

  8. Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints

    Oct 13, 2026Haoxuan Gao, Qin-Wen Luo, Xiang Chen +5RL for Tool UseReinforcement Learning

  9. FERPO: Forward Entropy-Regularized Policy Optimization

    Oct 1, 2026Sebastian Sanokowski, Alireza Sarmadi, Majid KhadivKL-Regularized RLPolicy Optimization

  10. Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis

    Sep 27, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1Diffusion PolicyRL for Diffusion Models

  11. A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

    Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouKL-Regularized RLAdversarial Robustness

  12. Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess

    Aug 27, 2026Szymon Miłosz, Piotr Duch, Szymon GrabowskiChessRL Exploration

  13. A Forward-Inverse Dynamic Game Framework for Enhanced Multi-Agent Trajectory Planning

    Aug 3, 2026Tianle Liu, Youcheng Niu, Jing Zeng +2KL-Regularized RLGame Theory

  14. Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

    Jul 31, 2026Xujun Che, Yuchen Yuan, Weida Zhao +1Reinforcement LearningKL-Regularized RL

  15. ββ-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

    Jul 30, 2026Jiawei Xu, Minghui Liu, Juzheng Zhang +2KL-Regularized RLOn-Policy Self-Distillation

  16. Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

    Jul 29, 2026Keegan Harris, Brian W. Lee, Ian Waudby-Smith +3Reinforcement LearningKL-Regularized RL