KL-Regularized RL

RL: Reinforcement Learning

KL-regularized reinforcement learning studies how to improve a policy while limiting how far it moves from a reference or behavior policy, often to stabilize learning or preserve useful behavior. Methods include trust-region policy updates, soft Bellman and Q-learning algorithms, and mirror-descent or Gibbs-style updates; forward and reverse KL penalties can produce different trade-offs in coverage and concentration. The approach is used in online and offline control, bandits and games, as well as language-model alignment with preference or verifiable rewards. Theory now gives fast-rate and convergence guarantees in specific settings, while a central practical tension is choosing the KL strength and direction to balance stability and capability retention against exploration, diversity, and distribution shift.

Latest papers

55 papers9 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Oct 14, 2026Xuanchang Zhang, Tong ZhangReinforcement Learning with Verifiable Rewards Off-Policy RL
University of Illinois Urbana-Champaign
Fast Rates for Offline Q-Learning with KL Regularization
Oct 13, 2026Yize Hao, Guangxin Jiang, Shuang QiuOffline RL Q-Learning
Harbin Institute of Technology. · City University of Hong Kong.
Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
Oct 13, 2026Hao Bai, Jiarui Yao, Tong ZhangOff-Policy RL Reinforcement Learning with Verifiable Rewards
University of Illinois Urbana-Champaign
Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints
Oct 13, 2026Haoxuan Gao, Qin-Wen Luo, Xiang Chen +5RL for Tool Use Reinforcement Learning
Nanjing University of Aeronautics and Astronautics · Security Capability Center · Nanjing University +2
Density-Matrix Reinforcement Learning
Oct 12, 2026Chiuman HoReinforcement Learning KL-Regularized RL
TetraMem Inc.
When KL Regularization Misfires in Group Policy Optimization
Oct 8, 2026Fei DingKL-Regularized RL Group Relative Policy Optimization
Alibaba Group
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
FERPO: Forward Entropy-Regularized Policy Optimization
Oct 1, 2026Sebastian Sanokowski, Alireza Sarmadi, Majid KhadivKL-Regularized RL Policy Optimization
Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich
Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
Sep 27, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1Diffusion Policy RL for Diffusion Models
School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. +1
IncentRL: The Trade-Off Between Preference Guidance and Task Performance
Sep 18, 2026Xuening Wu, Yanlan Kang, Shenqin YinReinforcement Learning Preference-Based RL
Fudan University
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy
Sep 17, 2026Luis LealNash Equilibrium Self-Play RL
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouKL-Regularized RL Adversarial Robustness
Mohammed VI Polytechnic University · Khalifa University · Concordia University
Unifying ICL, SFT, KL-Regularized RL Through a Bayesian Lens
Sep 4, 2026Junxin FanRL for Language Models In-Context Learning
Fudan University
Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
Aug 27, 2026Szymon Miłosz, Piotr Duch, Szymon GrabowskiChess RL Exploration
Institute of Applied Computer Science, Lodz University of Technology
Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery
Aug 27, 2026Fei DingGroup Relative Policy Optimization Reinforcement Learning with Verifiable Rewards
Alibaba Group
Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning
Aug 3, 2026Li Wang, Xiaodong Lu, Xiaohan Wang +4Continual Learning for LLMs KL-Regularized RL
1Meituan · 2Nanyang Technological University
A Forward-Inverse Dynamic Game Framework for Enhanced Multi-Agent Trajectory Planning
Aug 3, 2026Tianle Liu, Youcheng Niu, Jing Zeng +2KL-Regularized RL Game Theory
College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Jul 31, 2026Xujun Che, Yuchen Yuan, Weida Zhao +1Reinforcement Learning KL-Regularized RL
Department of Cybersecurity, University of North Carolina at Charlotte, Charlotte, NC, USA · Department of Information Sciences and Technology, George Mason University, Fairfax, VA, USA · Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA +1
$β$ -OPSD: Deriving with Policy Optimization, Training with Self-Distillation
Jul 30, 2026Jiawei Xu, Minghui Liu, Juzheng Zhang +2KL-Regularized RL On-Policy Self-Distillation
University of Maryland, College Park
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
Jul 29, 2026Keegan Harris, Brian W. Lee, Ian Waudby-Smith +3Reinforcement Learning KL-Regularized RL
University of California, Berkeley · Inria & École Normale Supérieure
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Jul 11, 2026Kexin Huang, Junkang Wu, Jinda Lu +7Reinforcement Learning KL-Regularized RL
University of Science and Technology of China · Peking University · Dartmouth College
REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation
Jul 11, 2026Li Guo, Anas M. Tahir, Z. Jane WangRadiology Report Generation Radiology VLMs
University of British Columbia, Vancouver, BC, Canada
On the Convergence of Self-Improving Online LLM Alignment
Jun 30, 2026Xudong Wu, Pangpang Liu, Vaneet Aggarwal +1LLM Alignment Optimization Convergence Analysis
The University of Hong Kong, Hong Kong SAR · Yale University, New Haven, CT, USA · Purdue University, West Lafayette, IN, USA
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology
Regularized Reward-Punishment Reinforcement Learning
Jun 26, 2026Jiexin Wang, Eiji UchibeReinforcement Learning KL-Regularized RL
Dept. of Brain Robot Interface, ATR Computational Neuroscience Laboratories, 2-2-2, Hikaridai, Seikacho, Soraku-gun, Kyoto, 619-0288, Japan
KLip-PPO: A per-sample KL perspective on PPO-Clip
Jun 22, 2026Riccardo Colletti, Robin HolzingerKL-Regularized RL Policy Gradient Methods
University of California, Berkeley
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reward Modeling KL-Regularized RL
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Jun 4, 2026Haoyang Hong, Zichen Wang, Quanquan Gu +1Regret Minimization in RL Reinforcement Learning
School of Electrical Engineering and Computer Science, Oregon State University · Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign · Department of Computer Science, University of California, Los Angeles
Trust Region Q Adjoint Matching
May 26, 2026Yonghoon Dong, Kyungmin Lee, Changyeon Kim +2Reinforcement Learning KL-Regularized RL
KAIST AI · Seoul National University · RLWRLD

Oct 19, 2026J/K move · Enter open · S save

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Haoqun Cao, Yurun Yuan, Tengyang Xie

University of Wisconsin–Madison

Policy Gradient Methods Off-Policy RL RL Post-Training KL-Regularized RL Importance Sampling for RL Offline RL

Open paper AskPDF arXiv

Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift. We show that for a general class of off-policy policy gradient updates, consistency, in the sense that the optimal policy remains a stationary point, requires one of two structures at the optimal policy: trajectory-level importance weighting, which suffers from the well-known curse of horizon, or KL regularization with a baseline equal to the optimal soft value, instantiated by the grouped mean. Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage. This also gives a theoretical account of a variant of the empirically successful GRPO objective. We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage. We instantiate this via the asymmetric REINFORCE objective, and connect it to the game-theoretic offline RL framework, showing a regret bound with single-policy coverage. Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.