KL-Regularized RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 36

All topics

On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
FERPO: Forward Entropy-Regularized Policy Optimization
Oct 1, 2026Sebastian Sanokowski, Alireza Sarmadi, Majid KhadivKL-Regularized RL Policy Optimization
Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy
Sep 17, 2026Luis LealNash Equilibrium Self-Play RL
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Sep 14, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouKL-Regularized RL Adversarial Robustness
Mohammed VI Polytechnic University · Khalifa University · Concordia University
Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning
Aug 3, 2026Li Wang, Xiaodong Lu, Xiaohan Wang +4Continual Learning for LLMs KL-Regularized RL
1Meituan · 2Nanyang Technological University
A Forward-Inverse Dynamic Game Framework for Enhanced Multi-Agent Trajectory Planning
Aug 3, 2026Tianle Liu, Youcheng Niu, Jing Zeng +2KL-Regularized RL Game Theory
College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Jul 31, 2026Xujun Che, Yuchen Yuan, Weida Zhao +1Reinforcement Learning KL-Regularized RL
Department of Cybersecurity, University of North Carolina at Charlotte, Charlotte, NC, USA · Department of Information Sciences and Technology, George Mason University, Fairfax, VA, USA · Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA +1
$β$ -OPSD: Deriving with Policy Optimization, Training with Self-Distillation
Jul 30, 2026Jiawei Xu, Minghui Liu, Juzheng Zhang +2KL-Regularized RL On-Policy Self-Distillation
University of Maryland, College Park
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
Jul 29, 2026Keegan Harris, Brian W. Lee, Ian Waudby-Smith +3Reinforcement Learning KL-Regularized RL
University of California, Berkeley · Inria & École Normale Supérieure
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Jul 11, 2026Kexin Huang, Junkang Wu, Jinda Lu +7Reinforcement Learning KL-Regularized RL
University of Science and Technology of China · Peking University · Dartmouth College
REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation
Jul 11, 2026Li Guo, Anas M. Tahir, Z. Jane WangRadiology Report Generation Radiology VLMs
University of British Columbia, Vancouver, BC, Canada
On the Convergence of Self-Improving Online LLM Alignment
Jun 30, 2026Xudong Wu, Pangpang Liu, Vaneet Aggarwal +1LLM Alignment Optimization Convergence Analysis
The University of Hong Kong, Hong Kong SAR · Yale University, New Haven, CT, USA · Purdue University, West Lafayette, IN, USA
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology
Regularized Reward-Punishment Reinforcement Learning
Jun 26, 2026Jiexin Wang, Eiji UchibeReinforcement Learning KL-Regularized RL
Dept. of Brain Robot Interface, ATR Computational Neuroscience Laboratories, 2-2-2, Hikaridai, Seikacho, Soraku-gun, Kyoto, 619-0288, Japan
KLip-PPO: A per-sample KL perspective on PPO-Clip
Jun 22, 2026Riccardo Colletti, Robin HolzingerKL-Regularized RL Policy Gradient Methods
University of California, Berkeley
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reward Modeling KL-Regularized RL
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Jun 4, 2026Haoyang Hong, Zichen Wang, Quanquan Gu +1Regret Minimization in RL Reinforcement Learning
School of Electrical Engineering and Computer Science, Oregon State University · Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign · Department of Computer Science, University of California, Los Angeles
Trust Region Q Adjoint Matching
May 26, 2026Yonghoon Dong, Kyungmin Lee, Changyeon Kim +2Reinforcement Learning KL-Regularized RL
KAIST AI · Seoul National University · RLWRLD
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanKL-Regularized RL RL for Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Wuhan University
How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
May 23, 2026Rei Higuchi, Ryotaro Kawata, Akifumi Wachi +3Reward Modeling Representation Learning
The University of Tokyo · RIKEN AIP · LY Corporation
Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
May 21, 2026Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2Multi-Objective Reinforcement Learning Intrinsic Reward Methods
Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2
Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
May 19, 2026Xiaozhe Li, Yang Li, Xinyu Fang +10KL-Regularized RL RL for Language Model Reasoning
1Tongji University, China · 6Independent · 2Shanghai AI Laboratory, China +3
Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Era
May 17, 2026Qiuhe Hong, Yuyang Liu, Shuo Yang +3KL-Regularized RL Continual Learning
1Shenzhen Graduate School of Peking University · Centre for Artificial Intelligence and Robotics, HKISI, CAS · 3Peng Cheng Laboratory
SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
May 15, 2026Chanuk Lee, Minki Kang, Sung Ju HwangKL-Regularized RL RL for Language Model Reasoning
KAIST
Offline Two-Player Zero-Sum Markov Games with KL Regularization
May 13, 2026Claire Chen, Yuheng Zhang, Xinyu Liu +3Nash Equilibrium Reinforcement Learning
1California Institute of Technology. · University of Illinois Urbana-Champaign. · University of Virginia. +2
The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives
May 12, 2026Ankur Moitra, Andrej Risteski, Dhruv RohatgiDiffusion Model Alignment KL-Regularized RL
expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling
May 11, 2026Mingxiong Lin, Zhangquan Gong, Maowen Tang +6Reinforcement Learning KL-Regularized RL
OPPO AI Center
Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
May 9, 2026Qingyue Zhao, Kaixuan Ji, Heyang Zhao +1KL-Regularized RL Contextual Bandits
Department of Computer Science, University of California, Los Angeles, CA 90095, USA
Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Reward Hacking LLM Alignment
Apple
Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
May 7, 2026Peiying Zhu, Sidi ChangReward Hacking Reinforcement Learning
Blossom AI · Blossom AI Labs

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

On KL-Regularized Policy Optimization

Yifan Zhang

Princeton University

RL for LLM Agents KL-Regularized RL Off-Policy RL Asynchronous RL Critic-Free RL

Open paper AskPDF arXiv

Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-

K

and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

Momentum

Related topics

Latest papers 36