Importance Sampling for RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Latest papers

16 papers6 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
Baselines and Score Centering in Policy Gradient Estimation
Oct 15, 2026Shuo Han, Ziqing HuPolicy Gradient Policy Gradient Methods
University of Illinois Chicago · Perplexity AI
Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling
Oct 10, 2026Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu +4RL for Language Models RL Post-Training
LinkedIn
Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
Oct 2, 2026Gabor Paczolay, Matteo Papini, Alberto Maria Metelli +2Policy Gradient Policy Gradient Methods
Politecnico di Milano. · Budapest University of Technology and Economics. · University of Milan.
Understanding Enrichment in Reinforcement Learning
Oct 2, 2026Jinwoo Kim, Shraddha BarkeImportance Sampling for RL Reinforcement Learning
University of California-San Diego La Jolla, CA, USA · Microsoft Research · Microsoft Research Redmond, WA, USA
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience Replay Proximal Policy Optimization
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse
Sep 30, 2026Nima H. SiboniImportance Sampling Proximal Policy Optimization
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany
Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
Aug 2, 2026Wenhao Zhang, Yibo Xie, Rui Wang +9Reinforcement Learning Importance Sampling for RL
1Tencent · 2Harbin Institute of Technology · 3Jinan University +1
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Jul 6, 2026Yu Li, Xiuyu Li, Mingyang Yi +4LLM Alignment Importance Sampling
Renmin University of China
What are Key Factors for Updates in RL for LLM Reasoning?
Jun 21, 2026Peidong Wang, Demi Wang, Xufang Luo +5Reinforcement Learning Importance Sampling for RL
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research +1
AIS: Adaptive Importance Sampling for Quantized RL
May 13, 2026Jiajun Zhou, Wei Shao, Lingchao Zheng +2RL for Language Models Reinforcement Learning
Huawei · The University of Hong Kong
Actor-Critic with Active Importance Sampling
May 8, 2026Majid Molaei, Gabor Paczolay, Matteo Papini +2Reinforcement Learning Importance Sampling for RL
Politecnico di Milano Milan, Italy
Beyond Importance Sampling: Rejection-Gated Policy Optimization
Apr 16, 2026Ziwu Sun, Zhen Gao, Jiyong Zhang +1Importance Sampling Policy Optimization
Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL
Mar 19, 2026Chenlu Ye, Xuanchang Zhang, Yifan Hao +6Importance Sampling for RL RL for Language Models
University of Illinois Urbana-Champaign · Amazon
GIPO: Gaussian Importance Sampling Policy Optimization
Mar 4, 2026Chengxuan Lu, Zhenquan Zhang, Shukuan Wang +3Reinforcement Learning Importance Sampling for RL
1IROOTECH TECHNOLOGY · 2Wolf 1069 b Lab, Sany Group
TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
Aug 4, 2025Lei Pang, Jun Luo, Ruinan JinImportance Sampling for RL RL for Language Models
Peking University Beijing, China · The Ohio State University Columbus, USA · Shugu Yuntai Technology Co., Ltd. +1

Oct 19, 2026J/K move · Enter open · S save

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Haoqun Cao, Yurun Yuan, Tengyang Xie

University of Wisconsin–Madison

Policy Gradient Methods Off-Policy RL RL Post-Training KL-Regularized RL Importance Sampling for RL Offline RL

Open paper AskPDF arXiv

Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift. We show that for a general class of off-policy policy gradient updates, consistency, in the sense that the optimal policy remains a stationary point, requires one of two structures at the optimal policy: trajectory-level importance weighting, which suffers from the well-known curse of horizon, or KL regularization with a baseline equal to the optimal soft value, instantiated by the grouped mean. Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage. This also gives a theoretical account of a variant of the empirically successful GRPO objective. We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage. We instantiate this via the asymmetric REINFORCE objective, and connect it to the game-theoretic offline RL framework, showing a regret bound with single-policy coverage. Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.