Off-Policy RL

RL: Reinforcement Learning

Off-policy reinforcement learning lets an agent learn from replay buffers, logged datasets, demonstrations, or rollouts produced by older or different policies, reducing the need for fresh interaction. Common approaches include importance sampling to correct policy mismatch, replay prioritization, multi-step and critic-based value learning, and conservative updates that limit errors from unsupported actions. The field spans continuous control and robotics as well as language-model training, where reusing stale or asynchronous rollouts can cut expensive generation and interaction costs. Its central tension is how much past data to reuse: more reuse improves efficiency, but staleness, high-variance corrections, and distribution shift can destabilize learning or erase useful training signals.

Latest papers

125 papers31 papers in the last four weeks, up 288% on the four weeks before. 0.2% of all new papers.

Efficient RL Training for LLMs with Experience Replay
Apr 9, 2026Charles Arnal, Vivien Cabannes, Taco Cohen +2Experience Replay RL Post-Training
FAIR at Meta · NYU Courant Institute and CDS
FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
Apr 6, 2026Donghu Kim, Youngdo Lee, Minho Park +10Off-Policy RL Robotic RL
Holiday Robotics · KAIST · KRAFTON +5
OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
Apr 5, 2026Liyu Zhang, Kehan Li, Tingrui Han +4RL for Generative Models Off-Policy RL
College of Control Science and Engineering, Zhejiang University · Central Research Institute, Huawei
D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation
Mar 28, 2026Yu Zhang, Karl MasonPrioritized Experience Replay Reinforcement Learning
School of Computer Science University of Galway Galway, Ireland
Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
Mar 25, 2026Guopeng Li, Matthijs T. J. Spaan, Julian F. P. KooijSafe RL Constrained RL
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, the Netherlands
Off-Policy Value-Based Reinforcement Learning for Large Language Models
Mar 24, 2026Peng-Yuan Wang, Ziniu Li, Tian Xu +8Reinforcement Learning RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
Off-Policy Evaluation and Learning for Survival Outcomes under Censoring
Mar 24, 2026Kohsuke Kubota, Mitsuhiro Takahashi, Yuta SaitoOff-Policy Evaluation Off-Policy RL
NTT DOCOMO, INC. · Hanjuku-kaso, Co, Ltd.
Delightful Distributed Policy Gradient
Mar 20, 2026Ian OsbandPolicy Gradient Methods Policy Learning
Google DeepMind
Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL
Mar 19, 2026Chenlu Ye, Xuanchang Zhang, Yifan Hao +6Importance Sampling for RL RL for Language Models
University of Illinois Urbana-Champaign · Amazon
Off-Policy Learning with Limited Supply
Mar 19, 2026Koichi Tanaka, Ren Kishimoto, Bushun Kawagishi +4Contextual Bandits Off-Policy RL
Institute of Science Tokyo Tokyo, Japan · Meiji University Tokyo, Japan · Yale University New Haven, CT, USA +2
Adapting Critic Match Loss Landscape Visualization to Off-policy Reinforcement Learning
Mar 15, 2026Jingyi Liu, Jian Guo, Eberhard GillLoss Landscape Geometry RL Control
Delft University of Technology, Delft, 2629 HS, the Netherlands
Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating
Mar 11, 2026Yuning Wu, Ke Wang, Haoran Liu +3Reinforcement Learning RL for Language Model Reasoning
Amazon
Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
Mar 9, 2026Riccardo De Monte, Matteo Cederle, Gian Antonio SustoReinforcement Learning Continuous Control
Department of Information Engineering, University of Padova
TADPO: Reinforcement Learning Goes Off-road
Mar 6, 2026Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3End-to-End Autonomous Driving Autonomous Driving
Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201
KARL: Knowledge Agents via Reinforcement Learning
Mar 5, 2026Jonathan D. Chang, Andrew Drozdov, Shubham Toshniwal +23Synthetic Data Generation for LLM Agents LLM Agent Training
Databricks AI Research
Reward-Conditioned Reinforcement Learning
Mar 5, 2026Michal Nauman, Marek Cygan, Pieter AbbeelReinforcement Learning Off-Policy RL
University of Warsaw · University of Warsaw, Nomagic · UC Berkeley, Amazon FAR
GIPO: Gaussian Importance Sampling Policy Optimization
Mar 4, 2026Chengxuan Lu, Zhenquan Zhang, Shukuan Wang +3Reinforcement Learning Importance Sampling for RL
1IROOTECH TECHNOLOGY · 2Wolf 1069 b Lab, Sany Group
RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
Mar 3, 2026Siwei Zhang, Yun Xiong, Xi Chen +4Agentic RL RL for Language Model Reasoning
Fudan University · Zhejiang University · UC Davis
SEAR: Sample Efficient Action Chunking Reinforcement Learning
Mar 2, 2026C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov +4Reinforcement Learning Sample-Efficient RL
Autonomous Learning Robots, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe
Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
Feb 27, 2026Yanwei Ren, Haotian Zhang, Likang Xiao +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · The Hong Kong Polytechnic University, Hong Kong SAR, China +3
Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Feb 26, 2026Zeyuan Liu, Jeonghye Kim, Xufang Luo +2RL for LLM Agents Reinforcement Learning
Microsoft Research · KAIST
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Feb 24, 2026Xu Wan, Yansheng Wang, Wenqi Huang +1RL for Language Model Reasoning RL Post-Training
Zhejiang University · Bytedance Seed Robotics · Peking University +1
LLMs Can Learn to Reason Via Off-Policy RL
Feb 22, 2026Daniel Ritter, Owen Oertell, Bradley Guo +3RL for Language Model Reasoning RL Post-Training
Cornell University · Databricks · Harvard University
Soft Sequence Policy Optimization
Feb 22, 2026Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman IschenkoLLM Alignment RL Post-Training
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia
Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies
Feb 20, 2026Zhuoran Li, Hai Zhong, Xun Wang +3Multi-Agent Reinforcement Learning Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · ByteDance
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Feb 19, 2026Luke J. Huang, Zhuoyang Zhang, Qinghao Hu +2RL for Language Models Asynchronous RL
MIT · NVIDIA
Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
Feb 18, 2026Ethan Blaser, Jiuqi Wang, Shangtong ZhangTemporal-Difference Learning Average-Reward RL
University of Virginia
Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Dec 28, 2025Yingru Li, Jiacai Liu, Jiawei Xu +4RL for Language Models Reinforcement Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7RL for Language Model Reasoning Importance Sampling
Texas A&M University · GE HealthCare · Independent Researcher +1

Apr 9, 2026J/K move · Enter open · S save

Efficient RL Training for LLMs with Experience Replay

Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe+1

FAIR at Meta · NYU Courant Institute and CDS

Experience Replay RL Post-Training LLM Post-Training Off-Policy RL Sample-Efficient RL

Open paper AskPDF arXiv

While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.