Off-Policy RL

RL: Reinforcement Learning

Off-policy reinforcement learning lets an agent learn from replay buffers, logged datasets, demonstrations, or rollouts produced by older or different policies, reducing the need for fresh interaction. Common approaches include importance sampling to correct policy mismatch, replay prioritization, multi-step and critic-based value learning, and conservative updates that limit errors from unsupported actions. The field spans continuous control and robotics as well as language-model training, where reusing stale or asynchronous rollouts can cut expensive generation and interaction costs. Its central tension is how much past data to reuse: more reuse improves efficiency, but staleness, high-variance corrections, and distribution shift can destabilize learning or erase useful training signals.

Latest papers

125 papers31 papers in the last four weeks, up 288% on the four weeks before. 0.2% of all new papers.

Echo: Scaling Visual Reinforcement Learning with Off-policy Experience
Oct 27, 2026Gagan KhandateVisual RL Robotic RL
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Oct 22, 2026Xuyang Zhong, Jiuding Yang, Yiwen Guo +1RL for Language Model Reasoning Policy Optimization
City University of Hong Kong · LIGHTSPEED · Independent Researcher
EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
Oct 21, 2026Chengxi Li, Hanlin Wu, Zhicheng Cai +4RL for Language Model Reasoning Proximal Policy Optimization
Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
Oct 20, 2026Chaorui Yao, Yuming Paul Zhang, Lin Yang +1Continuous-Time RL Model-Free RL
Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.
Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RL Model-Free RL
Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University
Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
Generative Policy Improvement from Observational Data
Oct 16, 2026Jonas Schweisthal, Dennis Frauen, Stefan Feuerriegel +1Offline RL RL for Generative Models
LMU Munich · Munich Center for Machine Learning (MCML) · Department of Computer Science, Stanford University
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Oct 16, 2026Heming Zou, Qi Wang, Yixiu Mao +7RL for LLM Agents Long-Horizon RL
Tsinghua University · Foundation Model Department, Tencent
Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
Oct 15, 2026Liangyao Jin, Zhixin Wang, Liming Liu +8Reinforcement Learning Off-Policy RL
Peking University · Infrawaves Inc · Shanghai Innovation Institute +2
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Oct 14, 2026Xuanchang Zhang, Tong ZhangReinforcement Learning with Verifiable Rewards Off-Policy RL
University of Illinois Urbana-Champaign
On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference Learning RL Stability
The University of Tokyo · Sun Intelligence · The University of Tokyo & RIKEN
Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
Oct 13, 2026Hao Bai, Jiarui Yao, Tong ZhangOff-Policy RL Reinforcement Learning with Verifiable Rewards
University of Illinois Urbana-Champaign
ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
Oct 13, 2026Changyeon Kim, Kimin Lee, Yashraj Narang +1Continual Robot Learning Off-Policy RL
NVIDIA · KAIST
From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning
Oct 13, 2026Yingjun Mo, Yajie Zhao, Fang KongIn-Context RL Transformer-Based RL
Southern University of Science and Technology (SUSTech) · University of Electronic Science and Technology of China (UESTC)
Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling
Oct 10, 2026Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu +4RL for Language Models RL Post-Training
LinkedIn
Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
Oct 9, 2026Xi Chen, Ruixue Ding, Zhigaoyuan Wang +9RL for LLM Agents RL Post-Training
Alibaba Token Hub, Alibaba Group · HKUST (Guangzhou) · Amap, Alibaba Group
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Oct 9, 2026Zhicheng Cai, Hanlin Wu, Weinan Dai +4Long-Horizon RL Reinforcement Learning
Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
Oct 9, 2026Du Cheng, Junyu Lu, Jingfan Zhang +3Policy Optimization RL for Language Model Reasoning
Q-Learning with Scalar Adjoint Matching
Oct 7, 2026Yonghoon Dong, Minsung Yoon, Jaehyuk Kim +3Q-Learning Off-Policy RL
KAIST · RLWRLD
COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RL RL Post-Training
Fudan University · Meituan · East China Normal University
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
QF3: Fast Flow RL with Filtered Q-Gradients
Oct 6, 2026Chung Min Kim, Brent Yi, David McAllister +7Robotic RL RL for Legged Locomotion
UC Berkeley · Amazon FAR
Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning
Oct 4, 2026Bingyun Liu, Yuheng JingHierarchical RL Temporal-Difference Learning
Institute of Automation Chinese Academy of Sciences
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Oct 3, 2026Nianli Peng, Geoffrey J. Gordon, Kianté BrantleyActor-Critic Methods Experience Replay
Harvard University · Carnegie Mellon University
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RL RL for Language Model Reasoning
The Ohio State University · Arizona State University
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience Replay Proximal Policy Optimization
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse
Sep 30, 2026Nima H. SiboniImportance Sampling Proximal Policy Optimization
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Sep 29, 2026Doohyuk Jang, Yoonsik Park, Gyouk Chu +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
KAIST · AITRICS
Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
Sep 29, 2026Zhiwei Wang, Yanxi Chen, Yaliang Li +1Policy Gradient Methods Off-Policy RL
Department of Mathematical Sciences Tsinghua University · Alibaba Group
Towards Better Training Signal: Advantage Clipped Policy Optimization
Sep 29, 2026Ruichuan Huang, Jinghan Liu, Congliang ChenPolicy Optimization RL for Language Model Reasoning
MIT · Shenzhen Loop Area Institute

Oct 27, 2026J/K move · Enter open · S save

Echo: Scaling Visual Reinforcement Learning with Off-policy Experience

Gagan Khandate

Visual RL Robotic RL Off-Policy RL Sample-Efficient RL

Open paper AskPDF arXiv

Scaling visual reinforcement learning (RL) with GPU-accelerated simulation is computationally expensive due to rendering and visual-policy inference costs. Consequently, two-stage teacher--student distillation is commonly used to train visual policies, but introduces an imitation gap. In this preliminary study, we introduce \echomethod{}, a single-stage off-policy RL framework that scales experience collection without proportionally increasing visual computation. Through inverse distillation, a state-based auxiliary policy imitates the evolving visual policy and collects inexpensive, unrendered experience. Echo leverages this experience for off-policy value learning, directly improving the visual policy through a shared privileged critic. We instantiate \echomethod{} with FastTD3 and evaluate it on three visual control tasks and two dexterous manipulation tasks in Genesis. Relative to \sdpgmethod{} under the evaluated configurations, Echo reduces time to reference returns by an average of 60% across the three locomotion tasks and 56% across both hand tasks, while improving peak mean returns by 62% and 12%, respectively. Both hand tasks also achieve higher mean returns at matched simulator-transition budgets. These initial results suggest that scaling off-policy experience is a promising approach to efficient direct visual RL.