Off-Policy RL

RL: Reinforcement Learning

Off-policy reinforcement learning lets an agent learn from replay buffers, logged datasets, demonstrations, or rollouts produced by older or different policies, reducing the need for fresh interaction. Common approaches include importance sampling to correct policy mismatch, replay prioritization, multi-step and critic-based value learning, and conservative updates that limit errors from unsupported actions. The field spans continuous control and robotics as well as language-model training, where reusing stale or asynchronous rollouts can cut expensive generation and interaction costs. Its central tension is how much past data to reuse: more reuse improves efficiency, but staleness, high-variance corrections, and distribution shift can destabilize learning or erase useful training signals.

Latest papers

125 papers31 papers in the last four weeks, up 288% on the four weeks before. 0.2% of all new papers.

Frictional Q-Learning
Sep 24, 2025Hyunwoo Kim, Hyo Kyung LeeReinforcement Learning Q-Learning
Data Analysis and Healthcare System Lab, Korea University, Seoul, Korea.
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2RL for Language Model Reasoning Efficient Language Model Training
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1
Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Exploration
Jun 3, 2025Tom Danino, Nahum ShimkinMulti-Agent Reinforcement Learning Sample-Efficient RL
The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion- Israel Institute of Technology
Transductive Off-policy Proximal Policy Optimization
Jun 6, 2024Yaozhong Gan, Xiaoyang Tan, Zhe Wu +1Policy Optimization Proximal Policy Optimization
Independent Researcher
Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
May 26, 2024Yilei Chen, Vittorio Giammarino, James Queeney +1Adversarial Imitation Learning Imitation Learning
Division of Systems Engineering Boston University Boston, MA 02215, USA · Department of Computer Science Purdue University West Lafayette, IN 47907, USA · Amazon Robotics North Reading, MA 01864, USA +1

Sep 24, 2025J/K move · Enter open · S save

Frictional Q-Learning

Hyunwoo Kim, Hyo Kyung Lee

Data Analysis and Healthcare System Lab, Korea University, Seoul, Korea.

Reinforcement Learning Q-Learning Policy Learning Off-Policy RL

Open paper AskPDF arXiv

Off-policy reinforcement learning suffers from extrapolation errors when a learned policy selects actions that are weakly supported in the replay buffer. In this study, we address this issue by drawing an analogy to static friction. From this perspective, the replay buffer is represented as a smooth, low-dimensional action manifold, where the support directions correspond to the tangential component, while the normal component captures the dominant first-order extrapolation error. This decomposition reveals an intrinsic anisotropy in value sensitivity that naturally induces a stability condition analogous to a friction threshold. To mitigate deviations toward unsupported actions, we propose Frictional Q-Learning, an off-policy algorithm that encodes supported actions as tangent directions using a contrastive variational autoencoder. We further show that an orthonormal basis of the orthogonal complement corresponds to normal components under mild local isometry assumptions. Extensive empirical results on standard continuous-control benchmarks consistently demonstrate robust and stable performance compared with competitive baselines.

Latest papers

Frictional Q-Learning

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Exploration

Transductive Off-policy Proximal Policy Optimization

Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees