Off-Policy RL

RL: Reinforcement Learning

Overview

Off-policy reinforcement learning lets an agent learn from replay buffers, logged datasets, demonstrations, or rollouts produced by older or different policies, reducing the need for fresh interaction. Common approaches include importance sampling to correct policy mismatch, replay prioritization, multi-step and critic-based value learning, and conservative updates that limit errors from unsupported actions. The field spans continuous control and robotics as well as language-model training, where reusing stale or asynchronous rollouts can cut expensive generation and interaction costs. Its central tension is how much past data to reuse: more reuse improves efficiency, but staleness, high-variance corrections, and distribution shift can destabilize learning or erase useful training signals.

Latest papers

CardsList

138 papers13 papers in the last four weeks, down 55% on the four weeks before. 0.1% of all new papers.

  1. MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

    Nov 18, 2026Seungyub Han, Taehyun Cho, Dohyeong Kim +2Temporal-Difference LearningOff-Policy RL

  2. Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts

    Nov 18, 2026Zhenrui Yue, Gongxiaohui Chen, Huimin Zeng +4Policy DistillationRL for Knowledge Distillation

  3. Delayed Heuristic-Behavior-Target Coupling for Off-Policy Reinforcement Learning in Inventory Control

    Nov 15, 2026Enhui Li, Yuan Cheng, Peiting HanReinforcement LearningOff-Policy RL

  4. Off-policy Learning with Excursion Policies

    Nov 9, 2026Jiamin He, Mark Rowland, Zhaohan Daniel Guo +2Off-Policy RLOff-Policy Evaluation

  5. Horizon-Scalable Q-Learning

    Nov 7, 2026Haoran Xu, Pragya Srivastava, Xinjie Liu +2Q-LearningTemporal-Difference Learning

  6. Diversity-Aware Experience Replay via Statistical Leverage Scores

    Nov 4, 2026Shaifalee Saxena, Alexander ScheinkerOff-Policy RLPrioritized Experience Replay

  7. Revisiting Critic Estimation in Soft Actor-Critic Algorithms

    Nov 3, 2026Jan Felix Kleuker, Koen Ponse, Aske Plaat +1Reinforcement LearningSoft Actor-Critic

  8. MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

    Oct 22, 2026Xuyang Zhong, Jiuding Yang, Yiwen Guo +1RL for Language Model ReasoningPolicy Optimization

  9. EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

    Oct 21, 2026Chengxi Li, Hanlin Wu, Zhicheng Cai +4RL for Language Model ReasoningProximal Policy Optimization

  10. A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

    Oct 20, 2026Chaorui Yao, Yuming Paul Zhang, Lin Yang +1Continuous-Time RLModel-Free RL

  11. Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

    Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RLModel-Free RL

  12. Generative Policy Improvement from Observational Data

    Oct 16, 2026Jonas Schweisthal, Dennis Frauen, Stefan Feuerriegel +1Offline RLRL for Generative Models

  13. Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

    Oct 16, 2026Heming Zou, Qi Wang, Yixiu Mao +7RL for LLM AgentsLong-Horizon RL

  14. Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

    Oct 15, 2026Liangyao Jin, Zhixin Wang, Liming Liu +8Reinforcement LearningOff-Policy RL

  15. On the Convergence of Semi-Gradient TD: A Mean Field Perspective

    Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference LearningRL Stability

  16. From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning

    Oct 13, 2026Yingjun Mo, Yajie Zhao, Fang KongIn-Context RLTransformer-Based RL

  17. Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

    Oct 9, 2026Xi Chen, Ruixue Ding, Zhigaoyuan Wang +9RL for LLM AgentsRL Post-Training

  18. Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL

    Oct 9, 2026Zhicheng Cai, Hanlin Wu, Weinan Dai +4Long-Horizon RLReinforcement Learning

  19. Q-Learning with Scalar Adjoint Matching

    Oct 7, 2026Yonghoon Dong, Minsung Yoon, Jaehyuk Kim +3Q-LearningOff-Policy RL

  20. COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

    Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RLRL Post-Training