Off-Policy RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 74

All topics

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Temporal-Difference Learning
Stanford University
OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
May 4, 2026Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal +14RL for Robotics RL for Generative Models
Carnegie Mellon University · University of California, Berkeley · University of Washington +1
Learning from the Near Future: Temporal Self-Distillation for RLVR
Apr 22, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +6Reinforcement Learning with Verifiable Rewards Self-Distillation
Institute of Information Engineering, CAS · School of Cyber Security, UCAS
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
Apr 21, 2026Yuan Zhuang, Yuexin Bian, Sihong He +7Low-Rank Adaptation Off-Policy RL
University of Connecticut · University of California San Diego · University of Texas at Arlington +1
Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating
Mar 11, 2026Yuning Wu, Ke Wang, Haoran Liu +3Reinforcement Learning RL for Language Model Reasoning
Amazon
TADPO: Reinforcement Learning Goes Off-road
Mar 6, 2026Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3End-to-End Autonomous Driving Autonomous Driving
Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201
GIPO: Gaussian Importance Sampling Policy Optimization
Mar 4, 2026Chengxuan Lu, Zhenquan Zhang, Shukuan Wang +3Reinforcement Learning RL Post-Training
1IROOTECH TECHNOLOGY · 2Wolf 1069 b Lab, Sany Group
Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Dec 28, 2025Yingru Li, Jiacai Liu, Jiawei Xu +4RL for Language Models Reinforcement Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7RL for Language Model Reasoning Importance Sampling
Texas A&M University · GE HealthCare · Independent Researcher +1
Frictional Q-Learning
Sep 24, 2025Hyunwoo Kim, Hyo Kyung LeeReinforcement Learning Q-Learning
Data Analysis and Healthcare System Lab, Korea University, Seoul, Korea.
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2RL for Language Model Reasoning Efficient Language Model Training
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1
Transductive Off-policy Proximal Policy Optimization
Jun 6, 2024Yaozhong Gan, Renye Yan, Xiaoyang Tan +2Policy Optimization Proximal Policy Optimization
Independent Researcher · Peking University · Nanjing University of Aeronautics and Astronautics +1
Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
May 26, 2024Yilei Chen, Vittorio Giammarino, James Queeney +1Adversarial Imitation Learning Imitation Learning
Division of Systems Engineering Boston University Boston, MA 02215, USA · Department of Computer Science Purdue University West Lafayette, IN 47907, USA · Amazon Robotics North Reading, MA 01864, USA +1

May 7, 2026, cs.AIJ/K move · Enter open · S save

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities

Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn

Stanford University

Q-Learning Temporal-Difference Learning Off-Policy RL

Open paper AskPDF arXiv

Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.

Momentum

Related topics

Latest papers 74