Off-Policy RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 74

All topics

MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
Jun 24, 2026Roy Xing, Seyoung Ree, Brian PlancherReinforcement Learning RL for Legged Locomotion
Dartmouth College · Harvard University
What are Key Factors for Updates in RL for LLM Reasoning?
Jun 21, 2026Peidong Wang, Demi Wang, Xufang Luo +5Reinforcement Learning RL for Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research +1
Reversal Q-Learning
Jun 16, 2026Aditya Oberai, Seohong Park, Sergey LevineReinforcement Learning Q-Learning
University of California, Berkeley.
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
Jun 13, 2026Zirui Pang, Chenlong Zhang, Haosheng Tan +3Experience Replay Machine Unlearning
The Hong Kong University of Science and Technology (Guangzhou) · Institute of Automation, Chinese Academy of Sciences · University of Glasgow
Mitigating Bias in Low-SNR Financial Reinforcement Learning via Quantum Representations
Jun 9, 2026Zeyu Liu, Xuanzhi Feng, Sing Kwong Lai +6Quantitative Finance Quantum Machine Learning
The Hong Kong University of Science and Technology
Rethinking the Divergence Regularization in LLM RL
Jun 8, 2026Jiarui Yao, Xiangxin Zhou, Penghui Qi +3RL for Language Models Policy Optimization
1Tencent Hunyuan · 2UIUC · 3NUS
ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
Jun 2, 2026Zehua Liu, Yuxuan Yao, Xiaojin Fu +2Asynchronous RL Reinforcement Learning
Huawei Technologies
Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
Jun 1, 2026Christian Scherer, Joe Watson, Theo Gruner +3Reinforcement Learning Robot Skill Learning
Technical University of Darmstadt · Zuse School ELIZA · University of Oxford +2
RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
May 31, 2026Yixiu Mao, Yun Qu, Qi Wang +2Reinforcement Learning RL for Language Model Reasoning
Department of Automation, Tsinghua University
PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning
May 29, 2026Daize Dong, Junlin Chen, Haolong Jia +9RL for Language Model Reasoning Expert Routing
♣Rutgers University · ♦AMD · ⋆MBZUAI
Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
May 27, 2026Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali +1Reinforcement Learning RL for Language Model Reasoning
Criteo AI Lab · Criteo AI Lab, CREST IP Paris
Extreme Region Policy Distillation
May 25, 2026Changyu Chen, Xiting Wang, Rui YanKL-Regularized RL RL for Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Wuhan University
Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion
May 24, 2026Gianluca Sabatini, Chenhao Li, Marco HutterRL for Legged Locomotion Soft Actor-Critic
ETH Zurich, Switzerland
Goal-Conditioned Agents that Learn Everything All at Once
May 22, 2026Michael Matthews, Matthew Jackson, Michael Beukman +5Policy Learning Sample-Efficient RL
University of Oxford · 2McGill University · 3MIT +1
F-TIS: Harnessing Diverse Models in Collaborative GRPO
May 21, 2026Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer +1RL for Language Models Communication-Efficient Distributed Training
Gensyn · University of Neuchatel · TU Delft
Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
May 20, 2026Zeyuan Wang, Da Li, Yulin Chen +6Reinforcement Learning Maximum Entropy RL
Laboratory for Big Data and Decision, National University of Defense Technology, China · Samsung AI Center Cambridge, UK · Queen Mary University of London, UK +3
How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
May 17, 2026Minghao Tian, Yunfei Xie, Chen WeiRL for Language Model Reasoning Group Relative Policy Optimization
Rice University
Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target
May 17, 2026Taesan Kim, Hyeongjun Yun, Jaegul Choo +1Continual Learning for LLMs Recommender Systems
1SK Telecom · 2KAIST
Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Prediction
May 17, 2026Xingguo Chen, Zhiang He, Yuchen Shen +4Temporal-Difference Learning Linear Function Approximation
Nanjing University of Posts and Telecommunications, Nanjing, China · Department of Computer Science and Technology, Nanjing University, Nanjing, China · College of Electronic Countermeasure, National University of Defense Technology, Hefei, China
Achieving $ε^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions
May 13, 2026Ishaq Hamza, Zaiwei ChenPolicy Gradient Last-Iterate Convergence
IISc · Edwardson School of Industrial Engineering, Purdue University · Purdue IE
Trajectory-Level Data Augmentation for Offline Reinforcement Learning
May 13, 2026Tobias Schmähling, Matthias Burkhardt, Tobias WindischData Augmentation Offline RL
University of Applied Sciences Kempten, Kempten, Germany.
Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
May 12, 2026Rasool Fakoor, Murdock Aubry, Nicholas Stranges +1Importance Sampling Policy Optimization
Boson AI
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
May 12, 2026Zhong Guan, Yongjian Guo, Haoran Sun +5Asynchronous RL Reinforcement Learning
Tianjin University · Tsinghua University · JDT AI Infra +1
Debiased Model-based Representations for Sample-efficient Continuous Control
May 12, 2026Jiafei Lyu, Zichuan Lin, Scott Fujimoto +5Representation Learning Continuous Control
1Tencent Hunyuan · 2McGill University · School of Computer
Selective Off-Policy Reference Tuning with Plan Guidance
May 12, 2026Duc Anh Le, Tien-Phat Nguyen, Thien Huu Nguyen +2Reinforcement Learning RL for Language Model Reasoning
Independent Author · Hanoi University of Science and Technology, Hanoi, Vietnam · University of Oregon, Eugene, OR, USA +1
When Does Non-Uniform Replay Matter in Reinforcement Learning?
May 11, 2026Michal Korniak, Mikołaj Czarnecki, Yarden As +3Experience Replay Sample-Efficient RL
1ETH Zurich · University of Warsaw · 3UC Berkeley +1
Actor-Critic Algorithm for Dynamic Expectile and CVaR
May 8, 2026Yudong Luo, Erick DelageReinforcement Learning Risk-Sensitive RL
1GERAD & Department of Decision Sciences, HEC Montr´eal, Canada · 2Mila - Quebec AI Institute, Canada
Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
May 8, 2026Yuheng Zhang, Chenlu Ye, Shuowei Jin +4RL for Language Model Reasoning Importance Sampling
UIUC · University of Michigan · Amazon
Actor-Critic with Active Importance Sampling
May 8, 2026Majid Molaei, Gabor Paczolay, Matteo Papini +2Reinforcement Learning Policy Gradient Methods
Politecnico di Milano Milan, Italy
How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment
May 7, 2026Rui Zhu, Weiheng Bai, Qiushi Wu +3LLM Alignment KV-Cache Compression
Yale University · University of Minnesota Twin Cities · Indiana University Bloomington

Jun 24, 2026, cs.ROJ/K move · Enter open · S save

MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

Roy Xing, Seyoung Ree, Brian Plancher

Dartmouth College · Harvard University

Reinforcement Learning RL for Legged Locomotion Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred behavior by inserting transitions generated in the same environment by a model predictive controller (MPC). Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. We analyze how the injected transitions bias the learning, allowing the policy to converge to behaviors that pure RL may fail to reach under simple reward functions. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection produces gaits qualitatively comparable to those of reward shaping and adversarial motion priors. We also show that MPC-Injection can complete a barrel roll that pure RL fails to achieve under the same simple reward and can select between trotting and bounding gaits only through changing the injected MPC data.

Momentum

Related topics

Latest papers 74