Off-Policy RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 74

All topics

Q-Learning with Scalar Adjoint Matching
Oct 7, 2026Yonghoon Dong, Minsung Yoon, Jaehyuk Kim +3Q-Learning Off-Policy RL
KAIST · RLWRLD
COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RL RL Post-Training
Fudan University · Meituan · East China Normal University
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
QF3: Fast Flow RL with Filtered Q-Gradients
Oct 6, 2026Chung Min Kim, Brent Yi, David McAllister +7Robotic RL RL for Legged Locomotion
UC Berkeley · Amazon FAR
Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning
Oct 4, 2026Bingyun Liu, Yuheng JingHierarchical RL Temporal-Difference Learning
Institute of Automation Chinese Academy of Sciences
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RL RL for Language Model Reasoning
The Ohio State University · Arizona State University
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience Replay Proximal Policy Optimization
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse
Sep 30, 2026Nima H. SiboniImportance Sampling Proximal Policy Optimization
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Sep 29, 2026Doohyuk Jang, Yoonsik Park, Gyouk Chu +2RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
KAIST · AITRICS
ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning
Sep 28, 2026Yihang Chen, Yuanhao Ban, Cho-Jui HsiehPolicy Optimization Reinforcement Learning with Verifiable Rewards
Department of Computer Science, University of California, Los Angeles
Limiting-Kernel Q( $λ$ ): Bridging Short and Long Horizons
Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function Estimation Policy Evaluation
Delft University of Technology, The Netherlands · Alpha Brain Technologies, The Netherlands · University of Toronto, Canada
On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
Sep 22, 2026Baptiste Bonin, Caro Strickland, Audrey DurandMulti-Objective Reinforcement Learning Experience Replay
Mila (Quebec AI Institute) · Université Laval · CIFAR AI Chair
Score Centering Stabilizes Off-policy Reinforcement Learning
Sep 17, 2026Martin Marek, Max RyabininRL for Language Models Off-Policy RL
Together AI
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Sep 17, 2026Xiang Zou, Shengzhu Shi, Junqi Gao +1Temporal-Difference Learning Actor-Critic Methods
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
MInTRL: Off-policy Intervention can boost On-policy RL
Sep 14, 2026Mingyu Chen, Yefan Tao, Gerald Friedland +2Reinforcement Learning Reinforcement Learning with Verifiable Rewards
Amazon Web Services · Boston University
Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Sep 14, 2026Tong Zheng, Xidong Wu, Zheng Zhang +14RL Exploration Self-Improving Agents
Google · University of Maryland, College Park · Google Deepmind +1
BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
Sep 9, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +3Asynchronous RL Off-Policy RL
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +1
Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving
Aug 11, 2026Jiazhuo Li, Linjiang Cao, Qi Liu +1World Models for Autonomous Driving RL for Autonomous Driving
Tongji University
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
Aug 3, 2026Abdelghani Ghanem, Mounir GhoghoQ-Learning Temporal-Difference Learning
College of Computing Mohammed VI Polytechnic University Rabat 11103, Morocco
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
Aug 3, 2026Wonseok Lee, Jimyeong Kim, Jungmin Ko +1RL for Language Model Reasoning RL for Diffusion Models
Interdisciplinary Program in Artificial Intelligence, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, South Korea · Artificial Intelligence Institute, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, South Korea · Department of Intelligence and Information, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, South Korea
Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
Aug 2, 2026Wenhao Zhang, Yibo Xie, Rui Wang +9Reinforcement Learning RL for Language Model Reasoning
1Tencent · 2Harbin Institute of Technology · 3Jinan University +1
Gated Q-learning: Add Off-Policy Bias to Taste
Jul 31, 2026Brett DaleyCredit Assignment in RL Step-Level Credit Assignment in RL
Meta
Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Jul 29, 2026Gong Gao, Xiao Lai, Ziqi Xie +3Reinforcement Learning Actor-Critic Methods
School of Computer Science and Technology, Tongji University, 200092, Shanghai, China · Shanghai University of Engineering Science, 201620, Shanghai, China
Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning
Jul 26, 2026Minh Vu, Konstantinos SlavakisGaussian Mixture Models Interpretability in RL
Institute of Science Tokyo, Department of Information and Communications Engineering, Yokohama, Japan.
Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning
Jul 24, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +5Asynchronous RL RL for Language Model Reasoning
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +2
Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Jul 24, 2026Yuta Natsubori, Masataka Ushiku, Yuta SaitoOff-Policy Evaluation Contextual Bandits
Hakuhodo DY Holdings, Inc. · Cornell University
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RL Asynchronous RL
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Jul 6, 2026Yu Li, Xiuyu Li, Mingyang Yi +4LLM Alignment Importance Sampling
Renmin University of China
Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning
Jul 5, 2026Faid Keddouri, Sohaib Houhou, Aissa Boulmerka +1Reward Shaping LLM-Guided RL
National School of Artificial Intelligence (ENSIA), Sidi Abdellah Campus, Algiers, Algeria · Cosys-Grettia, Univ Gustave Eiffel, F-77454 Marne-la-Vallee, France.
The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Jun 28, 2026Jing Liang, Hongyao Tang, Yi Ma +9Reinforcement Learning RL for Language Model Reasoning
1Tianjin · 2Alibaba

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Q-Learning with Scalar Adjoint Matching

Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park+2

KAIST · RLWRLD

Q-Learning Off-Policy RL

Open paper AskPDF arXiv

Flow policies capture rich and diverse action distributions, and fine-tuning them with off-policy RL to improve beyond the demonstrations has drawn growing interest. However, fine-tuning a flow policy against a learned value function is not trivial, because the policy generates its action over many flow steps. Adjoint matching offers a principled way to update the flow model itself by propagating value information from the final action back to each flow step, but it requires a vector--Jacobian product through the policy at every step, a cost that grows with the number of flow steps and the policy size. We observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal. Motivated by this finding, we derive a closed-form scalar adjoint that scales the value gradient at the final action by the flow time, eliminating the per-step vector--Jacobian products. We further find that controlling the critic's value at policy-generated actions is particularly important under the scalar adjoint. Based on these findings, we propose Q-learning with Scalar Adjoint Matching (SQAM), which combines the scalar adjoint with a value penalty at those actions. SQAM's gains concentrate on the four hardest OGBench domains, where its success rate exceeds that of the strongest baseline in each domain by 18 to 35 percentage points. To test whether SQAM extends to large pretrained policies, we also fine-tune a vision-language-action policy on a real bimanual robot. SQAM improves over supervised fine-tuning on all three tasks.

Momentum

Related topics

Latest papers 74