Policy Learning

Follow this topic Ask about this topic

Momentum

79 papers in the last four weeks, up 316% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 416

All topics

Reversal Q-Learning
Jun 16, 2026Aditya Oberai, Seohong Park, Sergey LevineReinforcement Learning Q-Learning
University of California, Berkeley.
When Robots Sleep: Offline Skill Consolidation for Shared-Policy Robot Learning
Jun 16, 2026Nethmi Jayasinghe, Diana Gontero, Amit Ranjan TrivediRobot Policy Learning Continual Robot Learning
Department of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, IL, USA.
Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies
Jun 16, 2026Meipo Dai, Qiyuan Zhuang, He-Yang Xu +4Robot Policy Learning Robotic Manipulation
Southeast University · The Chinese University of Hong Kong
A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
Jun 15, 2026Jianghan Shen, Siqi Luo, Yue Li +9RL for Language Models Group Relative Policy Optimization
1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4
Learning Policy from a Single Trajectory in Average-Reward Markov Decision Process
Jun 15, 2026Jongmin Lee, Ernest K. Ryu, Vaneet AggarwalMarkov Decision Processes Reinforcement Learning
Seoul National University · UCLA · Purdue University
Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
Jun 15, 2026Swaminathan S K, Damiya Gondha, Theyanesh Eswaramoorthy Rajahkrishnan +1Contrastive RL Policy Learning
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India · Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India.
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
Jun 13, 2026Zirui Pang, Chenlong Zhang, Haosheng Tan +3Experience Replay Machine Unlearning
The Hong Kong University of Science and Technology (Guangzhou) · Institute of Automation, Chinese Academy of Sciences · University of Glasgow
Discovering Lattice Reduction Strategies via Self-Play
Jun 13, 2026Mohamed Malhou, Kristin Lauter, Ludovic PerretReinforcement Learning RL for Combinatorial Optimization
FAIR, Meta Superintelligence Labs & Sorbonne Université CNRS, LIP6 Paris, France · EPITA, EPITA Research Lab (LRE) Le Kremlin-Bicêtre, France
Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Jun 12, 2026Xinbei Ma, Congmin Zheng, Jiyang Qiu +11Reward Shaping LLM Agent Training
Shanghai Jiao Tong University · OPPO Research Institute
Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs
Jun 12, 2026Tianhao Wu, Matthew Zurek, Weina Wang +1Multi-Armed Bandits Robust Markov Decision Processes
Department of Industrial and Systems Engineering, University of Wisconsin-Madison · Department of Computer Sciences, University of Wisconsin-Madison · Computer Science Department, Carnegie Mellon University
Human-like autonomy emerges from self-play and a pinch of human data
Jun 11, 2026Daphne Cornelisse, Julian Hunt, Zixu Zhang +4Self-Play RL Autonomous Driving
1NYU Tandon School of Engineering · 2NYU Courant · 3Princeton University +2
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Jun 11, 2026Yifan Ruan, Chenyang Cao, Andreas Burger +7Inference-Time Optimization Diffusion Policy
University of Toronto · Vector Institute · LG Electronics
APPO: Agentic Procedural Policy Optimization
Jun 10, 2026Xucong Wang, Ziyu Ma, Yong Wang +5Agentic RL Credit Assignment in RL
University of Science and Technology of China · AMAP, Alibaba Group · Southern University of Science and Technology
PAWS: Preference Learning with Advantage-Weighted Segments
Jun 10, 2026Aleksandar Taranovic, Onur Celik, Niklas Freymuth +6Credit Assignment in RL Robotic RL
Autonomous Learning Robots, Karlsruhe Institute of Technology, Karlsruhe, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany · Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, Karlsruhe, Germany
Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems
Jun 10, 2026Xin Guo, Yijie Huang, Xiang YuPolicy Gradient Stochastic Optimal Control
Department of Industrial Engineering and Operations Research, University of California, Berkeley, USA. · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong.
MODIP: Efficient Model-Based Optimization for Diffusion Policies
Jun 9, 2026Zakariae El Asri, Philippe Gratias-Quiquandon, Nicolas Thome +1Robot Policy Learning Robot Policy Adaptation
Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Institut Universitaire de France (IUF)
Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training
Jun 9, 2026João Coelho, João Magalhães, Bruno Martins +1Reinforcement Learning Group Relative Policy Optimization
Language Technologies Institute, Carnegie Mellon University, United States · Instituto Superior Técnico and INESC-ID, University of Lisbon, Portugal · NOVA LINCS, NOVA School of Science and Technology, Portugal
Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updates
Jun 9, 2026Octave Oliviers, Glenn VinnicombeMarkov Decision Processes Reinforcement Learning
Department of Engineering University of Cambridge Cambridge, UK
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
Jun 8, 2026Kaustubh Mani, Yann Pequignot, Vincent Mai +1Sharpness-Aware Minimization RL Exploration
Universit´e de Montr´eal · Mila - Qu´ebec AI Institute · Universit´e Laval +2
Discovering Interpretable Multi-Parameter Control Policies for Evolutionary Algorithms Using Deep Reinforcement Learning
Jun 8, 2026Tai Nguyen, Phong Le, Carola Doerr +1Evolutionary Optimization Deep Q-Learning
University of St Andrews, St Andrews, United Kingdom · Sorbonne Universit´e, CNRS, LIP6, Paris, France
An Agency-Transferring Model-Free Policy Enhancement Technique
Jun 8, 2026Anton Bolychev, Georgiy Malaniya, Sinan Ibrahim +1RL Control Transfer Learning
Center for Engineering Systems and Sciences · Central University; Center for Engineering Systems and Sciences; Sirius University of Science and Technology
Difference-Aware Retrieval Policies for Imitation Learning
Jun 8, 2026Quinn Pfeifer, Ethan Pronovost, Paarth Shah +3Imitation Learning Interactive Imitation Learning
1Paul G. Allen School of Computer Science & Engineering, University of Washington · 2Toyota Research Institute · 3Google DeepMind +1
3SPO: State-Score-Supervised Policy Optimization for LLM Agents
Jun 8, 2026Yu Han, Kailing Li, Yang Jiao +4Credit Assignment in RL Step-Level Credit Assignment in RL
School of Computer Science and Technology, East China Normal University · Fudan University · KAUST +1
PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment
Jun 8, 2026Yang Tian, Rui Wang, Xumeng Wen +5Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards
School of AI, Shanghai Jiao Tong University · 2XYZ AI Lab
Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generation
Jun 5, 2026Raphael C Kim, Jingsen Zhu, Ramin Zabih +1Reinforcement Learning Causal Counterfactual Generation
Cornell Tech, Cornell University, New York, NY · Department of Biostatistics, Department of Population Health, New York University Grossman School of Medicine, New York, NY
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
Jun 4, 2026Paavo Parmas, Yongmin Kim, Kohsei Matsutani +5Stochastic Optimization Risk-Sensitive RL
The University of Tokyo
On Advantage Estimates for Max@K Policy Gradients
Jun 4, 2026Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6Policy Gradient Policy Gradient Methods
The University of Tokyo
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
Jun 4, 2026Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili +1Group Relative Policy Optimization Instruction Following
Department of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Department of Statistics, Mathematics and Computer Science, Allameh Tabataba’i University, Tehran, Iran
Retry Policy Gradients in Continuous Action Spaces
Jun 4, 2026Soichiro Nishimori, Paavo ParmasPolicy Optimization RL Exploration
The University of Tokyo, Japan
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
Jun 4, 2026Yuanfan Li, Qi Zhou, Wenjing Duan +1Credit Assignment in RL Step-Level Credit Assignment in RL
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University

Jun 16, 2026, cs.LGJ/K move · Enter open · S save

Reversal Q-Learning

Aditya Oberai, Seohong Park, Sergey Levine

University of California, Berkeley.

Reinforcement Learning Q-Learning Offline RL Flow-Based Generative Modeling Policy Learning Off-Policy RL

Open paper AskPDF arXiv

Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.

Momentum

Related topics

Latest papers 416