Offline Policy Learning

Offline policy learning aims to train reinforcement learning agents using pre-collected datasets, avoiding the risks and costs of online interaction. Current research focuses on addressing challenges like data scarcity, distributional shifts, and the presence of suboptimal behaviors in the training data, employing techniques such as importance weighting regularization, skill-based abstractions, deep generative models, and dataset clustering to improve policy performance. These advancements are significant for applications where online learning is impractical or unsafe, such as healthcare, robotics, and safety-critical systems, enabling the development of robust and effective policies from existing data.

Papers

September 22, 2024

COSBO: Conservative Offline Simulation-Based Policy Optimization
Eshagh Kargar, Ville Kyrki
Offline Reinforcement Learning Policy OpTimization Reinforcement Learning Agent Long Term Cooperation Offline Policy Learning Reality Gap

August 27, 2024

Unsupervised-to-Online Reinforcement Learning
Junsu Kim, Seohong Park, Sergey Levine
Offline Reinforcement Learning Unsupervised Reinforcement Learning Offline Policy Learning Offline to Online Reinforcement Learning

August 21, 2024

Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks
Donghoon Kim, Minjong Yoo, Honguk Woo
Policy Learning Goal Conditioned Reinforcement Learning Goal Conditioned Offline Policy Learning Skill Abstraction

June 5, 2024

Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling
Imad Aouali, Victor-Emmanuel Brunel, David Rohde, Anna Korba
Generalization Bound Unified Alignment Importance Sampling PAC Bayesian Offline Policy Learning Weight Regularization

February 21, 2024

Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, Vaneet Aggarwal
Generative Adversarial Network Synthesized View Future Direction Tutorial Review Offline Policy Learning

February 16, 2024

Optimizing Warfarin Dosing Using Contextual Bandit: An Offline Policy Learning and Evaluation Method
Yong Huang, Charles A. Downs, Amir M. Rahmani
Reinforcement Learning Evaluation Method Based Policy Offline Policy Learning Contextual Bandit Setting Dosing Model Warfarin Dosing

February 14, 2024

Dataset Clustering for Improved Offline Policy Learning
Qiang Wang, Yixin Deng, Francisco Roldan Sanchez, Keru Wang, Kevin McGuinness, Noel O'Connor, Stephen J. Redmond
Diverse Behavior Multi Behavior Offline Policy Learning

January 19, 2024

Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu
Diffusion Model Safety Constraint Safe Policy Offline Policy Learning Offline Safe Reinforcement Learning

October 6, 2023

Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, Pulkit Agrawal
Reinforcement Learning Offline Reinforcement Learning Imbalanced Datasets Offline Reinforcement Learning Algorithm Uniform Sampling Offline Policy Learning

September 18, 2023

Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, Wei Zhan
Safe Reinforcement Learning Offline Policy Learning Offline Safe Reinforcement Learning Safe Reinforcement Learning Algorithm

March 30, 2023

MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
Anqi Li, Byron Boots, Ching-An Cheng
Imitation Learning Offline Reinforcement Learning Sequential Decision Making Evolving Observation Offline Imitation Offline Policy Learning

February 1, 2023

Selective Uncertainty Propagation in Offline RL
Sanath Kumar Krishnamurthy, Shrey Modi, Tanmay Gangwani, Sumeet Katariya, Branislav Kveton, Anshuka Rangi
Offline Reinforcement Learning Distribution Shift Contextual Bandit Dynamic Programming Uncertainty Propagation Offline Policy Learning

January 30, 2023

Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies
Qiang Wang, Robert McCarthy, David Cordova Bulens, Francisco Roldan Sanchez, Kevin McGuinness, Noel E. O'Connor, Stephen J. Redmond
Offline Reinforcement Learning Behavior Cloning Dexterous Robotic Manipulation Offline Policy Learning Expert Level Performance Robot Challenge Offline Dataset

January 27, 2023

Improving Behavioural Cloning with Positive Unlabeled Learning
Qiang Wang, Robert McCarthy, David Cordova Bulens, Kevin McGuinness, Noel E. O'Connor, Nico Gürtler, Felix Widmaier, Francisco Roldan Sanchez, Stephen J. Redmond
Behavior Cloning Control Policy Positive Unlabeled Learning Robotic Datasets Expert Trajectory Offline Policy Learning

October 12, 2022

Efficient Offline Policy Optimization with a Learned Model
Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng Yan, Zhongwen Xu
Monte Carlo Tree Search Offline Policy Learned Model Offline Policy Learning Atari Benchmark

July 26, 2022

Branch Ranking for Efficient Mixed-Integer Programming via Offline Ranking-based Policy Learning
Zeren Huang, Wenhao Chen, Weinan Zhang, Chuhan Shi, Furui Liu, Hui-Ling Zhen, Mingxuan Yuan, Jianye Hao, Yong Yu, Jun Wang
Offline Reinforcement Learning Mixed Integer Action Free Offline Strong Branching Offline Policy Learning Branch Prediction

November 27, 2021

Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization
Thanh Nguyen-Tang, Sunil Gupta, A. Tuan Nguyen, Svetha Venkatesh
Strong Generalization Optimization Purpose Stochastic Gradient Descent Policy OpTimization Offline Policy Learning Offline Contextual Bandit

November 8, 2021

Safe Data Collection for Offline and Online Policy Learning
Ruihao Zhu, Branislav Kveton
Policy Learning Optimal Regret Data Collection Action Free Offline Offline Policy Learning Policy Learning Method

Offline Policy Learning

Papers

COSBO: Conservative Offline Simulation-Based Policy Optimization

Unsupervised-to-Online Reinforcement Learning

Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks

Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling

Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions

Optimizing Warfarin Dosing Using Contextual Bandit: An Offline Policy Learning and Evaluation Method

Dataset Clustering for Improved Offline Policy Learning

Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets

Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration

MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations

Selective Uncertainty Propagation in Offline RL

Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies

Improving Behavioural Cloning with Positive Unlabeled Learning

Efficient Offline Policy Optimization with a Learned Model

Branch Ranking for Efficient Mixed-Integer Programming via Offline Ranking-based Policy Learning

Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization

Safe Data Collection for Offline and Online Policy Learning