Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

August 15, 2023

Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World
Nico Gürtler, Felix Widmaier, Cansu Sancaktar, Sebastian Blaes, Pavel Kolev, Stefan Bauer, Manuel Wüthrich, Markus Wulfmeier, Martin Riedmiller, Arthur Allshire, Qiang Wang, Robert McCarthy, Hangyeol Kim, Jongchan Baek, Wookyong Kwon, Shanliang Qian, Yasunori Toshimitsu, Mike Yan Michelis, Amirhossein Kazemipour, Arman Raayatsanati, Hehui Zheng, Barnabas Gavin Cangan, Bernhard Schölkopf, Georg Martius
Offline Reinforcement Learning Real World Real Robot Dexterous Manipulation Offline Data Robotic Datasets Robot Challenge

August 11, 2023

Learning Control Policies for Variable Objectives from Offline Data
Marc Weber, Phillip Swazinna, Daniel Hein, Steffen Udluft, Volkmar Sterzing
Offline Reinforcement Learning Control Policy Offline Data Control Strategy Time Varying Objective Model Based Policy Optimization Target

August 9, 2023

Bayesian Inverse Transition Learning for Offline Settings
Leo Benac, Sonali Parbhoo, Finale Doshi-Velez
Offline Reinforcement Learning Uncertainty Estimation Inverse Reinforcement Learning Sequential Decision Near Optimal Policy Offline Preference High Performing Policy

August 7, 2023

July 28, 2023

Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Bernhard Schölkopf, Georg Martius
Offline Reinforcement Learning Real World Dexterous Manipulation Offline Reinforcement Learning Algorithm

July 25, 2023

Offline Reinforcement Learning with On-Policy Q-Function Regularization
Laixi Shi, Robert Dadashi, Yuejie Chi, Pablo Samuel Castro, Matthieu Geist
Offline Reinforcement Learning Estimation Task Q Function Policy Regularization Poor Extrapolation Performance

July 23, 2023

On the Effectiveness of Offline RL for Dialogue Response Generation
Paloma Sodhi, Felix Wu, Ethan R. Elenberg, Kilian Q. Weinberger, Ryan McDonald
Language Model Offline Reinforcement Learning Human Language Dialogue Response Generation

July 21, 2023

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan
Multi Agent Reinforcement Learning Multi Agent Offline Reinforcement Learning Implicit Regularization Action Free Offline Offline Policy Offline Multi Agent Reinforcement Learning Multi Agent MuJoCo

July 13, 2023

Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
Marcel Hussing, Jorge A. Mendez, Anisha Singrodia, Cassandra Kent, Eric Eaton
Offline Reinforcement Learning Compositional Reinforcement Learning Manipulation Datasets

July 12, 2023

Budgeting Counterfactual for Offline RL
Yao Liu, Pratik Chaudhari, Rasool Fakoor
Offline Reinforcement Learning Dynamic Programming Counterfactual Reasoning Extrapolation Framework Counterfactual Path Counterfactual Plan

July 10, 2023

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
Suzan Ece Ada, Erhan Oztop, Emre Ugur
Offline Reinforcement Learning Distribution Generalization Behavior Cloning Offline Reinforcement Learning Algorithm Diffusion Policy

July 7, 2023

Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
Zilai Zeng, Ce Zhang, Shijie Wang, Chen Sun
Offline Reinforcement Learning Predictive Coding Goal Conditioned Policy Goal Conditioned

July 6, 2023

Offline Reinforcement Learning with Imbalanced Datasets
Li Jiang, Sijie Cheng, Jielin Qiu, Haoran Xu, Wai Kin Chan, Zhao Ding
Offline Reinforcement Learning Imbalanced Datasets Conservative Q Learning

June 24, 2023

June 23, 2023

June 22, 2023

Offline Reinforcement Learning

Papers

Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World

Learning Control Policies for Variable Objectives from Offline Data

Bayesian Inverse Transition Learning for Offline Settings

Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations

AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning

Benchmarking Offline Reinforcement Learning on Real-Robot Hardware

Offline Reinforcement Learning with On-Policy Q-Function Regularization

On the Effectiveness of Offline RL for Dialogue Response Generation

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning

Budgeting Counterfactual for Offline RL

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

Goal-Conditioned Predictive Coding for Offline Reinforcement Learning

Offline Reinforcement Learning with Imbalanced Datasets

Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching

Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets

Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation

CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning

Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning