Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

March 2, 2022

Reliable validation of Reinforcement Learning Benchmarks
Matthias Müller-Brockhausen, Aske Plaat, Mike Preuss
Reinforcement Learning Offline Reinforcement Learning Experimental Validation Reinforcement Learning Benchmark Game AI Reinforcement Learning Environment Interpretable Reward

February 28, 2022

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Yuejie Chi
Reinforcement Learning Offline Reinforcement Learning Sample Complexity Q Learning Finite Horizon Near Optimal Policy

February 26, 2022

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, Rui Song
Reinforcement Learning Offline Reinforcement Learning Policy OpTimization Policy Optimization New Horizon Function Estimation Advantage Learning

February 23, 2022

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, Zhaoran Wang
Offline Reinforcement Learning Bootstrapping End to End Linear MDPs Policy Algorithm Robust Offline Reinforcement Learning

February 19, 2022

A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, Mingyuan Zhou
Offline Reinforcement Learning BEHAVIOR Explanation State Action Pair Policy Based Reinforcement Learning Implicit Policy

February 17, 2022

Retrieval-Augmented Reinforcement Learning
Anirudh Goyal, Abram L. Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mehdi Mirza, Peter C. Humphreys, Ksenia Konyushkova, Laurent Sifre, Michal Valko, Simon Osindero, Timothy Lillicrap, Nicolas Heess, Charles Blundell
Deep Reinforcement Learning Offline Reinforcement Learning DQN Agent Policy Parameterization

February 13, 2022

Supported Policy Optimization for Offline Reinforcement Learning
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, Mingsheng Long
Reinforcement Learning Offline Reinforcement Learning Policy OpTimization Policy Reinforcement Learning Constrained Policy Optimization Support Estimation

February 11, 2022

Online Decision Transformer
Qinqing Zheng, Amy Zhang, Aditya Grover
Offline Reinforcement Learning Decision Transformer Sequence Modeling

February 10, 2022

Settling the Communication Complexity for Distributed Offline Reinforcement Learning
Juliusz Krysztof Ziomek, Jun Wang, Yaodong Yang
Offline Reinforcement Learning Contextual Bandit Offline Reinforcement Learning Algorithm Communication Complexity Episodic Markov Decision Process Minimax Rate

February 9, 2022

February 7, 2022

Model-Based Offline Meta-Reinforcement Learning with Regularization
Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, Junshan Zhang
Offline Reinforcement Learning Policy Regularization Offline Meta Reinforcement Learning

February 6, 2022

Stochastic Gradient Descent with Dependent Data for Offline Reinforcement Learning
Jing Dong, Xin T. Tong
Reinforcement Learning Stochastic Gradient Descent Offline Reinforcement Learning Q Learning Policy Learning Policy Iteration Dependent Data

February 5, 2022

Adversarially Trained Actor Critic for Offline Reinforcement Learning
Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal
Offline Reinforcement Learning Deep RL Stackelberg Game Public Policy Offline RL Algorithm

February 4, 2022

Offline Reinforcement Learning for Mobile Notifications
Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, Prakruthi Prabhakar
Offline Reinforcement Learning Large Scale Decision Notification System Mobile Notification

February 3, 2022

How to Leverage Unlabeled Data in Offline Reinforcement Learning
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, Sergey Levine
Offline Reinforcement Learning Unlabeled Data Robot Locomotion

January 31, 2022

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, Lerrel Pinto
Practical Algorithm Raw Data Offline Reinforcement Learning Policy Reinforcement Learning Offline Reinforcement Learning Algorithm Exploratory Data Analysis

January 28, 2022

Can Wikipedia Help Offline Reinforcement Learning?
Machel Reid, Yutaro Yamada, Shixiang Shane Gu
Reinforcement Learning Fine Tuning Offline Reinforcement Learning Sequence to Sequence Sequence Modeling Wikipedia Article Generative Task

January 27, 2022

The Challenges of Exploration for Offline Reinforcement Learning
Nathan Lambert, Markus Wulfmeier, William Whitney, Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi, Tim Hertweck, Martin Riedmiller
Reinforcement Learning Technical Challenge Offline Reinforcement Learning Environment Exploration Data Efficient