Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

May 17, 2022

Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
Kuan Fang, Patrick Yin, Ashvin Nair, Sergey Levine
Latent Space Offline Reinforcement Learning Long Horizon Task Goal Conditioned Reinforcement Learning Pseudo Goal Goal Conditioned Policy Online Fine Tuning Model Free Policy

May 6, 2022

How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
Alex X. Lee, Coline Devin, Jost Tobias Springenberg, Yuxiang Zhou, Thomas Lampe, Abbas Abdolmaleki, Konstantinos Bousmalis
Reinforcement Learning Offline Reinforcement Learning Reinforcement Learning Algorithm Vision Based Sub Optimal Policy Action Distribution Manipulation Benchmark Robot Time

May 5, 2022

Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning
Boxiang Lyu, Zhaoran Wang, Mladen Kolar, Zhuoran Yang
Markov Decision Process Offline Reinforcement Learning Mechanism Design Optimal Mechanism Offline RL Algorithm

April 26, 2022

April 18, 2022

CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
Siddharth Verma, Justin Fu, Mengjiao Yang, Sergey Levine
Language Model Reinforcement Learning Offline Reinforcement Learning Task Oriented Dialogue Agent AI Chatbots

April 12, 2022

When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
Aviral Kumar, Joey Hong, Anikait Singh, Sergey Levine
Imitation Learning Offline Reinforcement Learning Behavior Cloning

April 11, 2022

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, Yuting Wei
Offline Reinforcement Learning Sample Complexity Finite Horizon Model Based Offline Reinforcement Learning Infinite Horizon

April 7, 2022

Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes
Harry Emerson, Matthew Guy, Ryan McConville
Offline Reinforcement Learning Person Name Online Reinforcement Learning Blood Glucose Glucose Control

March 25, 2022

Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps
Jinglin Chen, Nan Jiang
Offline Reinforcement Learning Real Power Optimal Policy Filling Gap Net Present Value Density Ratio Multiple Realizability

March 24, 2022

Bellman Residual Orthogonalization for Offline Reinforcement Learning
Andrea Zanette, Martin J. Wainwright
Reinforcement Learning Offline Reinforcement Learning Policy Evaluation Policy Optimization Bellman Equation Bellman Residual

March 21, 2022

Optimizing Trajectories for Highway Driving with Offline Reinforcement Learning
Branka Mirchevska, Moritz Werling, Joschka Boedecker
Autonomous Driving Autonomous Vehicle Offline Reinforcement Learning Highway Driving Efficient Trajectory Continuous Time Trajectory Minimum Time Trajectory Trajectory Length

March 17, 2022

Semi-Markov Offline Reinforcement Learning for Healthcare
Mehdi Fatemi, Mary Wu, Jeremy Petch, Walter Nelson, Stuart J. Connolly, Alexander Benz, Anthony Carnicelli, Marzyeh Ghassemi
Reinforcement Learning Markov Decision Process Offline Reinforcement Learning Optimal Policy Healthcare System Offline Reinforcement Learning Algorithm

March 16, 2022

March 14, 2022

The Efficacy of Pessimism in Asynchronous Q-Learning
Yuling Yan, Gen Li, Yuxin Chen, Jianqing Fan
Offline Reinforcement Learning Q Learning State Action Pair Asynchronous Q Learning

March 13, 2022

DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
Jinxin Liu, Hongyin Zhang, Donglin Wang
Offline Reinforcement Learning Reward Function Offline Deep Reinforcement Learning

March 11, 2022

Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism
Ming Yin, Yaqi Duan, Mengdi Wang, Yu-Xiang Wang
Offline Reinforcement Learning Value Iteration Variance Information Linear Representation Offline Learning

March 3, 2022

Testing Stationarity and Change Point Detection in Reinforcement Learning
Mengbing Li, Chengchun Shi, Zhenke Wu, Piotr Fryzlewicz
Reinforcement Learning Offline Reinforcement Learning Change Point Detection Non Stationarity Nonstationary Environment Testing Stationarity

March 2, 2022

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, Esther Luna Colombini
Reinforcement Learning Timely Survey Offline Reinforcement Learning Narrative Review Comprehensive Taxonomy Robotic Agent Open Problem