Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

January 25, 2022

MOORe: Model-based Offline-to-Online Reinforcement Learning
Yihuan Mao, Chao Wang, Bin Wang, Chongjie Zhang
Offline Reinforcement Learning Performance Improvement Online Adaptation Model Based Offline Reinforcement Learning Causal Thread Sample Efficient Adaptation

January 17, 2022

An Improved Reinforcement Learning Algorithm for Learning to Branch
Qingyu Qu, Xijun Li, Yunfan Zhou, Jia Zeng, Mingxuan Yuan, Jie Wang, Jinhu Lv, Kexin Liu, Kun Mao
LeArning Abstract Offline Reinforcement Learning Multi Channel Branch Branch and Cut Double DQN

January 14, 2022

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning
Phillip Swazinna, Steffen Udluft, Daniel Hein, Thomas Runkler
Offline Reinforcement Learning Hybrid Approach Model Free Algorithm Simple Algorithm

January 7, 2022

Offline Reinforcement Learning for Road Traffic Control
Mayuresh Kunjir, Sanjay Chawla
Reinforcement Learning Offline Reinforcement Learning Traffic Signal Control Traffic Management Model Based Offline Reinforcement Learning

December 31, 2021

December 18, 2021

Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning
Giorgio Angelotti, Nicolas Drougard, Caroline P. C. Chanel
Data Augmentation Markov Decision Process Offline Reinforcement Learning System Performance Learned Model Symmetry Detection

December 2, 2021

Quantile Filtered Imitation Learning
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, Joan Bruna
Offline Reinforcement Learning Feature Imitation Push Forward

November 29, 2021

Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, Jonathan Tompson
Reinforcement Learning Contrastive Learning Zero Shot Strong Generalization Offline Reinforcement Learning Similarity Function

November 26, 2021

Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning
Phillip Swazinna, Steffen Udluft, Thomas Runkler
Offline Reinforcement Learning Data Quality Big Gain State of the Art Datasets Stochastic Action

November 23, 2021

The Impact of Data Distribution on Q-learning with Function Approximation
Pedro P. Santos, Diogo S. Carvalho, Alberto Sardinha, Francisco S. Melo
Global Impact Offline Reinforcement Learning Q Learning Dynamic Programming Function Approximation Data Distribution High Entropy

November 22, 2021

November 21, 2021

Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
Dylan J. Foster, Akshay Krishnamurthy, David Simchi-Levi, Yunzong Xu
Reinforcement Learning Offline Reinforcement Learning Value Function Knowledge Barrier Sample Efficient Offline Reinforcement Learning

November 16, 2021

Compressive Features in Offline Reinforcement Learning for Recommender Systems
Hung Nguyen, Minh Nguyen, Long Pham, Jennifer Adorno Nieves
Recommender System Offline Reinforcement Learning Q Learning High Dimension Compression Insensitive Feature Big Data Cup

November 15, 2021

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
Vincent Liu, James R. Wright, Martha White
Offline Reinforcement Learning Exogenous Variable Fitted Q Iteration

November 9, 2021

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan
Reinforcement Learning Offline Reinforcement Learning Distribution Shift

November 8, 2021

A Dataset Perspective on Offline Reinforcement Learning
Kajetan Schweighofer, Andreas Radler, Marius-Constantin Dinu, Markus Hofmarcher, Vihang Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, Sepp Hochreiter
Reinforcement Learning Offline Reinforcement Learning Deep Q Network Data Perspective

November 4, 2021

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning
Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, Olivier Pietquin, Nikola Momchev
Reinforcement Learning Data Set Imitation Learning Offline Reinforcement Learning Generate Quick Complex Ecosystem

November 3, 2021

Curriculum Offline Imitation Learning
Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, Tie-Yan Liu
Reinforcement Learning Offline Reinforcement Learning Behavior Cloning Offline Imitation Offline Imitation Learning

Offline Reinforcement Learning

Papers

MOORe: Model-based Offline-to-Online Reinforcement Learning

An Improved Reinforcement Learning Algorithm for Learning to Branch

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

Offline Reinforcement Learning for Road Traffic Control

Single-Shot Pruning for Offline Reinforcement Learning

Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learning

Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning

Quantile Filtered Imitation Learning

Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions

Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning

The Impact of Data Distribution on Q-learning with Function Approximation

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning

Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

Compressive Features in Offline Reinforcement Learning for Recommender Systems

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning

Dealing with the Unknown: Pessimistic Offline Reinforcement Learning

A Dataset Perspective on Offline Reinforcement Learning

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Curriculum Offline Imitation Learning