Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

June 9, 2022

Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
Deyao Zhu, Li Erran Li, Mohamed Elhoseiny
Reinforcement Learning Offline Reinforcement Learning State Action Space Graph Memory First Order Policy

June 7, 2022

On the Role of Discount Factor in Offline Reinforcement Learning
Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang
Offline Reinforcement Learning Integral Role Sample Efficient Offline Reinforcement Learning Algorithm Offline Adaptation Discount Factor

June 6, 2022

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han
Reinforcement Learning Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Linear MDPs Robust Offline Reinforcement Learning Conservative Value Estimation

June 4, 2022

Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
Xue-Kun Jin, Xu-Hui Liu, Shengyi Jiang, Yang Yu
Offline Reinforcement Learning Policy Evaluation Value Function

June 3, 2022

Offline Reinforcement Learning with Causal Structured World Models
Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang, Yang Yu
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Causal World Model

June 2, 2022

June 1, 2022

May 31, 2022

Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, Tong Zhang
Offline Reinforcement Learning Markov Game Single Agent Linear Function Approximation Offline Multi Agent Reinforcement Learning Tabular Markov Decision Process Minimax Optimal Algorithm

May 27, 2022

Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum
Offline Reinforcement Learning Diverse Ensemble High Uncertainty Value Local Independence Efficient Uncertainty Estimation Ensemble Deep Reinforcement Learning

May 26, 2022

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
Miao Lu, Yifei Min, Zhaoran Wang, Zhuoran Yang
Markov Decision Process Offline Reinforcement Learning Offline Reinforcement Learning Algorithm C POMDPs Text Based Confounders

May 24, 2022

Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
Jinning Li, Chen Tang, Masayoshi Tomizuka, Wei Zhan
Offline Reinforcement Learning Hierarchical Planning Interactive Planning

May 23, 2022

When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, Ya-Qin Zhang
Offline Reinforcement Learning Policy Learning Deep Q Network Geometric Data Deep Function

May 22, 2022

Offline Policy Comparison with Confidence: Benchmarks and Baselines
Anurag Koul, Mariano Phielipp, Alan Fern
New Benchmark Offline Reinforcement Learning Baseline Result High Confidence Comparative Evaluation Baseline Model Offline Policy Evaluation Sequential Decision Making Policy

May 21, 2022

User-Interactive Offline Reinforcement Learning
Phillip Swazinna, Steffen Udluft, Thomas Runkler
Offline Reinforcement Learning New Hyperparameter Offline RL Algorithm Offline Deep Reinforcement Learning Hyperparameter Importance

May 19, 2022

Data Valuation for Offline Reinforcement Learning
Amir Abolfazli, Gregory Palmer, Daniel Kudenko
Offline Reinforcement Learning Data Valuation

Offline Reinforcement Learning

Papers

Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning

On the Role of Discount Factor in Offline Reinforcement Learning

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning

Offline Reinforcement Learning with Causal Structured World Models

Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning

When does return-conditioned supervised learning work for offline reinforcement learning?

Offline Reinforcement Learning with Differential Privacy

Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL

Model Generation with Provable Coverability for Offline Reinforcement Learning

On Gap-dependent Bounds for Offline Reinforcement Learning

Byzantine-Robust Online and Offline Distributed Reinforcement Learning

Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game

Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning

Offline Policy Comparison with Confidence: Benchmarks and Baselines

User-Interactive Offline Reinforcement Learning

Data Valuation for Offline Reinforcement Learning