Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

August 12, 2022

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
Zhendong Wang, Jonathan J Hunt, Mingyuan Zhou
Offline Reinforcement Learning Conditional Diffusion Model Bandit Algorithm Diffusion Policy Expressive Policy Visuomotor Latent Diffusion Policy

August 11, 2022

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
Laixi Shi, Yuejie Chi
Offline Reinforcement Learning Robust Policy Value Iteration Distributionally Robust Model Based Offline Reinforcement Learning Tabular Markov Decision Process Robust Offline Reinforcement Learning

August 1, 2022

Safe Policy Improvement Approaches and their Limitations
Philipp Scholl, Felix Dietrich, Clemens Otte, Steffen Udluft
Offline Reinforcement Learning Fundamental Limitation State Action Pair Policy Improvement Safe Algorithm

July 26, 2022

July 21, 2022

Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning
Adam Villaflor, Zhe Huang, Swapnil Pande, John Dolan, Jeff Schneider
Reinforcement Learning Autonomous Driving Offline Reinforcement Learning Sequence Modeling Stochastic Environment Autonomous Driving Task Optimism Bias

July 20, 2022

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
Haoran Xu, Xianyuan Zhan, Honglei Yin, Huiling Qin
Offline Reinforcement Learning Reward Learning Suboptimal Demonstration Offline Imitation Learning Expert Policy

July 16, 2022

BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion
Fanglin Chen, Xiao Liu, Bo Tang, Feiyu Xiong, Serim Hwang, Guomian Zhuang
Offline Reinforcement Learning Constrained Reinforcement Learning Budget Constraint

July 12, 2022

July 7, 2022

Multi-objective Optimization of Notifications Using Offline Reinforcement Learning
Prakruthi Prabhakar, Yiping Yuan, Guangyu Yang, Wensheng Sun, Ajith Muralidharan
Offline Reinforcement Learning Multi Objective Optimization Conservative Q Learning Large Scale Decision Notification System Mobile Notification

July 5, 2022

June 22, 2022

Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
Zeyu Wang, Huiying Zhao, Peng Ren, Yuxi Zhou, Ming Sheng
Deep Reinforcement Learning Offline Reinforcement Learning Real Time Continuous Space Dynamic Treatment Regime Standard Sepsis Optimal Treatment Medical Decision

June 17, 2022

Bootstrapped Transformer for Offline Reinforcement Learning
Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, Dongsheng Li
Offline Reinforcement Learning Sequence Model Sequence Level Training

June 14, 2022

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
Shentao Yang, Yihao Feng, Shujian Zhang, Mingyuan Zhou
Offline Reinforcement Learning Policy Optimization State Action Distribution Stationary Policy

June 13, 2022

Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward
Tengyu Xu, Yue Wang, Shaofeng Zou, Yingbin Liang
Offline Reinforcement Learning Proxy Reward Trajectory Wise Reward rEward Decomposition Reward Redistribution

June 11, 2022

Federated Offline Reinforcement Learning
Doudou Zhou, Yufeng Zhang, Aaron Sonabend-W, Zhaoran Wang, Junwei Lu, Tianxi Cai
Markov Decision Process Offline Reinforcement Learning Policy Optimization Offline Reinforcement Learning Algorithm Dynamic Treatment Regime Federated Offline Reinforcement Learning

June 9, 2022

Offline Reinforcement Learning

Papers

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity

Safe Policy Improvement Approaches and their Limitations

Offline Reinforcement Learning at Multiple Frequencies

Branch Ranking for Efficient Mixed-Integer Programming via Offline Ranking-based Policy Learning

Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations

BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion

Learning Bellman Complete Representations for Offline Policy Evaluation

Offline Equilibrium Finding

Multi-objective Optimization of Notifications Using Offline Reinforcement Learning

Offline RL Policies Should be Trained to be Adaptive

An Empirical Study of Implicit Regularization in Deep Offline RL

Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space

Bootstrapped Transformer for Offline Reinforcement Learning

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward

Federated Offline Reinforcement Learning

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

Mildly Conservative Q-Learning for Offline Reinforcement Learning