Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

June 13, 2024

June 12, 2024

June 11, 2024

June 10, 2024

June 8, 2024

Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, Liqiang Nie
Offline Reinforcement Learning Mamba in Mamba Value Based Reinforcement Learning Evolutionary Policy Search Evolving Policy

June 6, 2024

Strategically Conservative Q-Learning
Yutaka Shimizu, Joey Hong, Sergey Levine, Masayoshi Tomizuka
Offline Reinforcement Learning Distribution Data Conservative Q Learning Conservative Value Estimation

June 5, 2024

UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
Yu Zhang, Rui Yu, Zhipeng Yao, Wenyuan Zhang, Jun Wang, Liming Zhang
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Mean Squared Error Optimal Value Function Operator Prediction

June 4, 2024

Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning
Jiahang Cao, Qiang Zhang, Ziqing Wang, Jingkai Sun, Jiaxu Wang, Hao Cheng, Yecheng Shao, Wen Zhao, Gang Han, Yijie Guo, Renjing Xu
Reinforcement Learning Offline Reinforcement Learning Multi Scale Mamba in Mamba Decision Transformer Action Prediction Sequential Model Decision Maker

June 3, 2024

A Theory of Learnability for Offline Decision Making
Chenjie Mao, Qiaosheng Zhang
Markov Decision Process Offline Reinforcement Learning Theoretical Understanding

May 31, 2024

May 30, 2024

Offline Reinforcement Learning

Papers

Is Value Learning Really the Main Bottleneck in Offline RL?

DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learning

Orthogonalized Estimation of Difference of $Q$-functions

Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

Augmenting Offline RL with Unlabeled Data

Integrating Domain Knowledge for handling Limited Data in Offline RL

Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?

Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning

Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL

Strategically Conservative Q-Learning

UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning

Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning

A Theory of Learnability for Offline Decision Making

Bayesian Design Principles for Offline-to-Online Reinforcement Learning

In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought

Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models

Diffusion Policies creating a Trust Region for Offline Reinforcement Learning