Based Offline RL

Based offline reinforcement learning (RL) aims to train RL agents using only pre-collected data, avoiding the cost and risk of online interaction. Current research focuses on improving the robustness and efficiency of offline RL algorithms, exploring model-based approaches that leverage learned environment dynamics, and employing techniques like diffusion models and return-conditioned supervised learning to overcome limitations of traditional methods such as Bellman completeness issues. These advancements are significant because they enable the application of RL in scenarios where online data collection is impractical or unsafe, with potential impact across robotics, healthcare, and other fields.

Papers

July 29, 2024

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, Amy Zhang
Offline Reinforcement Learning DIstribution Estimation Diffusion Guidance Diffusion Inference Policy Distribution Based Offline RL

January 6, 2024

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
Rafael Rafailov, Kyle Hatch, Victor Kolev, John D. Martin, Mariano Phielipp, Chelsea Finn
Reinforcement Learning Action Free Offline Online Fine Tuning Based Offline RL

October 30, 2023

Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning
Zhaoyi Zhou, Chuning Zhu, Runlong Zhou, Qiwen Cui, Abhishek Gupta, Simon Shaolei Du
Supervised Learning Policy Learning Policy Reinforcement Learning Model Based Dynamic Programming Q$ Learning Based Offline RL Bellman Completeness Trajectory Stitching

October 9, 2023

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning
Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, Xueqian Wang
Diffusion Model Offline Reinforcement Learning Based Offline RL

June 1, 2023

IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control
Rohan Chitnis, Yingchen Xu, Bobak Hashemi, Lucas Lehnert, Urun Dogan, Zheqing Zhu, Olivier Delalleau
Model Based Reinforcement Learning Implicit Language Td MPC Implicit Q Learning Based Offline RL

March 7, 2023

On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
Mustafa O. Karabag, Ufuk Topcu
Reinforcement Learning Offline Reinforcement Learning Sample Complexity Policy Evaluation Policy Optimization Model Based Offline Reinforcement Learning Based Offline RL

November 27, 2022

Domain Generalization for Robust Model-Based Offline Reinforcement Learning
Alan Clark, Shoaib Ahmed Siddiqui, Robert Kirk, Usman Anwar, Stephen Chung, David Krueger
Offline Reinforcement Learning Domain Generalization Robust Version Model Based Offline Reinforcement Learning Offline RL Based Offline RL

November 8, 2022

ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data
Tengyang Xie, Mohak Bhardwaj, Nan Jiang, Ching-An Cheng
Offline Reinforcement Learning Offline Data Model Driven Adversarial Model Based Offline RL Baseline Policy

September 30, 2022

S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning
Daesol Cho, Dongseok Shim, H. Jin Kim
Data Augmentation Offline Reinforcement Learning Image Synthesis Based Offline RL

Based Offline RL

Papers

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning

Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning

IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control

On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples

Domain Generalization for Robust Model-Based Offline Reinforcement Learning

ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data

S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning