Model Based Offline Reinforcement Learning

Model-based offline reinforcement learning (RL) aims to train effective policies using only pre-collected data, avoiding the need for costly and potentially risky online interaction with the environment. Current research focuses on addressing the challenges of distribution shift and model uncertainty, employing techniques like conservative reward shaping, uncertainty-aware model architectures (e.g., ensembles, autoregressive models), and pessimism-based policy optimization to improve robustness and generalization. This field is significant because it enables RL applications in domains where online learning is impractical or unsafe, such as healthcare and autonomous driving, and is driving advancements in both theoretical understanding and practical algorithm design.

Papers

May 17, 2023

Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning
Gen Li, Wenhao Zhan, Jason D. Lee, Yuejie Chi, Yuxin Chen
Reinforcement Learning Model Based Offline Reinforcement Learning Provable Benefit Hybrid Reinforcement Learning Reward Fine Tuning Agnostic Exploration Policy Fine Tuning

May 5, 2023

A Survey on Offline Model-Based Reinforcement Learning
Haoyang He
Timely Survey Offline Reinforcement Learning Model Based Reinforcement Learning Model Based Action Free Offline Model Based Offline Reinforcement Learning

April 10, 2023

Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning
Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, Xiu Li
Data Augmentation Offline Reinforcement Learning Model Based Offline Reinforcement Learning Synthetic Trajectory Trajectory Truncation

March 7, 2023

February 24, 2023

Neural Laplace Control for Continuous-time Delayed Systems
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, Mihaela van der Schaar
Offline Reinforcement Learning Continuous Time Offline Reinforcement Learning Algorithm Model Based Offline Reinforcement Learning Continuous Environment Neural Laplace

February 21, 2023

Adversarial Model for Offline Reinforcement Learning
Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang, Ching-An Cheng
Adversarial Training Offline Reinforcement Learning Model Based Offline Reinforcement Learning

January 26, 2023

Model-based Offline Reinforcement Learning with Local Misspecification
Kefan Dong, Yannis Flet-Berliac, Allen Nie, Emma Brunskill
State Action Pair Model Based Offline Reinforcement Learning Model Misspecification Offline Policy Policy Selection

November 30, 2022

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes
Offline Reinforcement Learning Epistemic Uncertainty Risk Sensitive High Quality Risk Description Model Based Offline Reinforcement Learning Risk Sensitive Reinforcement Learning High Epistemic Uncertainty

November 27, 2022

Domain Generalization for Robust Model-Based Offline Reinforcement Learning
Alan Clark, Shoaib Ahmed Siddiqui, Robert Kirk, Usman Anwar, Stephen Chung, David Krueger
Offline Reinforcement Learning Domain Generalization Robust Version Model Based Offline Reinforcement Learning Offline RL Based Offline RL

October 13, 2022

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
Kaiyang Guo, Yunfeng Shao, Yanhui Geng
Offline Reinforcement Learning Policy Learning Policy Optimization Model Based Offline Reinforcement Learning

October 12, 2022

A Unified Framework for Alternating Offline Model Training and Policy Learning
Shentao Yang, Shujian Zhang, Yihao Feng, Mingyuan Zhou
Reinforcement Learning Offline Reinforcement Learning Unified Framework Policy Learning Model Based Offline Reinforcement Learning

September 7, 2022

Concept-modulated model-based offline reinforcement learning for rapid generalization
Nicholas A. Ketz, Praveen K. Pilly
Zero Shot Model Based Reinforcement Learning Model Based Offline Reinforcement Learning Provable Generalization

August 11, 2022

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
Laixi Shi, Yuejie Chi
Offline Reinforcement Learning Robust Policy Value Iteration Distributionally Robust Model Based Offline Reinforcement Learning Tabular Markov Decision Process Robust Offline Reinforcement Learning

June 3, 2022

Offline Reinforcement Learning with Causal Structured World Models
Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang, Yang Yu
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Causal World Model

April 26, 2022

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes
Reinforcement Learning Offline Reinforcement Learning Robust Version Model Based Offline Reinforcement Learning Adversarial Environment Conservative Policy Robust Adversarial

April 11, 2022

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, Yuting Wei
Offline Reinforcement Learning Sample Complexity Finite Horizon Model Based Offline Reinforcement Learning Infinite Horizon

January 25, 2022

MOORe: Model-based Offline-to-Online Reinforcement Learning
Yihuan Mao, Chao Wang, Bin Wang, Chongjie Zhang
Offline Reinforcement Learning Performance Improvement Online Adaptation Model Based Offline Reinforcement Learning Causal Thread Sample Efficient Adaptation

January 7, 2022

Offline Reinforcement Learning for Road Traffic Control
Mayuresh Kunjir, Sanjay Chawla
Reinforcement Learning Offline Reinforcement Learning Traffic Signal Control Traffic Management Model Based Offline Reinforcement Learning

November 22, 2021

UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
Christopher Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, Torsten Bertram
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Simulated Driving Stochastic Action