Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

February 3, 2023

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, Ping Luo
Diffusion Model Offline Reinforcement Learning

February 1, 2023

January 31, 2023

January 30, 2023

January 28, 2023

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning
Jing Zhang, Chi Zhang, Wenjia Wang, Bing-Yi Jing
Offline Reinforcement Learning Constrained Policy Optimization Advantage Learning

January 12, 2023

Safe Policy Improvement for POMDPs via Finite-State Controllers
Thiago D. Simão, Marnix Suilen, Nils Jansen
Markov Decision Process Offline Reinforcement Learning Optimal Policy Policy Improvement POMDP Policy Finite State Controller

January 5, 2023

Extreme Q-Learning: MaxEnt RL without Entropy
Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon
Offline Reinforcement Learning Q Learning Q Function Stable Entropy Maximum Entropy Reinforcement Learning MaxEnt RL Modern Deep Reinforcement Learning

January 3, 2023

Contextual Conservative Q-Learning for Offline Reinforcement Learning
Ke Jiang, Jiayu Yao, Xiaoyang Tan
Offline Reinforcement Learning Conservative Q Learning Inverse Dynamic

December 28, 2022

Revisiting the Linear-Programming Framework for Offline RL with General Function Approximation
Asuman Ozdaglar, Sarath Pattathil, Jiawei Zhang, Kaiqing Zhang
New Framework Markov Decision Process Offline Reinforcement Learning Primal Dual Sample Efficient Offline Reinforcement Learning

December 23, 2022

Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information
Zuyue Fu, Zhengling Qi, Zhuoran Yang, Zhaoran Wang, Lan Wang
Offline Reinforcement Learning Policy Evaluation Private Data Human Machine Interaction Policy Algorithm

December 16, 2022

December 15, 2022

Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
Shivakanth Sujit, Pedro H. M. Braga, Jorg Bornschein, Samira Ebrahimi Kahou
Reinforcement Learning Offline Reinforcement Learning Deep RL Action Free Offline

December 8, 2022

December 5, 2022

TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets
Yuanying Cai, Chuheng Zhang, Li Zhao, Wei Shen, Xuyun Zhang, Lei Song, Jiang Bian, Tao Qin, Tieyan Liu
Offline Reinforcement Learning Behavior Cloning Offline RL Algorithm Behavior Policy

Offline Reinforcement Learning

Papers

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders

Selective Uncertainty Propagation in Offline RL

Anti-Exploration by Random Network Distillation

Skill Decision Transformer

Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings

Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning

Safe Policy Improvement for POMDPs via Finite-State Controllers

Extreme Q-Learning: MaxEnt RL without Entropy

Contextual Conservative Q-Learning for Offline Reinforcement Learning

Revisiting the Linear-Programming Framework for Offline RL with General Function Approximation

Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information

Offline Reinforcement Learning for Visual Navigation

Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling

Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies

Confidence-Conditioned Value Functions for Offline Reinforcement Learning

Model-based trajectory stitching for improved behavioural cloning and its applications

TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets