Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

October 11, 2023

October 10, 2023

October 9, 2023

October 7, 2023

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration
Ziqi Zhang, Xiao Xiong, Zifeng Zhuang, Jinxin Liu, Donglin Wang
Reinforcement Learning Offline Reinforcement Learning Reinforcement Learning Algorithm Online Fine Tuning Entropy Exploration

October 6, 2023

October 4, 2023

Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
Jeonghye Kim, Suyoung Lee, Woojun Kim, Youngchul Sung
Transformer Based Offline Reinforcement Learning Decision Making Decision Transformer Action Prediction Local Filter

October 3, 2023

PCGPT: Procedural Content Generation via Transformers
Sajad Mohaghegh, Mohammad Amin Ramezan Dehnavi, Golnoosh Abdollahinejad, Matin Hashemi
Transformer Megatron Decepticons Offline Reinforcement Learning Procedural Content Generation Content Creator Incentive

October 2, 2023

October 1, 2023

September 30, 2023

Efficient Planning with Latent Diffusion
Wenhao Li
Offline Reinforcement Learning Latent Diffusion Efficient Planning Latent Action Latent Action Space

September 28, 2023

Uncertainty-Aware Decision Transformer for Stochastic Driving Environments
Zenan Li, Fan Nie, Qiao Sun, Fang Da, Hang Zhao
Offline Reinforcement Learning Decision Transformer Stochastic Environment

September 24, 2023

Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills
Zenan Li, Fan Nie, Qiao Sun, Fang Da, Hang Zhao
Autonomous Driving Offline Reinforcement Learning Variational Autoencoder Long Horizon Latent Skill

September 22, 2023

H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang, Guyue Zhou, Jianming Hu, Xianyuan Zhan
Reinforcement Learning Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Efficient Framework Transfer Reinforcement Learning Real Robot Experiment Offline to Online Reinforcement Learning Dynamic Mismatch

September 18, 2023

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
Reinforcement Learning Offline Reinforcement Learning Q Learning Q Function Task Specific Policy Q Transformer

Offline Reinforcement Learning

Papers

Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples

Score Regularized Policy Optimization through Diffusion Behavior

Boosting Continuous Control with Consistency Policy

Bi-Level Offline Policy Optimization with Limited Exploration

Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Self-Confirming Transformer for Locally Consistent Online Adaptation in Multi-Agent Reinforcement Learning

Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets

Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making

PCGPT: Procedural Content Generation via Transformers

Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

Pre-training with Synthetic Data Helps Offline Reinforcement Learning

A General Offline Reinforcement Learning Framework for Interactive Recommendation

Efficient Planning with Latent Diffusion

Uncertainty-Aware Decision Transformer for Stochastic Driving Environments

Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills

H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions