Offline RL Algorithm

Offline reinforcement learning (RL) aims to train agents using pre-collected datasets, avoiding the need for costly or risky real-world exploration. Current research focuses on improving the robustness and performance of offline RL algorithms, addressing challenges like distribution shift between the training data and optimal policies, inaccurate Q-value estimation, and vulnerability to reward poisoning attacks. Prominent approaches include model-based methods incorporating conservative Q-learning and model-free algorithms like actor-critic architectures with various regularization techniques. These advancements are crucial for enabling safe and efficient RL deployment in real-world applications, particularly in domains like robotics and healthcare where extensive online interaction is impractical or dangerous.

Papers

August 15, 2024

D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
Rafael Rafailov, Kyle Hatch, Anikait Singh, Laura Smith, Aviral Kumar, Ilya Kostrikov, Philippe Hansen-Estruch, Victor Kolev, Philip Ball, Jiajun Wu, Chelsea Finn, Sergey Levine
Real World Data Driven Diverse Datasets Offline Deep Reinforcement Learning Offline RL Algorithm

February 15, 2024

Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning
Yinglun Xu, Rohan Gumaste, Gagandeep Singh
Contrastive Learning Offline Reinforcement Learning Offline RL Algorithm Reward Poisoning Attack

January 16, 2024

Solving Continual Offline Reinforcement Learning with Decision Transformer
Kaixin Huang, Li Shen, Chen Zhao, Chun Yuan, Dacheng Tao
Offline Reinforcement Learning Decision Transformer Offline RL Algorithm Offline Learning Continual Offline Reinforcement Learning

December 12, 2023

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, Wanli Ouyang
Reinforcement Learning Visual Perspective Q Value Offline RL Algorithm Offline to Online Reinforcement Learning

October 9, 2023

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
Trevor McInroe, Adam Jelley, Stefano V. Albrecht, Amos Storkey
Task Planning Distribution Data Intrinsic Reward Offline RL Algorithm Offline to Online Reinforcement Learning RL Method Data Rich Region

October 6, 2023

Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL
Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, Gao Huang
Human Prediction Human Understanding Good Better Q Value Offline RL Algorithm Function Estimation

August 7, 2023

AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
Michaël Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Richard Powell, Konrad Żołna, Julian Schrittwieser, David Choi, Petko Georgiev, Daniel Toyama, Aja Huang, Roman Ring, Igor Babuschkin, Timo Ewalds, Mahyar Bordbar, Sarah Henderson, Sergio Gómez Colmenarejo, Aäron van den Oord, Wojciech Marian Czarnecki, Nando de Freitas, Oriol Vinyals
Offline Reinforcement Learning Reinforcement Learning Environment StarCraft II Offline RL Algorithm Cloning Agent

July 21, 2023

Model-based Offline Reinforcement Learning with Count-based Conservatism
Byeongchan Kim, Min-hwan Oh
Model Based Offline Reinforcement Learning Offline RL Algorithm Conservative Objective Model

June 16, 2023

Automatic Trade-off Adaptation in Offline RL
Phillip Swazinna, Steffen Udluft, Thomas Runkler
Behavior Cloning Performance Optimization Offline RL Algorithm Adaptive Autopilot

June 1, 2023

Improving Offline RL by Blending Heuristics
Sinong Geng, Aldo Pacchiano, Andrey Kolobov, Ching-An Cheng
Offline RL Algorithm Offline RL Dual Source Blending Attack

May 4, 2023

Masked Trajectory Models for Prediction, Representation, and Control
Philipp Wu, Arjun Majumdar, Kevin Stone, Yixin Lin, Igor Mordatch, Pieter Abbeel, Aravind Rajeswaran
Human Prediction External Control Individual Representation State Representation State Action Pair Offline RL Algorithm

December 5, 2022

TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets
Yuanying Cai, Chuheng Zhang, Li Zhao, Wei Shen, Xuyun Zhang, Lei Song, Jiang Bian, Tao Qin, Tieyan Liu
Offline Reinforcement Learning Behavior Cloning Offline RL Algorithm Behavior Policy

May 21, 2022

User-Interactive Offline Reinforcement Learning
Phillip Swazinna, Steffen Udluft, Thomas Runkler
Offline Reinforcement Learning New Hyperparameter Offline Deep Reinforcement Learning Offline RL Algorithm Hyperparameter Importance

May 5, 2022

Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning
Boxiang Lyu, Zhaoran Wang, Mladen Kolar, Zhuoran Yang
Markov Decision Process Offline Reinforcement Learning Mechanism Design Optimal Mechanism Offline RL Algorithm

March 25, 2022

A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies
Pramod Kaushik, Sneha Kummetha, Perusha Moodley, Raju S. Bapi
Distribution Shift Q Learning Optimal Policy Conservative Q Learning Offline RL Algorithm Sepsis Treatment

February 5, 2022

Adversarially Trained Actor Critic for Offline Reinforcement Learning
Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal
Offline Reinforcement Learning Deep RL Stackelberg Game Public Policy Offline RL Algorithm