Offline Reinforcement Learning Algorithm

Offline reinforcement learning (RL) aims to train agents using pre-collected datasets, eliminating the need for costly online interaction with the environment. Current research focuses on addressing challenges like limited data, distribution shifts between training and deployment, and the impact of data quality on performance, employing techniques such as conservative Q-learning, diffusion models, and data augmentation methods to improve policy learning and generalization. These advancements are significant for real-world applications where online learning is impractical or unsafe, particularly in robotics, autonomous driving, and other domains with high-stakes decision-making.

Papers

September 22, 2023

H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang, Guyue Zhou, Jianming Hu, Xianyuan Zhan
Reinforcement Learning Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Efficient Framework Transfer Reinforcement Learning Real Robot Experiment Offline to Online Reinforcement Learning Dynamic Mismatch

September 4, 2023

Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance
Qisen Yang, Shenzhi Wang, Qihang Zhang, Gao Huang, Shiji Song
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Policy Constraint

July 28, 2023

Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Bernhard Schölkopf, Georg Martius
Offline Reinforcement Learning Real World Dexterous Manipulation Offline Reinforcement Learning Algorithm

July 10, 2023

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
Suzan Ece Ada, Erhan Oztop, Emre Ugur
Offline Reinforcement Learning Distribution Generalization Behavior Cloning Offline Reinforcement Learning Algorithm Diffusion Policy

June 27, 2023

Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
Jinyi Liu, Yi Ma, Jianye Hao, Yujing Hu, Yan Zheng, Tangjie Lv, Changjie Fan
Data Driven Offline Reinforcement Learning Algorithm Memory Replay Trajectory Sampling Trajectory Replay

June 22, 2023

Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
Zhang-Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, Romain Laroche
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Expert Trajectory Trajectory Distribution

June 13, 2023

Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective
Zeyu Zhang, Yi Su, Hui Yuan, Yiran Wu, Rishab Balasubramanian, Qingyun Wu, Huazheng Wang, Mengdi Wang
Offline Reinforcement Learning Policy Learning Stable Rank Reinforcement Learning Approach Offline Reinforcement Learning Algorithm Click Model

June 12, 2023

Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
Kai Zhao, Yi Ma, Jianye Hao, Jinyi Liu, Yan Zheng, Zhaopeng Meng
Reinforcement Learning Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Ensemble Based Q Ensemble

June 8, 2023

Decoupled Prioritized Resampling for Offline RL
Yang Yue, Bingyi Kang, Xiao Ma, Qisen Yang, Gao Huang, Shiji Song, Shuicheng Yan
Offline Reinforcement Learning Experience Replay Offline Reinforcement Learning Algorithm Policy Constraint

June 7, 2023

Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang, Shoucheng Song, Han Wang, Youfang Lin, Li Jiang
Offline Reinforcement Learning Object Surface Offline Reinforcement Learning Algorithm State Action Space Passive Symmetry Sample Efficient Offline Reinforcement Learning

June 5, 2023

Survival Instinct in Offline Reinforcement Learning
Anqi Li, Dipendra Misra, Andrey Kolobov, Ching-An Cheng
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm State of the Art Robustness

June 1, 2023

Improving and Benchmarking Offline Reinforcement Learning Algorithms
Bingyi Kang, Xiao Ma, Yirui Wang, Yang Yue, Shuicheng Yan
Practical Algorithm Offline Reinforcement Learning Large Relevance Improvement Offline Reinforcement Learning Algorithm

March 30, 2023

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
Yicheng Luo, Jackie Kay, Edward Grefenstette, Marc Peter Deisenroth
Technical Challenge Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Policy Algorithm Online Fine Tuning Simple Solution

March 13, 2023

Deploying Offline Reinforcement Learning with Human Feedback
Ziniu Li, Ke Xu, Liu Liu, Lanqing Li, Deheng Ye, Peilin Zhao
Reinforcement Learning Offline Reinforcement Learning Human Feedback Offline Reinforcement Learning Algorithm Policy Parameterization Robot Locomotion

March 9, 2023

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine
Offline Reinforcement Learning Q Learning Offline Reinforcement Learning Algorithm Conservative Q Learning Online Fine Tuning

February 27, 2023

The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning
Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Provable Benefit Reward Free Self Supervised Reinforcement Learning

February 24, 2023

Neural Laplace Control for Continuous-time Delayed Systems
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, Mihaela van der Schaar
Offline Reinforcement Learning Continuous Time Offline Reinforcement Learning Algorithm Model Based Offline Reinforcement Learning Continuous Environment Neural Laplace

February 6, 2023

A Strong Baseline for Batch Imitation Learning
Matthew Smith, Lucas Maystre, Zhenwen Dai, Kamil Ciosek
Markov Decision Process Feature Imitation Strong Baseline Offline Reinforcement Learning Algorithm Sequential Decision Making Batch Reinforcement Learning

November 21, 2022

A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning
Lang Qin, Rui Yan, Huajin Tang
Reinforcement Learning Deep Reinforcement Learning Spike Train Offline Reinforcement Learning Algorithm Matrix Multiplication

November 2, 2022

Behavior Prior Representation learning for Offline Reinforcement Learning
Hongyu Zang, Xin Li, Jie Yu, Chen Liu, Riashat Islam, Remi Tachet Des Combes, Romain Laroche
Offline Reinforcement Learning Behavior Cloning Offline Reinforcement Learning Algorithm