Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to train agents using pre-collected data, eliminating the need for costly and potentially risky online interactions with the environment. Current research focuses on addressing challenges like distributional shift (mismatch between training and target data) and improving generalization across diverse tasks, employing model architectures such as transformers, convolutional networks, and diffusion models, along with algorithms like conservative Q-learning and decision transformers. These advancements are significant for deploying RL in real-world applications where online learning is impractical or unsafe, impacting fields ranging from robotics and healthcare to personalized recommendations and autonomous systems.

Papers

November 3, 2022

November 2, 2022

November 1, 2022

Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, Jiantao Jiao
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Conservative Policy Single Policy Concentrability Augmented Lagrangian

October 27, 2022

Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision
Ashvin Nair, Brian Zhu, Gokul Narayanan, Eugen Solowjow, Sergey Levine
LeArning Abstract Strong Generalization Offline Reinforcement Learning Vision Paper Reward Function Reward Learning Insertion Task Connector Insertion

October 25, 2022

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, Joni Pajarinen
Offline Reinforcement Learning Behavior Cloning Online Fine Tuning Behavior Regularization

October 24, 2022

Dichotomy of Control: Separating What You Can Control from What You Cannot
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, Ofir Nachum
Offline Reinforcement Learning External Control Multi Scenario Stochastic Reward Return Conditioned Supervised Learning Market Return

October 21, 2022

October 20, 2022

Learning and Retrieval from Prior Data for Skill-based Imitation Learning
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, Yuke Zhu
LeArning Abstract Imitation Learning Offline Reinforcement Learning App to App Retrieval

October 19, 2022

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
Chengqian Gao, Ke Xu, Liu Liu, Deheng Ye, Peilin Zhao, Zhiqiang Xu
Offline Reinforcement Learning Policy Constraint Robust Offline Reinforcement Learning Constraint Relaxation Gradient Penalty

October 17, 2022

Boosting Offline Reinforcement Learning via Data Rebalancing
Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, Shuicheng Yan
Offline Reinforcement Learning Distribution Data Offline Reinforcement Learning Algorithm

October 16, 2022

Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
Allen Nie, Yannis Flet-Berliac, Deon R. Jordan, William Steenbergen, Emma Brunskill
Offline Reinforcement Learning Hyperparameter Optimization Limited Data Efficient Pipeline Hyperparameter Selection

October 15, 2022

A Policy-Guided Imitation Approach for Offline Reinforcement Learning
Haoran Xu, Li Jiang, Jianxiong Li, Xianyuan Zhan
Offline Reinforcement Learning Based Imitation State of the Art Imitation Imitation Learned Policy Reward Maximizing Policy

October 14, 2022

Mutual Information Regularized Offline Reinforcement Learning
Xiao Ma, Bingyi Kang, Zhongwen Xu, Min Lin, Shuicheng Yan
Offline Reinforcement Learning Mutual Information Conservative Q Learning Mutual Information Regularization Behavior Regularization

October 13, 2022

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
Kaiyang Guo, Yunfeng Shao, Yanhui Geng
Offline Reinforcement Learning Policy Learning Policy Optimization Model Based Offline Reinforcement Learning

October 12, 2022