Offline Policy

Offline policy optimization in reinforcement learning aims to learn optimal policies from pre-collected datasets, eliminating the need for costly online interaction with the environment. Current research focuses on addressing the challenges of distributional shift and limited exploration in offline data, employing techniques like model-based methods (e.g., using world models and optimistic/pessimistic MDPs), actor-critic architectures with various regularization strategies (e.g., variance regularization, behavior proximal methods), and importance weighting. These advancements are significant for deploying reinforcement learning in real-world scenarios where online exploration is impractical or unsafe, particularly in robotics, healthcare, and personalized recommendation systems.

Papers

October 31, 2024

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning
JaeYoon Kim, Junyu Xuan, Christy Liang, Farookh Hussain
LeArning Abstract Offline Policy Exploration Method Offline to Online Reinforcement Learning Dexterous Policy Evolving Policy

October 24, 2024

SAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance
Liyu Zhang, Haochi Wu, Xu Wan, Quan Kong, Ruilong Deng, Mingyang Sun
Reinforcement Learning Reinforcement Learning Algorithm Offline Policy Adversarial Team Game

July 6, 2024

FOSP: Fine-tuning Offline Safe Policy through World Models
Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He, Zichen Yan, Junbo Tan, Xueqian Wang
Model Based Reinforcement Learning World Model Offline Policy Model Based Safe Reinforcement Learning

June 13, 2024

SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets
Shenghua Wan, Ziyuan Chen, Le Gan, Shuai Feng, De-Chuan Zhan
Model Uncertainty Health Policy Recommendation Offline Data Learned Dynamic Offline Policy

May 28, 2024

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, Xianyuan Zhan
Policy Reinforcement Learning Offline Policy Policy Deep Reinforcement Learning Policy Synthesis

May 14, 2024

Understanding the performance gap between online and offline alignment algorithms
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, Will Dabney
Reinforcement Learning AI Alignment Offline Policy Large Language Model Alignment Performance Gap Policy Sampling Online Alignment

January 11, 2024

Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
Yuanzhao Zhai, Yiying Li, Zijian Gao, Xudong Gong, Kele Xu, Dawei Feng, Ding Bo, Huaimin Wang
Offline Reinforcement Learning Policy Optimization Model Based Offline Reinforcement Learning Offline Policy Model Rollouts

December 17, 2023

Probabilistic Offline Policy Ranking with Approximate Bayesian Computation
Longchao Da, Porter Jenkins, Trevor Schwantes, Jeffrey Dotson, Hua Wei
Policy Evaluation Offline Policy Posterior Mode Approximate Bayesian Computation

December 4, 2023

When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
Vincent Liu, Prabhat Nagarajan, Andrew Patterson, Martha White
Reinforcement Learning Policy Evaluation Hyperparameter Tuning Offline Policy Offline Deep Reinforcement Learning

November 6, 2023

Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, Huazhe Xu
Reinforcement Learning Online Reinforcement Learning Online Service Offline Policy Policy Improvement Online Fine Tuning Offline Deep Reinforcement Learning

October 28, 2023

Robust Offline Reinforcement learning with Heavy-Tailed Rewards
Jin Zhu, Runzhe Wan, Zhengling Qi, Shikai Luo, Chengchun Shi
Offline Reinforcement Learning Policy Evaluation Offline Policy Robust Offline Reinforcement Learning Heavy Tailed Reward

October 26, 2023

CROP: Conservative Reward for Model-based Offline Policy Optimization
Hao Li, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Xiao-Yin Liu, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Bo-Xian Yao, Zeng-Guang Hou
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Offline Policy Conservative Reward

October 16, 2023

End-to-end Offline Reinforcement Learning for Glycemia Control
Tristan Beolet, Alice Adenis, Erik Huneker, Maxime Louis
Offline Reinforcement Learning Closed Loop Patient Data Offline Policy Glucose Control

October 10, 2023

Bi-Level Offline Policy Optimization with Limited Exploration
Wenzhuo Zhou
Offline Reinforcement Learning Policy Optimization Regret Guarantee Offline Policy Pure Exploration Bellman Error Adversarial Estimator

October 6, 2023

Self-Confirming Transformer for Locally Consistent Online Adaptation in Multi-Agent Reinforcement Learning
Tao Li, Juan Guevara, Xinghong Xie, Quanyan Zhu
Multi Agent Reinforcement Learning Offline Reinforcement Learning Non Stationary Online Adaptation Offline Policy

September 27, 2023

Importance-Weighted Offline Learning Done Right
Germano Gabbianelli, Gergely Neu, Matteo Papini
Contextual Bandit Near Optimal Policy Optimal Behavior Offline Policy Implicit Exploration

September 12, 2023

ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning
Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Rui Kong, Zongzhang Zhang, Yang Yu
Dynamic Programming Decision Transformer Offline Policy Consistent Conditioning Action Generation Streaming Transformer Conditional Transformer

September 5, 2023

Model-based Offline Policy Optimization with Adversarial Network
Junming Yang, Xingguo Chen, Shengyuan Wang, Bolei Zhang
Offline Reinforcement Learning Adversarial Learning Offline Policy Transition Model

July 21, 2023

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Xiangsen Wang, Haoran Xu, Yinan Zheng, Xianyuan Zhan
Multi Agent Reinforcement Learning Multi Agent Offline Reinforcement Learning Implicit Regularization Action Free Offline Offline Policy Offline Multi Agent Reinforcement Learning Multi Agent MuJoCo

July 4, 2023

MOPO-LSI: A User Guide
Yong Zheng, Kumar Neelotpal Shukla, Jasmine Xu, David, Wang, Michael O'Leary
Hyper Parameter Offline Policy User Manual Constrained Multi Objective Optimization Problem Sustainable Investment