Robust Offline Reinforcement Learning

Robust offline reinforcement learning (RL) aims to train effective and safe policies using only pre-collected data, avoiding the risks and costs of online interaction. Current research heavily focuses on addressing the inherent uncertainties in offline data, employing techniques like distributional RL, pessimistic value estimation, and robust optimization to mitigate the impact of data limitations and model inaccuracies. This field is crucial for deploying RL in real-world scenarios where online exploration is impractical or dangerous, with applications ranging from robotics to healthcare. The development of provably efficient algorithms and robust model architectures is driving progress towards reliable and safe offline RL deployments.

Papers

November 12, 2024

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
Ruiquan Huang, Yingbin Liang, Jing Yang
Markov Decision Process Low Rank Complex Process Non Markovian Robust Offline Reinforcement Learning Optimal Robust Policy

November 1, 2024

Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
Rui Yang, Jie Wang, Guoping Wu, Bin Li
Native Robustness Bayesian Inference Robust Offline Reinforcement Learning Data Corruption Bayesian Inference Framework Robust Agent

October 11, 2024

Robust Offline Policy Learning with Observational Data from Multiple Sources
Aldo Gael Carranza, Susan Athey
Bandit Feedback Regret Analysis Source Table Observational Data Robust Offline Reinforcement Learning Robust Decision

March 26, 2024

Uncertainty-aware Distributional Offline Reinforcement Learning
Xiaocong Chen, Siyu Wang, Tong Yu, Lina Yao
Offline Reinforcement Learning Epistemic Uncertainty Robust Offline Reinforcement Learning Risk Aware Policy

March 19, 2024

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes
He Wang, Laixi Shi, Yuejie Chi
Native Robustness Offline Reinforcement Learning Sample Complexity Action Space Model Robustness Robust Offline Reinforcement Learning

March 14, 2024

Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
Zhishuai Liu, Pan Xu
Robust Policy Function Approximation Efficient Algorithm Minimax Optimality Robust Offline Reinforcement Learning

October 28, 2023

Robust Offline Reinforcement learning with Heavy-Tailed Rewards
Jin Zhu, Runzhe Wan, Zhengling Qi, Shikai Luo, Chengchun Shi
Offline Reinforcement Learning Policy Evaluation Offline Policy Robust Offline Reinforcement Learning Heavy Tailed Reward

October 19, 2023

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, Tong Zhang
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Robust Offline Reinforcement Learning Data Corruption Implicit Q Learning

September 29, 2023

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
Xiaoyu Wen, Xudong Yu, Rui Yang, Haoyuan Chen, Chenjia Bai, Zhen Wang
Reinforcement Learning High Uncertainty Anticipation Offline Reinforcement Learning Algorithm Near Optimal Policy Policy Improvement Robust Offline Reinforcement Learning Continual Offline Reinforcement Learning

September 28, 2023

Robust Offline Reinforcement Learning -- Certify the Confidence Interval
Jiarui Yao, Simon Shaolei Du
Reinforcement Learning Adversarial Attack Deep Reinforcement Learning Native Robustness Adversarial Training Confidence Interval Robust Offline Reinforcement Learning

June 14, 2023

Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources
Chengshuai Shi, Wei Xiong, Cong Shen, Jing Yang
Offline Reinforcement Learning Lloyd Algorithm Robust Offline Reinforcement Learning Data Perturbation

May 16, 2023

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
Jose Blanchet, Miao Lu, Tong Zhang, Han Zhong
Policy Optimization Robust Offline Reinforcement Learning Optimal Robust Policy Stein Coverage

March 26, 2023

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
Alex Beeson, Giovanni Montana
Offline Reinforcement Learning Optimal Policy Robust Policy Online Fine Tuning Policy Constraint Robust Offline Reinforcement Learning

October 19, 2022

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
Chengqian Gao, Ke Xu, Liu Liu, Deheng Ye, Peilin Zhao, Zhiqiang Xu
Offline Reinforcement Learning Policy Constraint Robust Offline Reinforcement Learning Constraint Relaxation Gradient Penalty

September 14, 2022

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation
Xiaoteng Ma, Zhipeng Liang, Jose Blanchet, Mingwen Liu, Li Xia, Jiheng Zhang, Qianchuan Zhao, Zhengyuan Zhou
Reinforcement Learning Optimal Policy Robust Policy Linear Function Approximation Robust Offline Reinforcement Learning

August 11, 2022

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
Laixi Shi, Yuejie Chi
Offline Reinforcement Learning Robust Policy Value Iteration Distributionally Robust Model Based Offline Reinforcement Learning Tabular Markov Decision Process Robust Offline Reinforcement Learning

June 6, 2022

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han
Reinforcement Learning Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Linear MDPs Robust Offline Reinforcement Learning Conservative Value Estimation

February 23, 2022

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, Zhaoran Wang
Offline Reinforcement Learning Bootstrapping End to End Linear MDPs Policy Algorithm Robust Offline Reinforcement Learning

Robust Offline Reinforcement Learning

Papers

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions

Robust Offline Policy Learning with Observational Data from Multiple Sources

Uncertainty-aware Distributional Offline Reinforcement Learning

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning

Robust Offline Reinforcement learning with Heavy-Tailed Rewards

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness

Robust Offline Reinforcement Learning -- Certify the Confidence Interval

Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning