Optimal Robust Policy

Optimal robust policy research aims to design control strategies that perform well despite uncertainties in the environment or model parameters, a crucial challenge in reinforcement learning. Current efforts focus on developing algorithms, such as robust fitted Q-iteration and natural actor-critic methods, that leverage both offline and online data, often incorporating various divergence measures (e.g., Wasserstein, total variation, KL-divergence) to define uncertainty sets. These advancements address limitations of traditional approaches, particularly in high-dimensional state spaces, by improving sample efficiency and providing theoretical guarantees on the performance of learned robust policies. This work has significant implications for deploying reinforcement learning agents in real-world settings where model inaccuracies and unexpected disturbances are inevitable.

Papers

November 12, 2024

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
Ruiquan Huang, Yingbin Liang, Jing Yang
Markov Decision Process Low Rank Complex Process Non Markovian Robust Offline Reinforcement Learning Optimal Robust Policy

July 23, 2024

Data-driven Multistage Distributionally Robust Linear Optimization with Nested Distance
Rui Gao, Rohit Arora, Yizhe Huang
Convex Optimization Robust Optimization Dynamic Programming Distributionally Robust Optimization Robust Evaluation Optimal Robust Policy

May 8, 2024

Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data
Kishan Panaganti, Adam Wierman, Eric Mazumdar
Reinforcement Learning Markov Decision Process Action Free Offline Divergence Regularization Optimal Robust Policy

February 3, 2024

Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, Shichen Liao
Deep Reinforcement Learning Q Learning Robust Policy Bellman Error Optimal Robust Policy State Adversarial

September 5, 2023

Distributionally Robust Model-based Reinforcement Learning with Large State Spaces
Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, Ilija Bogunovic
Reinforcement Learning State Space Robust Reinforcement Learning Distributionally Robust Optimal Robust Policy

July 17, 2023

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, P. R. Kumar, Chao Tian
Robust Reinforcement Learning Function Approximation Optimal Robust Policy Robust Policy Evaluation

July 15, 2023

Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation
Wenhao Ding, Laixi Shi, Yuejie Chi, Ding Zhao
Reinforcement Learning Native Robustness Markov Decision Process Robust Reinforcement Learning Plain Sight Optimal Robust Policy

May 16, 2023

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
Jose Blanchet, Miao Lu, Tong Zhang, Han Zhong
Policy Optimization Robust Offline Reinforcement Learning Optimal Robust Policy Stein Coverage

January 2, 2023

Robust Average-Reward Markov Decision Processes
Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, Shaofeng Zou
Average Reward Markov Decision Process Robust Reward Optimal Robust Policy

August 10, 2022

Robust Reinforcement Learning using Offline Data
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh
Robust Reinforcement Learning Offline Data Optimal Robust Policy

March 13, 2022

Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model
Jialian Li, Tongzheng Ren, Dong Yan, Hang Su, Jun Zhu
Policy Learning Robust Policy Partially Observable Markov Decision Process Optimal Robust Policy

December 2, 2021

Sample Complexity of Robust Reinforcement Learning with a Generative Model
Kishan Panaganti, Dileep Kalathil
Generative Modeling Sample Complexity Robust Policy Robust Reinforcement Learning Optimal Robust Policy Robust Markov Decision

Optimal Robust Policy

Papers

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Data-driven Multistage Distributionally Robust Linear Optimization with Nested Distance

Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data

Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error

Distributionally Robust Model-based Reinforcement Learning with Large State Spaces

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

Robust Average-Reward Markov Decision Processes

Robust Reinforcement Learning using Offline Data

Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model

Sample Complexity of Robust Reinforcement Learning with a Generative Model