Robust Policy Evaluation

Robust policy evaluation aims to reliably assess the performance of a policy in reinforcement learning and contextual bandits, especially when faced with data discrepancies between training and deployment environments or noisy, heavy-tailed reward distributions. Current research focuses on developing robust estimators and algorithms, including distributionally robust methods, first-order policy optimization techniques, and natural actor-critic approaches, often incorporating function approximation to handle high-dimensional state spaces and addressing issues like unobserved confounders. These advancements enhance the trustworthiness and applicability of reinforcement learning in real-world scenarios where assumptions of perfect data or model accuracy are unrealistic, improving decision-making in domains like healthcare and robotics.

Papers

January 21, 2024

Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Bandits
Yihong Guo, Hao Liu, Yisong Yue, Anqi Liu
Contextual Bandit Covariate Shift Robust Regression Policy Distribution Gaussian Reward Robust Policy Evaluation

October 4, 2023

Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learning
Weidong Liu, Jiyuan Tu, Yichen Zhang, Xi Chen
Reinforcement Learning Scientific Inference Online Estimation Robust Statistic Heavy Tailed Reward Robust Policy Evaluation

July 29, 2023

First-order Policy Optimization for Robust Policy Evaluation
Yan Li, Guanghui Lan
State Action Value First Order Policy Robust Policy Evaluation Frank Wolfe Policy Optimization

July 17, 2023

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, P. R. Kumar, Chao Tian
Robust Reinforcement Learning Function Approximation Optimal Robust Policy Robust Policy Evaluation

May 30, 2023

Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets
Mengmeng Li, Daniel Kuhn, Tobias Sutter
Policy Gradient Robust Policy Uncertainty Set Robust Markov Decision Process Robust Policy Evaluation

February 1, 2023

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders
David Bruns-Smith, Angela Zhou
Offline Reinforcement Learning Unobserved Confounders Q Function Fitted Q Iteration Q Evaluation Robust Policy Evaluation

Robust Policy Evaluation

Papers

Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Bandits

Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learning

First-order Policy Optimization for Robust Policy Evaluation

Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders