Robust Markov Decision Process

Robust Markov Decision Processes (RMDPs) extend standard Markov Decision Processes by incorporating uncertainty in transition probabilities, aiming to find policies optimal under various possible system dynamics. Current research focuses on developing efficient algorithms, particularly policy iteration and gradient methods, to solve RMDPs with diverse uncertainty sets, including non-rectangular and ambiguity sets defined by phi-divergences. These advancements address computational challenges and improve the applicability of RMDPs to real-world problems where model uncertainty is prevalent, leading to more reliable and robust decision-making in various domains.

Papers

November 18, 2024

Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
Marnix Suilen, Thom Badings, Eline M. Bovy, David Parker, Nils Jansen
Reinforcement Learning Artificial Intelligence Markov Decision Process Policy Iteration Place Formal Method MDP Model Robust Markov Decision Process

October 29, 2024

Policy Gradient for Robust Markov Decision Processes
Qiuhao Wang, Shaohang Xu, Chin Pang Ho, Marek Petrick
Policy Gradient Policy OpTimization Policy Mirror Descent Robust Markov Decision Process

December 21, 2023

Solving Long-run Average Reward Robust MDPs via Stochastic Games
Krishnendu Chatterjee, Ehsan Kafshdar Goharshady, Mehrdad Karrabi, Petr Novotný, Đorđe Žikelić
Markov Decision Process Stochastic Game Robust Markov Decision Process

May 30, 2023

Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets
Mengmeng Li, Daniel Kuhn, Tobias Sutter
Policy Gradient Robust Policy Uncertainty Set Robust Markov Decision Process Robust Policy Evaluation

April 24, 2023

Model-Free Learning and Optimal Policy Design in Multi-Agent MDPs Under Probabilistic Agent Dropout
Carmel Fiscko, Soummya Kar, Bruno Sinopoli
Multi Agent Optimal Policy Structured Dropout Robust Policy Free Learning Robust Markov Decision Process Dropout Method Robust Markov Decision

March 12, 2023

Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization
Esther Derman, Yevgeniy Men, Matthieu Geist, Shie Mannor
Native Robustness Markov Decision Process Policy Regularization Regularized Reinforcement Learning Robust Markov Decision Process

February 2, 2023

Robust Markov Decision Processes without Model Estimation
Wenhao Yang, Han Wang, Tadashi Kozuno, Scott M. Jordan, Zhihua Zhang
Parameter Estimation Robust Policy Robust Markov Decision Process Markovian Sequence

January 31, 2023

Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factor
Julien Grand-Clément, Marek Petrik
Markov Decision Process Optimality Gap Robust Markov Decision Process Discount Factor

January 3, 2023

Risk-Averse MDPs under Reward Ambiguity
Haolin Ruan, Zhi Chen, Chin Pang Ho
Markov Decision Process Reward Ambiguity Robust Markov Decision Process Robust Risk

September 28, 2022

Online Policy Optimization for Robust MDP
Jing Dong, Jingwei Li, Baoxiang Wang, Jingzhao Zhang
Reinforcement Learning Robust Markov Decision Process Optimistic Policy Gradient

May 28, 2022

Efficient Policy Iteration for Robust Markov Decision Processes via Regularization
Navdeep Kumar, Kfir Levy, Kaixin Wang, Shie Mannor
Policy Iteration Robust Markov Decision Process Rectangular Robust Markov Decision Process

May 27, 2022

Robust Phi-Divergence MDPs
Chin Pang Ho, Marek Petrik, Wolfram Wiesemann
Robust Markov Decision Process Ambiguity Set Rectangular Robust Markov Decision Process

January 30, 2022

The Geometry of Robust Value Functions
Kaixin Wang, Navdeep Kumar, Kuangqi Zhou, Bryan Hooi, Jiashi Feng, Shie Mannor
Reinforcement Learning Markov Decision Process Geometric Analysis Function Space Robust Markov Decision Process