Robust Markov Decision Processes

Ask about this topic

Overview

Robust Markov decision processes study how to plan and learn when transition dynamics—or sometimes rewards—are uncertain, often choosing actions to maximize performance against the worst plausible model. Methods define uncertainty sets using intervals, polytopes, total variation, or Wasserstein distance, then solve robust Bellman equations with dynamic programming, linear programming, or policy iteration. Learning approaches include robust Q-learning, actor-critic and policy-gradient methods, plus plug-in and PAC methods that estimate dynamics while providing safety or performance guarantees. A central tension is that worst-case protection can make policies overly conservative, while coupled uncertainty, average-reward transients, and complex objectives can make efficient planning and reliable guarantees difficult.

Latest papers

CardsList

27 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. Provably Efficient Q-learning for Robust Reinforcement Learning

    Oct 26, 2026Yuchen Jiao, Gen LiRobust RLDistributionally Robust RL

    Department of Statistics and Data Science, Chinese University of Hong Kong.

  2. Reward Objectives for Robust (PO)MDPs: A Law-Preserving Equivalence with (Partially Observable) Games

    Oct 19, 2026Durgam Latha, Dion Reji, Dorde Zikelic +1Partially Observable Stochastic GamesRobust Partially Observable Markov Decision Processes

    Indian Institute of Technology Bombay Mumbai, India · Nanyang Technological University Singapore, Singapore

  3. Robust Counterfactuals for Markov Decision Processes under Statistical and Causal Uncertainties

    Oct 14, 2026Jessica Lally, Milad Kazemi, Nicola PaolettiCounterfactual ExplanationsMarkov Decision Processes

    King’s College London London, United Kingdom · Durham University Durham, United Kingdom

  4. Scalable Robust MCTS with Anytime Deterministic Optimality Guarantees

    Oct 11, 2026Tomer Massas, Vadim IndelmanRobust Markov Decision ProcessesModel-Based Planning

    Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology · Stephen B. Klein Faculty of Aerospace Engineering, Technion – Israel Institute of Technology · Faculty of Data and Decision Sciences, Technion – Israel Institute of Technology

  5. Linear Programming Representations and Strongly Polynomial Algorithms for Robust Markov Decision Processes

    Oct 1, 2026Han Zhong, Yinyu YeRobust Markov Decision ProcessesMarkov Decision Processes

    Shanghai Jiao Tong University · Shanghai Jiao Tong University, SIMIS, and Stanford University

  6. Vector Bellman Theory for Multichain Robust Average-Reward Markov Decision Processes

    Sep 23, 2026Yue Wang, George AtiaRobust Markov Decision ProcessesMarkov Decision Processes

    Department of Electrical and Computer Engineering University of Central Florida

  7. Supermartingale Certificates for Parametric MDPs

    Sep 11, 2026Kaushik Mallik, Ðorđe ŽikelićRobust Markov Decision ProcessesMarkov Decision Processes

    IMDEA Software Institute, Madrid, Spain

  8. Quantitative Analysis of ωω-Regular Robust MDPs

    Aug 26, 2026Ali Asadi, Krishnendu Chatterjee, Ehsan Kafshdar Goharshady +3Robust Markov Decision ProcessesSequential Decision Making

    Institute of Science and Technology Austria

  9. Solving Robust POMDPs with Omega-regular Objectives via Partially Observable Stochastic Games

    Aug 25, 2026Durgam Latha, Dion Reji, S. Akshay +2Robust Markov Decision ProcessesPartially Observable Stochastic Games

    Indian Institute of Technology Bombay Mumbai, India · Nanyang Technological University Singapore, Singapore

  10. Adaptive Policy Portfolios for Robust Markov Decision Processes

    Aug 18, 2026Kasper Engelen, Sebastian Junges, Guillermo A. Pérez +1Robust Markov Decision ProcessesPolicy Learning

    University of Antwerp, Belgium · Radboud University, Nijmegen, The Netherlands

  11. Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions

    Aug 6, 2026Yuepeng Yang, Yuxin Chen, Yuejie ChiRobust Markov Decision ProcessesAverage-Reward RL

    *Department of Statistics and Data Science, Yale University. · †Department of Statistics and Data Science, the Wharton School, University of Pennsylvania.

  12. PAC Approximation and DIRECT Optimization for Parametric Markov Models

    Aug 3, 2026Zhiming Chi, Ying Liu, Andrea Turrini +2Robust Markov Decision ProcessesMarkov Decision Processes

    Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2

  13. Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

    Jul 3, 2026Jialun Cao, Fernando Acero, David Šiška +1Robust Markov Decision ProcessesMaximum Entropy RL

    School of Mathematics, University of Edinburgh, United Kingdom. · J.P. Morgan AI Research, United Kingdom. · Department of Mathematics, Imperial College London, United Kingdom.

  14. Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs

    Jun 12, 2026Tianhao Wu, Matthew Zurek, Weina Wang +1Multi-Armed BanditsRobust Markov Decision Processes

    Department of Industrial and Systems Engineering, University of Wisconsin-Madison · Department of Computer Sciences, University of Wisconsin-Madison · Computer Science Department, Carnegie Mellon University

  15. Policy Gradient for Continuous-Time Robust Markov Decision Processes

    Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaRobust Markov Decision ProcessesMarkov Decision Processes

    Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR) · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR) · College of Computing and Data Science, Nanyang Technological University

  16. Robust Shielding for Safe Reinforcement Learning

    May 29, 2026Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2Robust Markov Decision ProcessesFormal Verification

    Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1

  17. Robust Parameter Learning for Uncertain MDPs

    May 2, 2026Yannik Schnitzer, Alessandro Abate, David ParkerRobust Markov Decision ProcessesParameter Estimation

    University of Oxford, Oxford, United Kingdom

  18. Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approach

    Mar 9, 2026Sivaramakrishnan RamaniRobust Markov Decision ProcessesDistributionally Robust Optimization

    Department of Computational Applied Mathematics and Operations Research, Rice University, Houston, USA.

  19. Robust Transfer Learning with Side Information

    Mar 9, 2026Akram S. Awad, Shihab Ahmed, Yue Wang +1Reinforcement LearningRobust Markov Decision Processes

    Department of Electrical and Computer Engineering, University of Central Florida, USA · Department of Computer Science, University of Central Florida, USA

  20. Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Values

    Mar 1, 2026Shengbo Wang, Nian SiRobust Markov Decision ProcessesAverage-Reward RL

    Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California · Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology

  21. Resilient Strategies for Stochastic Systems: How Much Does It Take to Break a Winning Strategy?

    Feb 27, 2026Kush Grover, Markel Zubia, Debraj Chakraborty +3Robust Markov Decision Processes

    Fondazione Bruno Kessler, Italy · Ruhr University Bochum, Germany · Nanyang Technological University, Singapore +3

  22. Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes

    Feb 2, 2026Mingyuan Xu, Zongqi Xia, Tianxi Cai +2Robust Markov Decision ProcessesOffline RL

    Department of Statistics and Data Science, National University of Singapore · Department of Neurology, University of Pittsburgh · Department of Biostatistics, Harvard T.H. Chan School of Public Health +2

  23. Bayesian Ambiguity Contraction-based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missions

    Dec 1, 2025Jimin Choi, Max Z. LiRobust Markov Decision ProcessesModel-Based Planning

    University of Michigan, Ann Arbor, MI 48109, USA

  24. Optimization-Based Robust Permissive Synthesis for Interval MDPs

    Oct 3, 2025Khang Vo Huynh, David Parker, Lu FengRobust Markov Decision ProcessesMarkov Decision Processes

    Department of Computer Science, University of Virginia, Charlottesville, VA, USA · Department of Computer Science, University of Oxford, Oxford, UK

  25. Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain

    Sep 17, 2025Shengbo Wang, Nian SiRobust Markov Decision ProcessesErgodic Control

    Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California · Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology

  26. Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning

    Jun 8, 2025Yang Xu, Swetha Ganesh, Vaneet AggarwalRobust Markov Decision ProcessesReinforcement Learning

    Purdue University, West Lafayette, Indiana, USA 47907

  27. Distributionally Robust Deep Q-Learning

    May 25, 2025Chung I Lu, Julian Sester, Aijia ZhangRobust Markov Decision ProcessesDeep Q-Learning

    National University of Singapore, Department of Mathematics, 21 Lower Kent Ridge Road, 119077.