Markov Decision Processes

Follow this topic Ask about this topic

Recent momentum

-78%

6 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

4 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Markov Decision Processes.

173 papers

Latest in Markov Decision Processes

Open your feed →

Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
Sep 17, 2026Weiwei Wang, Yuqiang Li, Xianyi Wu +1Off-Policy Evaluation Multiplier Bootstrap
Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation
Sep 17, 2026Tung Tran, Viet Bao Mai, Hoang Ta +1Monte Carlo Tree Search Markov Decision Processes
A Geometric Theory of Decision Boundaries in Structured Markov Decision Processes
Sep 16, 2026Fredy PokouMarkov Decision Processes Boundaries
Strong and Compact Policies for Submodular Markov Decision Processes via LP-Based Submodular Orienteering
Sep 14, 2026Lars Rohwedder, Rico ZenklusenMarkov Decision Processes Submodular Maximization
Evaluation Metrics for Safe Reinforcement Learning
Sep 14, 2026Lindsay Spoor, Aske Plaat, Thomas MoerlandSafety Evaluation Joint Reinforcement Learning
Supermartingale Certificates for Parametric MDPs
Sep 14, 2026Kaushik Mallik, \DHorj̣e ŽikelićMarkov Decision Processes Conformal Test Martingales
Adaptive Agent Design
Sep 14, 2026Raj Kiriti Velicheti, Subhonmesh Bose, Tamer BaşarQ-Learning Markov Decision Processes
A Bellman Optimality Equation for Plasticity
Sep 11, 2026Jeremy Lucas, Doina PrecupPlasticity Bellman Equation
DIA: Denoising Intermediate Advantage for Diffusion Policy Optimization
Sep 10, 2026Arjun Sohal, Yuchi Zhao, Miroslav Bogdanovic +1Diffusion Policies Behavior Cloning
Belief-State Engine: Augmenting LLMs for Principled Planning Under Partial Observability
Sep 9, 2026Arnab Chattopadhayay, Debdipta HalderPartially Observable Markov Decision Process Latent States
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study
Sep 7, 2026Aayush Patel, Andrzej RuszczyńskiMarkov Decision Processes Social Robot Navigation
Temporal-Causal Inference for Reinforcement Learning via Automata Learning
Sep 7, 2026Jan Corazza, Daniil Kaminskyi, Simon Lutz +4Entropy Regularized Reinforcement Learning Markov Decision Processes
GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space
Aug 31, 2026Boqi Chen, Xudong Liu, Yunke Ao +2Large Language Model Agents Opti-Agent-Bench
Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry
Aug 13, 2026Larissa Xu, King Bi, William ChangQ-Learning Multi-Agent Reinforcement Learning
Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problem
Aug 12, 2026Hongyao TangAgent Memory Substantial Memory
Threshold Structure of Optimal Policies in Restart POMDPs
Aug 11, 2026Konstantin Avrachenkov, Alexey Piunovskiy, Yi ZhangPartially Observable Markov Decision Process Markov Decision Processes
Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
Aug 10, 2026Chenhua Fan, Jiahui Zhu, Yuhang Zhang +1Policy Gradient Safety Constraints
Path-dependent Discrete Amortized Inference
Aug 9, 2026Tiago da Silva, Esmeralda S. Whitammer, Salem LahlouMarkov Decision Processes Posterior
Finite Constant Frontiers and Auditable Regret Certificates for Average-Reward Reinforcement Learning
Aug 7, 2026Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Md Najmus SwaqeebEntropy Regularized Reinforcement Learning Markov Decision Processes
From Optimal Actions to World Models: Identifiability of Transition Kernels in Discounted MDPs
Aug 7, 2026Neal BatraMarkov Decision Processes Optimal Actions
Sub-Quadratic Bisimulation Metrics via Approximate Nearest Neighbors: Coverage-Augmented Guarantees and Computable Two-Sided Certificates
Aug 7, 2026Ibne Farabi Shihab, Joyanta Jyoti MondalFinite-Sample Certificates Markov Decision Processes
Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions
Aug 6, 2026Yuepeng Yang, Yuxin Chen, Yuejie ChiMarkov Decision Processes Optimal Policies
Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attacks
Aug 6, 2026Ximing Sun, Yue WangByzantine Attacks Multi-Agent Systems
MACRO: Markov Chain Routing of Transformer Layers
Aug 6, 2026Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2Large Language Model Routing Transformer Layers
Exact Model-Free Policy Iteration for Co-safe LTL Planning
Aug 5, 2026Zetong Xuan, Yu WangModel-Free Reinforcement Learning Linear Temporal Logics
Optimal Constrained sc-LTL Planning in MDPs via Switching Policies
Aug 5, 2026Zetong Xuan, Yu WangLinear Temporal Logics Markov Decision Processes
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
Aug 3, 2026Evan Assmus, Qining Zhang, Lei YingProximal Policy Optimization Reward Modeling
Robust Counterfactual Policy Optimisation via Nondeterministic Causal Models
Aug 3, 2026Jessica Lally, Milad Kazemi, Nicola Paoletti +2Counterfactual Rollouts Counterfactual Reasoning
Improved Quantum Algorithms for Reinforcement Learning Under a Generative Model
Aug 3, 2026Joao F. DoriguelloQ-Learning Quantum Advantage
Optimizing Minimax Regret in Uncertain MDPs with Small Sets of Policies
Aug 3, 2026Sterre Lutz, Daniël Vos, Matthijs T. J. Spaan +1Markov Decision Processes Minimax
PAC Approximation and DIRECT Optimization for Parametric Markov Models
Aug 3, 2026Zhiming Chi, Ying Liu, Andrea Turrini +2Markov Decision Processes Llm-Based Optimization Modeling
Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs
Jul 30, 2026Ankur Naskar, Vaneet AggarwalSoft Actor-Critic Markov Decision Processes
Property-driven Causal Abstractions for Markov Decision Processes
Jul 29, 2026Jule Schmidt, Maximilian Weininger, Clemens Dubslaff +2Markov Decision Processes Structural Causal Model
dtControl2+ $\varepsilon$ : Trading Optimality for Explainability in MDPs via Decision TreesdtControl2+\varepsilon: Trading Optimality for Explainability in MDPs via Decision Trees
Jul 28, 2026Tereza Kinská, Jan Křetínský, Tobias Meggendorfer +2Decision Trees Markov Decision Processes
A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics
Jul 28, 2026Zheshun Wu, Renjie Zheng, Jinhang Zuo +2Offline Reinforcement Learning Markov Decision Processes
Online Policy Evaluation for MDPs with Dynamic UBSR Measures
Jul 25, 2026Weikai Wang, Erick DelageMarkov Decision Processes Online
Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes
Jul 25, 2026Asha Barua, Sajad KhodadadianPolicy Gradient Finite Time Analysis
Active Inference as a Convex Markov Decision Process
Jul 22, 2026Nikola Milosevic, Nicolás Hinrichs, Nico ScherfEntropy Regularized Reinforcement Learning Markov Decision Processes
Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces
Jul 20, 2026Dongming Wang, Pengcheng Dai, Wenwu Yu +1Multi-Agent Reinforcement Learning Markov Decision Processes
Beyond Fixed Goal Delivery: Online POMDP Planning for Target Interception in Crowds
Jul 20, 2026Himanshu Gupta, Kelvin Aladum, Nisar Ahmed +2Coverage Path Planning Planning Methods
Learning-Driven Adaptive Audit Scheduling: A Sequential Decision Approach to Off-Chain Data Integrity
Jul 19, 2026Changting Lin, Fan Li, Weihang Yu +4Algorithm Auditing Markov Decision Processes
Non-Asymptotic Best Policy Identification Guarantees in Online Reinforcement Learning
Jul 19, 2026Joseph Lazzaro, Alessio Russo, Aldo PacchianoOptimal Policies Markov Decision Processes
Mask-Aware Policy Gradients for Diffusion Language Models
Jul 16, 2026Haran Raajesh, Kulin Shah, Adam Klivans +1Masked Diffusion Language Models Policy Gradient
Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes
Jul 14, 2026Jonas Ehrhardt, René Heesch, Oliver NiggemannMarkov Decision Processes Temporal Difference
Comparing Socially-Equitable Renewable Energy Budget Allocation MDP Policies in Mature and Emerging Economies
Jul 11, 2026Riya Kinnarkar, Mansur M. Arief, Yan Pratama Akhra +1Markov Decision Processes Economies
Risk-Aware General-Utility Markov Decision Processes
Jul 10, 2026Pedro P. Santos, Fábio Vital, Alberto Sardinha +1Markov Decision Processes Conditional-Value-At-Risk
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Jul 9, 2026Elham Daneshmand, Majid Khadiv, Glen Berseth +1Proximal Policy Optimization Imitation Learning
Mathematical methods of reinforcement learning
Jul 8, 2026Denis Belomestny, Alexander Gasnikov, Egor Gladin +5Entropy Regularized Reinforcement Learning Markov Decision Processes
Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
Jul 3, 2026Jialun Cao, Fernando Acero, David Šiška +1Entropy Regularized Reinforcement Learning Kullback-Leibler Regularization
Policy Optimization Achieves Data-Dependent Regret Bounds in MDPs with Unknown Transitions
Jun 30, 2026Mingyi Li, Taira Tsuchiya, Kenji YamanishiMarkov Decision Processes Transitions
Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees
Jun 29, 2026Ryohei Oura, Georgios Fainekos, Hideki Okamoto +1Markov Decision Processes Reachability
Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPs
Jun 27, 2026Kai HidajatOptimal Policies Markov Decision Processes
Learning in Markovian bandits with non-observable states and constrained decision epochs
Jun 25, 2026Thomas Hira, Victor Boone, Urtzi Ayesta +1O(T^Β)$ Simultaneous Regret Markov Decision Processes
Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
Jun 24, 2026Aniruddha Joshi, Niklas Lauffer, Sanjit SeshiaConstrained Reinforcement Learning Optimal Policies
Confidence Sequences for Online Statistical Model Checking of Markov Decision Processes
Jun 24, 2026Konstantin Kueffner, Tobias Meggendorfer, Maximilian Weininger +1Markov Decision Processes Confidence Intervals
Low-Complexity Policy Tessellations in Structured Markov Decision Processes
Jun 24, 2026Fredy PokouMarkov Decision Processes Optimal Policies
Minimax PAC Bounds for Learning in Exogenous Contextual MDPs
Jun 23, 2026Corentin Pla, Hugo Richard, Marc Abeille +1Markov Decision Processes Minimax
Solving Markov Decision Processes with Future Information via MPC
Jun 23, 2026Shambhuraj Sawant, Akhil S Anand, Dirk Reinhardt +1Model Predictive Control Markov Decision Processes
Welfarist Control Design -- How to fulfill the societal mandate in multi-agent control?
Jun 22, 2026Sophie Hall, Kai Zhang, Ilia Shilov +2Feedback Linearization Sociotechnical Systems
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Jun 20, 2026Zhao Yang, Yuxuan Jiang, Ting-Chih Chen +18Deep Reinforcement Learning Off-Policy Evaluation

Sep 17, 2026 · stat.MLJ/K move · Enter open · S save

Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning

Weiwei Wang, Yuqiang Li, Xianyi Wu, Bingyi Jing

Offline policy evaluation (OPE) is crucial in high-stakes reinforcement learning applications, where new policies must be assessed reliably before deployment. In such settings, point estimates alone are insufficient; principled uncertainty quantification, such as confidence intervals and variance estimates, is essential for safe and risk-aware decision-making. A comprehensive way to unify these tasks is to estimate the sampling distribution of the evaluation error. Existing approaches, however, often suffer from limited robustness, scalability, or finite-sample validity. In this paper, we propose a model-based bootstrap framework for uncertainty quantification of OPE in finite-horizon, time-inhomogeneous Markov decision processes (MDPs). Unlike classical bootstrap methods that rely on resampling complete episodes, the proposed method regenerates trajectories from an estimated MDP and can therefore accommodate a much broader range of offline data formats, including complete trajectories, transition-level observations, and trajectory fragments. This flexibility further improves finite-sample statistical efficiency. We establish bootstrap distributional consistency, asymptotically valid confidence intervals, and consistent variance estimation for the target policy value. Extensive simulations show that the proposed method accurately captures the sampling distribution of the OPE estimator, yielding tighter confidence intervals and more accurate variance estimates in most settings.

Off-Policy Evaluation Multiplier Bootstrap Markov Decision Processes Estimators

Open paper Ask about it PDF