Markov Decision Processes

Follow this topic Ask about this topic

Markov decision processes describe how an agent chooses actions as a system changes state probabilistically and rewards accumulate over time. Dynamic programming methods such as value iteration and policy iteration use Bellman equations to plan, while reinforcement learning methods such as Q-learning, policy gradients, and posterior sampling learn when transition models are unknown. Current work extends these tools to settings with constraints, risk, partial observability, non-stationarity, and large or structured state spaces, using methods such as robust optimization, state abstraction, and function approximation. A central tension is obtaining reliable safety and performance guarantees while keeping learning and planning computationally and statistically efficient beyond small tabular problems.

Latest papers

159 papers30 papers in the last four weeks, up 150% on the four weeks before. 0.2% of all new papers.

To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning
Oct 3, 2025Yuda Song, Dhruv Rohatgi, Aarti Singh +1Partially Observable RL Learning Using Privileged Information
Carnegie Mellon University · Massachusetts Institute of Technology · Aurora Innovation
Beyond Slater's Condition in Online CMDPs with Stochastic and Adversarial Constraints
Sep 24, 2025Francesco Emanuele Stradi, Eleonora Fidelia Chiefari, Matteo Castiglioni +2Markov Decision Processes Constrained RL
Politecnico di Milano
From Dueling Bandits to Episodic MDPs: Learning with Adversarial Preference Feedback
Jul 15, 2025Taira Tsuchiya, Shinji Ito, Haipeng LuoPreference-Based RL Dueling Bandits
The University of Tokyo and RIKEN · University of Southern California
Learning The Minimum Action Distance
Jun 10, 2025Lorenzo Steccanella, Joshua B. Evans, Özgür Şimşek +1Markov Decision Processes State Representation Learning
Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain · Department of Computer Science, University of Bath, Bath, United Kingdom
Adaptive Resolving Methods for Markov Decision Processes with Function Approximations
May 17, 2025Jiashuo Jiang, Yinyu Ye, Yiming ZongMarkov Decision Processes Reinforcement Learning
Department of Industrial Engineering & Decision Analytics, The Hong Kong University of Science and Technology · Department of Management Science & Engineering, Stanford University
Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis
Mar 24, 2025Mohsen Amiri, Sindri MagnússonNon-Stationary RL Markov Decision Processes
Department of Computer and System Science, Stockholm University, SE-164 25 Stockholm, Sweden
Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning
Sep 22, 2024Jia Wan, Sean R. Sinclair, Devavrat Shah +1Regret Minimization in RL Markov Decision Processes
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology · Department of Industrial Engineering and Management Sciences, Northwestern University
Thompson Sampling for Infinite-Horizon Discounted Decision Processes
May 14, 2024Daniel Adelman, Cagla Keceli, Alba V. Olivares-NadalThompson Sampling Regret Minimization in RL
Booth School of Business, The University of Chicago, Chicago, IL 60637, USA · United Airlines, Chicago, IL 60605, USA · UNSW Business School, UNSW, Kensington NSW 2033, Australia
Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration
Oct 11, 2023Zeyang Li, Chuxiong Hu, Yunan Wang +4Markov Decision Processes Reinforcement Learning
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA 02139, United States · Department of Mechanical Engineering, Tsinghua University, Beijing 100084, China · College of Artificial Intelligence and the School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China

Oct 3, 2025J/K move · Enter open · S save

To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning

Yuda Song, Dhruv Rohatgi, Aarti Singh, J. Andrew Bagnell

Carnegie Mellon University · Massachusetts Institute of Technology · Aurora Innovation

Partially Observable RL Learning Using Privileged Information Policy Distillation Markov Decision Processes

Open paper AskPDF arXiv

Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent policies. Recent empirical successes have used privileged expert distillation--which leverages availability of latent state information during training (e.g., from a simulator) to learn and imitate the optimal latent, Markovian policy--to disentangle the task of "learning to see" from "learning to act". While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes. In this paper--through a simple but instructive theoretical model called the perturbed Block MDP, and controlled experiments on challenging simulated locomotion tasks--we investigate the algorithmic trade-off between privileged expert distillation and standard RL without privileged information. Our main findings are: (1) The trade-off empirically hinges on the stochasticity of the latent dynamics, as theoretically predicted by contrasting approximate decodability with belief contraction in the perturbed Block MDP; and (2) The optimal latent policy is not always the best latent policy to distill. Our results suggest new guidelines for effectively exploiting privileged information, potentially advancing the efficiency of policy learning across many practical partially observable domains.