Markov Decision Processes
Overview
Markov decision processes describe how an agent chooses actions as a system changes state probabilistically and rewards accumulate over time. Dynamic programming methods such as value iteration and policy iteration use Bellman equations to plan, while reinforcement learning methods such as Q-learning, policy gradients, and posterior sampling learn when transition models are unknown. Current work extends these tools to settings with constraints, risk, partial observability, non-stationarity, and large or structured state spaces, using methods such as robust optimization, state abstraction, and function approximation. A central tension is obtaining reliable safety and performance guarantees while keeping learning and planning computationally and statistically efficient beyond small tabular problems.
Latest papers
177 papers15 papers in the last four weeks, down 53% on the four weeks before. 0.1% of all new papers.
Approximate Dynamic Programming with Basis Function Learning
GP--RL--MPC: Uncertainty-Aware Reinforcement Learning-Augmented Model Predictive Control with Gaussian Process Postdecision State-Value Functions
Subprocess-Constrained Markov Decision Processes
Lossless First-Action Menus for Linear-Reward Planning
Improved Regret via Variance-Aware Policy Optimization in Adversarial MDPs
Introducing Feedback Delays to Non-Delayed Adversarial MDPs: Optimal and Dependencies in Tabular and Linear AMDPs
Understanding the Asymptotic Transience of Q-Learning via Distributional Expansion
An Analysis of Reinforcement Learning with Function Approximation
Provably Efficient Representation Learning for Low-Rank CMDPs
MFRLC (Model-Free Representation Learning for Low-Rank CMDP), a model-free algorithm that uses a primal-dual scheme to effectively balance reward regret and constraint violations. To the best of our knowledge, MFRLC is the first representation-learning method for low-rank CMDPs that combines the Least-Squares Value Iteration with Upper Confidence Bound (LSVI-UCB) and primal-dual techniques. MFRLC further enhances value function estimation with bonuses for function approximation and learns the mapping from representations to value functions directly, avoiding explicit estimation of the transition kernel. We prove that, with high probability, MFRLC achieves reward regret and cumulative constraint violation over deployment rounds, independent of the size of the state space. To our knowledge, these are the first sublinear regret and violation guarantees for CMDPs with unknown features.