Markov Decision Processes

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

16 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Period ending 2026-09-14

11 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Period ending 2026-09-07

5 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Inside this field

Focused directions

Focused topic

Markov Decision Processes

Within Markov Decision Processes

174 papersExplore

Focused topic

Optimal Policies

Within Markov Decision Processes

105 papersExplore

Focused topic

Value Functions

Within Markov Decision Processes

71 papersExplore

Focused topic

Bellman Equation

Within Markov Decision Processes

67 papersExplore

Focused topic

Partially Observable Markov Decision Process

Within Markov Decision Processes

66 papersExplore

419 papers

Latest in Markov Decision Processes

Browse all topics →

Scalar-Stepsize Nonuniform Monte Carlo Optimistic Policy Iteration: A Certified Counterexample
Jun 14, 2026Yuanlong ChenStochastic Approximation Optimal Policies
PO-PDDL: Learning Symbolic POMDPs from Visual Demonstrations for Robot Planning Under Uncertainty
Jun 14, 2026Wenjing Tang, Xuanjin Jin, Yuan Liu +3Partially Observable Markov Decision Process Belief Space
Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs
Jun 12, 2026Tianhao Wu, Matthew Zurek, Weina Wang +1Markov Decision Processes Sample Complexity
Utility-Constrained Policy Optimization
Jun 12, 2026Mehrdad Moghimi, Bernardo Avila PiresConstrained Reinforcement Learning Safety Constraints
UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning
Jun 10, 2026Haoyuan Deng, Yitong Gao, Yudong Lin +3Value Functions Agentic Tool Use
A Mathematical Theory of Value: a synthesis on goal-directed agency under resource constraints
Jun 10, 2026Cheng QianValue Goal-Directed Behavior
Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems
Jun 10, 2026Xin Guo, Yijie Huang, Xiang YuOptimal Policies Optimal Control
Bellman-sufficient Information Complexity
Jun 9, 2026Yunbei XuInformation-Theoretic Limits Optimal Sample Complexity
Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets
Jun 9, 2026Yi Chen, Rushuai Yang, Qiang Chen +2Markov Decision Processes Bellman Equation
Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updates
Jun 9, 2026Octave Oliviers, Glenn VinnicombeOptimal Policies Monte Carlo
Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models
Jun 9, 2026Samuel Tetteh, Cody FlemingSafety Constraints Constrained Reinforcement Learning
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
Jun 7, 2026Jiashun Liu, Runze Liu, Xu Wan +3Training-Inference Mismatch Reinforcement Learning Post-Training
FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&A
Jun 5, 2026Ambuj Mehrish, Sebastiano VasconMultimodal Documents Top-K
Bayesian learning for the stochastic shortest path problem
Jun 3, 2026Chon Wai Ho, Sumeetpal S. Singh, Jiaqi GuoMarkov Decision Processes Bayesian
Think Fast and Far: Long-Horizon Online POMDP Planning via Rapid State Sampling
Jun 3, 2026Yuanchu Liang, Edward Kim, J. Arden Knoll +4Partially Observable Markov Decision Process
Policy Gradient for Continuous-Time Robust Markov Decision Processes
Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaMarkov Decision Processes Policy Gradient
Taiji: Pareto Optimal Policy Optimization with Semantics-IDs Trade-off for Industrial LLM-Enhanced Recommendation
Jun 2, 2026Yuecheng Li, Zeyu Song, Jing Yao +3Recommendation Industrial
A Goal-Set Characterization of Task Composition in the Boolean Task Algebra
Jun 2, 2026Eduardo Terrés-Caballero, Herke van HoofValue Functions Goal-Conditioned Reinforcement Learning
Minimax-Optimal Policy Regret in Partially Observable Markov Games
Jun 1, 2026Raman AroraPartially Observable Markov Decision Process Expectation-Maximization
ShaplEIG: Bayesian Experimental Design for Shapley Value Estimation
Jun 1, 2026David Rundel, Fabian Fumagalli, Maximilian Muschalik +2Shapley Value Bayesian Experimental Design
Task-Induced Representational Invariances Depend on Learning Objective in Deep RL
Jun 1, 2026Manu Srinath Halvagal, Sebastian Lee, SueYeon ChungDeep Reinforcement Learning Representation Learning
SEArch: Optimistic Policy Selection Between Scene Noise and Drift for UAV Radar Search
May 31, 2026Noor Khial, Naram Mhaisen, Loay Ismail +1Unmanned Aerial Vehicles Radar
Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback
May 31, 2026Wyame Benslimane, Tinghan Ye, Pascal Van Hentenryck +1Linear Contextual Bandits Optimal Policies
Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment
May 31, 2026Xun Shen, Yuepeng Wang, Akifumi Wachi +13Safety Constraints Treatment
Prospect-Theory Behavior from Bellman Optimality in MDPs with Catastrophic States
May 31, 2026Yujiao ChenMarkov Decision Processes Bellman Equation
Shape Your Body: Value Gradients for Multi-Embodiment Robot Design
May 30, 2026Nico Bohlinger, Jan PetersGeneralizable Robotic Manipulations Value Functions
Topology-Aware State Abstraction with Tangle Cores for Markov Decision Processes
May 29, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaMarkov Decision Processes Topological Data Analysis
Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity
May 29, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaAlgorithm Auditing Optimal Policies
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
May 29, 2026Jonathan Colaço Carr, Prakash Panangaden, Doina Precup +1Markov Decision Processes Optimal Policies
Robust Shielding for Safe Reinforcement Learning
May 29, 2026Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2Shielding Markov Decision Processes
Value Functions as Supermartingale Certificates
May 29, 2026Alessandro Abate, Daniel Contro, Mirco Giacobbe +2Value Functions Reinforcement Learning With Verifiable Reward
Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning
May 29, 2026Yike Zhao, Onno Eberhard, Malek Khammassi +2Recurrent Model Partially Observable Markov Decision Process
Multivariate Distributional Reinforcement Learning Using Sliced Divergences
May 29, 2026Baptiste Debes, Tinne TuytelaarsBregman Divergences Bellman Equation
The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems
May 29, 2026Tobias Lademann, Théo Vincent, Jan Peters +1Iterative Reinforcement Learning Markov Decision Processes
PatchWorld: Gradient-Free Optimization of Executable World Models
May 29, 2026Jiaxin Bai, Yue Guo, Yifei Dong +13World Models Markov Decision Processes
Safe Equilibrium Policy Optimization for Strategic Agent Policies
May 29, 2026Karthika Arumugam, Kiran Kumar Manku, Amit DhandaGroup Relative Policy Optimization Optimal Policies
On Distributional Reinforcement Learning in Chaotic Dynamical Systems
May 28, 2026James Rudd-Jones, Mirco Musolesi, María Pérez-OrtizChaos Bellman Equation
Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk
May 28, 2026Tim Woydt, Paul-David ZuercherOptimal Policies Thompson Sampling
The Sample Complexity of Multiclass and Sparse Contextual Bandits
May 28, 2026Liad Erez, Fan Chen, Alon Cohen +4Linear Contextual Bandits Sample Complexity
Variance-Adaptive Optimal Algorithm for Reinforcement Learning with Multinomial Logit Function Approximation
May 27, 2026Wonyoung Kim, Min-Hwan Oh, Garud Iyengar +1Near-Optimal Regret Guarantees Markov Decision Processes
Commit to the Bit: Reactive Reinforcement Learning Done Right
May 27, 2026Onno Eberhard, Claire Vernade, Michael MuehlebachQ-Learning Markov Decision Processes
Learning to Bid in Repeated Second-Price Auctions with Dynamic Values and Aggregated Feedback
May 27, 2026Benjamin Heymann, Otmane SakhiFirst-Price Auctions Optimal Policies
Human Decision-Making with AI Assistance under Correlated Features
May 27, 2026Yanru Guan, Naveen Raman, Fei FangSequential Decision Making Spurious Correlations
Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach
May 27, 2026Guang-Yuan Hao, Lars van der Laan, Aurélien Bibaut +1Entropy Regularized Reinforcement Learning Bellman Equation
Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment
May 26, 2026Gengyue Han, Yiheng FengZero-Shot Sim-To-Real Transfer Sim-To-Real Gap
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
May 26, 2026Shijin Gong, Erhan Xu, Kai Ye +3Value Functions Verifiable Rewards
In LLM Reasoning, there is Irrationality on top of Value Misalignment
May 26, 2026Kejiang Qian, Fengxiang HeLarge Language Model Reasoning Rationality
Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPs
May 24, 2026Ziyue Chen, David Šiška, Lukasz SzpruchPolicy Gradient Entropy-Regularized Equilibrium Selection
A Contractive Feedback Semantics for Reinforcement Learning
May 23, 2026Zuyuan ZhangBellman Equation Feedback Learning
Reinforcement Learning for Reachability: Guaranteeing Asymptotic Optimality
May 23, 2026Amogh Palasamudram, Jakub Svoboda, Suguman Bansal +1Optimal Policies Reachability
Evolving Robustness--Exploration Trade-off in Online Reinforcement Learning via Quantile Bayesian Risk MDPs
May 23, 2026Meichen Song, Yuhao Wang, Enlu ZhouMarkov Decision Processes Bayesian
Learning Kernel-Based MDPs from Episodic Preferential Feedback
May 22, 2026Nikola Pavlovic, Sattar Vakili, Qing ZhaoReinforcement Learning From Human Feedback Entropy Regularized Reinforcement Learning
Reflex: Reinforcement Learning with Reflection Symmetry Exploitation in State-Based Continuous Control
May 22, 2026Shuai Zhen, Yifan Zhang, Yuling Wang +1Markov Decision Processes Reflections
Pure Exploration for a Good Policy in Reinforcement Learning with Bandit Feedback
May 22, 2026Zitian Li, Wang Chi CheungOptimal Policies Bandits
Abstraction for Offline Goal-Conditioned Reinforcement Learning
May 21, 2026Clarisse Wibault, Alexander Goldie, Antonio Villares +2Goal-Conditioned Reinforcement Learning Markov Decision Processes
A note on convergence of Wasserstein policy optimization
May 21, 2026David Šiška, Yufei ZhangWasserstein Gradient Flows Markov Decision Processes
Generative Modeling by Value-Driven Transport
May 21, 2026Pablo Moreno-Muñoz, Adrian Müller, Gergely NeuGenerative Models Stochastic Optimal Control
Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
May 21, 2026Wei Liu, Ting LongOffline Reinforcement Learning Bellman Equation
Scaling Observation-aware Planning in Uncertain Domains
May 21, 2026Adrian Zvizdenco, Arthur Conrado Veiga Bosquetti, Alberto Lluch Lafuente +1Partially Observable Markov Decision Process Partial Observability
On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalents
May 20, 2026Oliver Mortensen, Mohammad Sadegh TalebiValue Functions Markov Decision Processes

Jun 14, 2026 · cs.LGJ/K move · Enter open · S save

Scalar-Stepsize Nonuniform Monte Carlo Optimistic Policy Iteration: A Certified Counterexample

Yuanlong Chen

Tsitsiklis proved convergence of Monte Carlo optimistic policy iteration under a uniform update structure and identified nonuniform update frequencies as a delicate obstruction. We give a certified negative answer for the natural scalar-stepsize, unnormalized asynchronous state-value recursion with fixed nonuniform state-selection probabilities. In a three-state, two-action discounted MDP, the nonuniform update frequencies induce a diagonally scaled greedy-policy mean field with a certified nonconstant attracting hybrid periodic orbit. With a bounded unbiased geometric-horizon estimator and Robbins--Monro stepsizes, the original stochastic recursion remains trapped near the cycle with positive probability and therefore fails to converge. The example pinpoints a geometric obstruction: uniform sampling gives radial residual contraction, whereas scalar nonuniform sampling anisotropically distorts the residual dynamics and can generate switched attracting cycles.

Stochastic Approximation Optimal Policies Monte Carlo Estimators Counterexample

Open paper Ask about it PDF