Markov Decision Processes

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

16 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Period ending 2026-09-14

11 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Period ending 2026-09-07

5 new papers

A weekly snapshot of new work published in Markov Decision Processes.

Inside this field

Focused directions

Focused topic

Markov Decision Processes

Within Markov Decision Processes

174 papersExplore

Focused topic

Optimal Policies

Within Markov Decision Processes

105 papersExplore

Focused topic

Value Functions

Within Markov Decision Processes

71 papersExplore

Focused topic

Bellman Equation

Within Markov Decision Processes

67 papersExplore

Focused topic

Partially Observable Markov Decision Process

Within Markov Decision Processes

66 papersExplore

419 papers

Latest in Markov Decision Processes

Browse all topics →

Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL
May 8, 2026Shengbo Wang, Zexi ZhangCentral Limit Theorem Model-Free Reinforcement Learning
Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
May 8, 2026Gugan Thoppe, L. A. Prashanth, Ankur Naskar +1Q-Learning Markov Decision Processes
Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow
May 8, 2026Juil Koo, Mingue Park, Jiwon Choi +2Wasserstein Gradient Flows Behavior Cloning
Finite-Time Analysis of MCTS in Continuous POMDP Planning
May 8, 2026Da Kong, Vadim IndelmanPartially Observable Markov Decision Process Monte Carlo Tree Search
Multi-Environment POMDPs with Finite-Horizon Objectives
May 8, 2026Léonard Brice, Filip Cano, Krishnendu Chatterjee +2Partially Observable Markov Decision Process Markov Decision Processes
Improved Model-based Reinforcement Learning with Smooth Kernels
May 8, 2026Kun Long, Yuqiang Li, Xianyi WuMarkov Decision Processes Smoothing
MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments
May 8, 2026Yicheng Gao, Xiaolin Zhou, Yahan Li +2Clinical Reasoning Diagnostic Benchmark
The Context Gathering Decision Process: A POMDP Framework for Agentic Search
May 7, 2026Chinmaya Kausik, Adith Swaminathan, Nathan KallusLarge Language Model Agents Agentic Reasoning
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
May 7, 2026Boyang Xu, Qing Zou, Siqin Yang +1Flow-Based Policy Bellman Equation
Dynamic Treatment on Networks
May 7, 2026Bengusu Nar, Jiguang Li, Veronika Ročková +1Treatment Effect Optimal Policies
Optimizing Social Utility in Sequential Experiments
May 7, 2026Ander Artola Velasco, Stratis Tsirtsis, Manuel Gomez-RodriguezExpected Utility Bayesian Experimental Design
Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
May 7, 2026Xiang Li, Nan JiangOff-Policy Evaluation Value Functions
Prediction and Empowerment: A Theory of Agency through Bridge Interfaces
May 7, 2026Richard CsakyAgentic Control Partially Observable Markov Decision Process
Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonSafety Evaluation Compliance
Soft Deterministic Policy Gradient with Gaussian Smoothing
May 7, 2026Hyunjun Na, Donghwan LeeValue Functions Critic-Free Reinforcement Learning
QuadraSHAP: Stable and Scalable Shapley Values for Product Games via Gauss-Legendre Quadrature
May 7, 2026Majid Mohammadi, Grigory Reznikov, Pavel Sinitcyn +2Shapley Value Coalition Formation
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Value Functions
Resource-Constrained Robotic Planning in the face of Mixed Uncertainty
May 7, 2026Yihao Yin, Pian Yu, Andrea Turrini +3Robot Planning Robotics
A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration
May 7, 2026Manuel Haussmann, Mustafa Mert Çelikok, Melih KandemirQ-Learning Model-Free Reinforcement Learning
MEMOA: Massive Mixtures of Online Agents via Mean-Field Decentralized Nash Equilibria
May 6, 2026Xuwei Yang, David B. Emerson, Fatemeh Tavakoli +1Federated Reinforcement Learning Multi-Agent Reinforcement Learning
Provable imitation learning for control of instability in partially-observed Vlasov--Poisson equations
May 6, 2026Xiaofan Xia, Qin Li, Wenlong MouPlasma Shape Control Imitation Learning
On-line Learning in Tree MDPs by Treating Policies as Bandit Arms
May 6, 2026Anvay Shah, Ramsundar Anandanarayanan, Sharayu Moharir +1Markov Decision Processes Bandits
A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs
May 6, 2026Erel Shtossel, Alicia Vidler, Uri Shaham +1Markov Decision Processes
Exact Dual Geometry of SOC-ICNN Value Functions
May 6, 2026Kang Liu, Jianchen Hu, Wei PengGraph Neural Network Inference Convex Sets
Structural Equivalence and Learning Dynamics in Delayed MARL
May 5, 2026Jules Sintes, Ana Bušić, Jiamin ZhuPartially Observable Markov Decision Process Multi-Agent Reinforcement Learning
Optimal Posterior Sampling for Policy Identification in Tabular Markov Decision Processes
May 5, 2026Cyrille Kone, Kevin JamiesonOptimal Policies Markov Decision Processes
Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity
May 5, 2026Riddhiman Bhattacharyya, Sayak Chakrabarty, Imon BanerjeeMarkov Decision Processes Linear Contextual Bandits
Taming the Curses of Multiagency in Robust Markov Games with Large State Space through Linear Function Approximation
May 4, 2026Jingchu Gai, Laixi ShiMulti-Agent Reinforcement Learning Markov Decision Processes
The Design and Composition of Structural Causal Decision Processes
May 4, 2026Sebastian Benthall, Alan LujanStructural Causal Model Agent Decisions
Middle-mile logistics through the lens of goal-conditioned reinforcement learning
May 4, 2026Onno Eberhard, Thibaut Cuvelier, Michal Valko +1Goal-Conditioned Reinforcement Learning Last Mile
Reinforcement Learning Trained Observer Control for Bearings-Only Tracking
May 4, 2026Branko Ristic, Sanjeev ArulampalamExtended Kalman Filter Deep Q-Networks
Bridging the Gap Between Average and Discounted TD Learning
May 3, 2026Haoxing Tian, Zaiwei Chen, Ioannis Ch. Paschalidis +1Temporal Difference Bellman Equation
Regularized Centered Emphatic Temporal Difference Learning
May 2, 2026Xingguo Chen, Chaohui Wu, Jinguo Ye +5Temporal Difference Regularization Techniques
Robust Parameter Learning for Uncertain MDPs
May 2, 2026Yannik Schnitzer, Alessandro Abate, David ParkerMarkov Decision Processes Confidence Intervals
Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
May 2, 2026Ruiquan Huang, Donghao Li, Yingbin Liang +1Soft Actor-Critic Markov Decision Processes
Right-Sizing Communication and Recommendation Set Size in AI-Assisted Search
May 2, 2026Jing Dong, Prakirt Raj Jhunjhunwala, Yash KanoriaAgentic Search Communication Efficiency
Value Functions for Temporal Logic: Optimal Policies and Safety Filters
May 1, 2026Oswin So, William Sharpless, Sylvia Herbert +1Value Functions Signal Temporal Logic
SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
May 1, 2026Stavros Orfanoudakis, Pedro P. VergaraSimulation-Based Reinforcement Learning Value Functions
Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimation
May 1, 2026Haichen Hu, Jian Qian, David Simchi-LeviOffline Reinforcement Learning Markov Decision Processes
Data Deletion Can Help in Adaptive RL
Apr 30, 2026Param Budhraja, Aditya Gangrade, Alex Olshevsky +1Linear Contextual Bandits Markov Decision Processes
Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
Apr 30, 2026Shijin Gong, Kai Ye, Jin Zhu +3Large Language Model Reasoning Policy Gradient
Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations
Apr 30, 2026Zhenjie Ren, Xiaoli Wei, Xiang Yu +1Stochastic Optimal Control Q-Learning
A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations
Apr 29, 2026Timothy Flavin, Sandip SenPartially Observable Markov Decision Process Multi-Agent Reinforcement Learning
Learning to Route Electric Trucks Under Operational Uncertainty
Apr 29, 2026Stavros Orfanoudakis, Ziyan Li, Ruixiao Yang +4Electric Vehicles Vehicle Routing Problem
Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
Apr 28, 2026Clinton Enwerem, Shreya Kalyanaraman, John S. Baras +1Grasp Generation Belief Space
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
Apr 28, 2026An Nguyen, Hoang Nguyen, Phuong Le +3Electric Vehicles Constrained Reinforcement Learning
DGLight: DQN-Guided GRPO Fine-Tuning of Large Language Models for Traffic Signal Control
Apr 28, 2026Chenbo YuIterative Reinforcement Learning Group Relative Policy Optimization
Optimally Auditing Adversarial Agents
Apr 28, 2026Sanmay Das, Fang-Yi Yu, Yuang ZhangAlgorithm Auditing Optimal Policies
Distributional Reinforcement Learning via the Cramér Distance
Apr 26, 2026Vanya Aziz, Ivo Nowak, E. M. T HendrixSoft Actor-Critic Value Functions
V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think
Apr 25, 2026Bingda Tang, Yuhui Zhang, Xiaohan Wang +3Group Relative Policy Optimization Generative Models
Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
Apr 24, 2026Alex NikulkovProcess Reward Modeling Value Functions
Optimal sequential decision-making for error propagation mitigation in digital twins
Apr 24, 2026Annice Najafi, Shokoufeh MirzaeiDigital Twins Digital Twin Framework
Rate-Optimal Regret for the Safe Learning-based Control of the Constrained Linear Quadratic Regulator
Apr 24, 2026Spencer Hutchinson, Nanfei Jiang, Mahnoosh AlizadehLinear Quadratic Regulator O(T^Β)$ Simultaneous Regret
SLAM as a Stochastic Control Problem with Partial Information: Optimal Solutions and Rigorous Approximations
Apr 23, 2026Ilir Gusija, Fady Alajaji, Serdar YükselSimultaneous Localization And Mapping Partially Observable Markov Decision Process
Robustness Analysis of POMDP Policies to Observation Perturbations
Apr 23, 2026Benjamin Kraske, Qi Heng Ho, Federico Rossi +2Partially Observable Markov Decision Process Non-Stationarity
Graph Neural Network-Informed Predictive Flows for Faster Ford-Fulkerson and PAC-Learnability
Apr 23, 2026Eleanor Wiesler, Trace BaxleyGraph Neural Network Inference Importance
Interval POMDP Shielding for Imperfect-Perception Agents
Apr 22, 2026William Scarbro, Ravi MangalShielding Partially Observable Markov Decision Process
Planning in entropy-regularized Markov decision processes and games
Apr 21, 2026Jean-Bastien Grill, Omar Darwiche Domingues, Pierre Ménard +2Markov Decision Processes Entropy Regularized Reinforcement Learning
LASER: Learning Active Sensing for Continuum Field Reconstruction
Apr 21, 2026Huayu Deng, Jinghui Zhong, Xiangming Zhu +2Latent World Models Physical Dynamics
Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback
Apr 21, 2026Qiang Liu, Adrienne Kline, Ermin WeiReinforcement Learning From Human Feedback Deep Reinforcement Learning

May 8, 2026 · cs.LGJ/K move · Enter open · S save

Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RL

Shengbo Wang, Zexi Zhang

Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California

Designing model-free algorithms for distributionally robust reinforcement learning (DRRL) poses fundamental challenges. The robust Bellman operator is nonlinear in the transition kernel, which makes one-sample Bellman updates biased, while the adversarial optimization underlying robustness makes robust evaluation computationally demanding. To address these difficulties, we consider the natural small-ambiguity regime under Kullback--Leibler ambiguity sets and propose an approximate DRRL framework based on a first-order expansion of the relevant robust functional. This yields an approximate robust Bellman equation that removes the adversarial optimization while remaining first-order accurate in the ambiguity radius. To learn the fixed point of this approximate equation, we propose Mean-Variance Stochastic Approximation (MVSA), a model-free algorithm that uses only one-sample updates. This is achieved via a lifted stochastic approximation dynamics and a two-time-scale design. We then prove convergence and a central limit theorem for MVSA: its main iterate satisfies a central limit theorem at the canonical

n^{-1/2}

scale, with explicitly characterized asymptotic covariances. Finally, we validate our theoretical findings with a numerical experiment.

Central Limit Theorem Model-Free Reinforcement Learning Bellman Equation Low-Data Regimes

Open paper Ask about it PDF