Semi-Markov Decision Process

Also known as SMDP

Follow this topic Ask about this topic

Latest papers

10 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
Oct 4, 2026Bingyun Liu, Yuheng JingReinforcement Learning Sequential Decision Making
Institute of Automation Chinese Academy of Sciences
Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes
Aug 24, 2026Ege C. Kaya, Aliasghar Pourghani, Mahsa Ghasemi +2Semi-Markov Decision Process Markov Decision Processes
Elmore Family School of Electrical and Computer Engineering Purdue University
Let it Cook: Learning to Wait in Sequential Decision Making
Aug 11, 2026Christopher Watson, Arjun Krishna, Dinesh Jayaraman +1Sequential Decision Making Learning to Defer
Department of Computer and Information Science, University of Pennsylvania, USA
Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment
May 31, 2026Yuepeng Wang, Ken Kawano, Akifumi Wachi +13Reinforcement Learning Sequential Decision Making
Tokyo University of Agriculture and Technology · Institute of Science Tokyo · LY Corporation +6
Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints
May 18, 2026Jiayu Li, Enpei Zhang, Dawei Zhou +2Sequential MARL Sequential Decision Making
Stern School of Business · New York University · Department of Computer Science +2
A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs
May 6, 2026Erel Shtossel, Alicia Vidler, Uri Shaham +1Non-Stationary RL Reinforcement Learning
Bar Ilan University Ramat Gan, Israel
Learning to Route Electric Trucks Under Operational Uncertainty
Apr 29, 2026Stavros Orfanoudakis, Ziyan Li, Ruixiao Yang +4Sequential Decision Making Electric Vehicle Charging
Delft University of Technology, Intelligent Electrical Power Grids, Delft, The Netherlands · Massachusetts Institute of Technology, Center of Transportation and Logistics, Cambridge, MA, United States of America · Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, Cambridge, MA, United States of America
Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL
Mar 24, 2026Igor JankowskiSemi-Markov Decision Process Reinforcement Learning
Joint MDPs and Reinforcement Learning in Coupled-Dynamics Environments
Mar 6, 2026Ege C. Kaya, Mahsa Ghasemi, Abolfazl HashemiSemi-Markov Decision Process Markov Decision Processes
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
Multimodal Remote Inference
Aug 11, 2025Keyuan Zhang, Yin Sun, Bo JiSemi-Markov Decision Process Efficient Multimodal Inference
Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University

Oct 4, 2026J/K move · Enter open · S save

On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning

Bingyun Liu, Yuheng Jing

Institute of Automation Chinese Academy of Sciences

Reinforcement Learning Sequential Decision Making Hierarchical RL Semi-Markov Decision Process

Open paper AskPDF arXiv

Hierarchical reinforcement learning uses temporally extended subtasks for exploration, yet committing to their execution can restrict both deployment and policy learning. We identify and separate the resulting execution and policy suboptimality. Task and execution trees distinguish reward objectives from policy choices and decision interruption. A Unified Value Function for HRL and a four-stage Generalized Hierarchical Bellman Equation then support a common analysis of both losses. Under bounded rewards and uniform termination, we establish hierarchical policy and execution improvement results. With the remaining node policies fixed, task-subtree compatibility and node-policy optimality under the original execution mode establish when Markov execution is optimal. The resulting decomposition leads to independent execution choices for behavior, targets, and deployment. We instantiate this principle through execution improvement and one-stage or two-stage policy improvement at arbitrary hierarchy depth. Option-based and goal-conditioned experiments demonstrate complementary gains from changing execution and changing the learning target. Controlled stochastic environments show how these gains depend on stochastic transition strength and spatial structure. This framework makes execution design an explicit component of hierarchical policy optimization.