Semi Markov Decision Process

Semi-Markov Decision Processes (SMDPs) extend Markov Decision Processes by allowing for variable-length time steps between decisions, reflecting the realities of many real-world systems where events don't occur at fixed intervals. Current research focuses on developing efficient algorithms, such as asynchronous stochastic approximation and deep reinforcement learning methods (including Q-learning variants), to solve SMDPs, particularly in complex scenarios with continuous action spaces or hierarchical structures (like option-based approaches). These advancements are improving the modeling and optimization of diverse applications, including dynamic vehicle routing, resource allocation (e.g., caching and medical evacuation), and other systems with stochastic and time-varying dynamics. The resulting improved decision-making capabilities have significant implications across various fields.

Papers

October 17, 2024

Contextual Bandits with Arm Request Costs and Delays
Lai Wei, Ambuj Tewari, Michael A. Cianfrocco
Optimal Policy Contextual Bandit Contextual Bandit Problem Bellman Equation Significant Delay Change Arm Selection Semi Markov Decision Process

October 6, 2024

Semi-Markovian Planning to Coordinate Aerial and Maritime Medical Evacuation Platforms
Mahdi Al-Husseini, Kyle H. Wray, Mykel J. Kochenderfer
Swap Distance Minimization Semi Markov Decision Process

September 5, 2024

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning
Huizhen Yu, Yi Wan, Richard S. Sutton
Reinforcement Learning Average Reward Reinforcement Learning Asynchronous Stochastic Semi Markov Decision Process

June 21, 2024

A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning
Gianluca Drappo, Alberto Maria Metelli, Marcello Restelli
High Efficiency Practical Algorithm Hierarchical Reinforcement Learning Finite Horizon High Level Semi Markov Decision Process Policy Selection

February 8, 2024

Attention-Enhanced Prioritized Proximal Policy Optimization for Adaptive Edge Caching
Farnaz Niknia, Ping Wang, Zixu Wang, Aakash Agarwal, Adib S. Rezaei
Deep Reinforcement Learning Transfer Learning Markov Decision Process Caching Strategy Edge Caching Redundant Computation Semi Markov Decision Process Juniper Router

July 13, 2023

Deep reinforcement learning for the dynamic vehicle dispatching problem: An event-based approach
Edyvalberty Alenquer Cordeiro, Anselmo Ramalho Pitombeira-Neto
Deep Reinforcement Learning Event Based Continuous Variable Semi Markov Decision Process Heuristic Policy Epoch Based

May 14, 2023

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees
Marcus Hoerger, Hanna Kurniawati, Dirk Kroese, Nan Ye
Observable Markov Decision Process Automatic Solver Partially Observable Markov Decision Process Decision Tree Policy Continuous Action Semi Markov Decision Process Entropy Search

May 10, 2023

An Option-Dependent Analysis of Regret Minimization Algorithms in Finite-Horizon Semi-Markov Decision Processes
Gianluca Drappo, Alberto Maria Metelli, Marcello Restelli
Hierarchical Reinforcement Learning Finite Horizon Real World Reinforcement Learning Planning Horizon Semi Markov Decision Process Path Dependent Option

July 16, 2022

ChronosPerseus: Randomized Point-based Value Iteration with Importance Sampling for POSMDPs
Richard Kohar, François Rivest, Alain Gosselin
Markov Decision Process Importance Sampling Value Iteration POMDP Policy Semi Markov Decision Process

December 13, 2021

Solving the non-preemptive two queue polling model with generally distributed service and switch-over durations and Poisson arrivals as a Semi-Markov Decision Process
Dylan Solms
Service Provider Poisson Process Discrete Event Continuous Time Markov Decision Process Semi Markov Decision Process Semi Markov

Semi Markov Decision Process

Papers

Contextual Bandits with Arm Request Costs and Delays

Semi-Markovian Planning to Coordinate Aerial and Maritime Medical Evacuation Platforms

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning

A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning

Attention-Enhanced Prioritized Proximal Policy Optimization for Adaptive Edge Caching

Deep reinforcement learning for the dynamic vehicle dispatching problem: An event-based approach

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees

An Option-Dependent Analysis of Regret Minimization Algorithms in Finite-Horizon Semi-Markov Decision Processes

ChronosPerseus: Randomized Point-based Value Iteration with Importance Sampling for POSMDPs

Solving the non-preemptive two queue polling model with generally distributed service and switch-over durations and Poisson arrivals as a Semi-Markov Decision Process