cs.LGAug 21, 2026

Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing

Authors: Huiling Meng, Ningyuan Chen, Xuefeng Gao

Organizations: Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Rotman School of Management, University of Toronto, Toronto, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study reinforcement learning (RL) in Continuous-Time Jump Markov Decision Processes (CTJMDPs) featuring general discrete state spaces (which need not possess a vector space structure) and continuous/discrete action spaces. The setup covers many well-known applications in operations such as multi-product dynamic pricing with capacitated resources (Gallego and van Ryzin 1997). To model the exploration-exploitation tradeoff, we formulate an entropy-regularized continuous-time control problem with stochastic policies. Recent continuous-time RL techniques such as qq-learning for controlled diffusions in (Jia and Zhou 2023) focus on continuous state spaces Rd\mathbb{R}^d and rely heavily on semimartingale theory in Rd\mathbb{R}^d for their theoretical analysis. Consequently, their methods cannot be directly applied to CTJMDPs with general discrete state spaces, which may lack the algebraic addition and subtraction structures inherent to Euclidean spaces. To bridge this gap, we establish the theoretical foundations of qq-learning for CTJMDPs and develop model-free qq-learning algorithms. Compared to naïve time discretization and approximating CTJMDPs using discrete-time MDPs, our approach has several conceptual and empirical benefits. Numerical experiments in network dynamic pricing (Gallego and van Ryzin 1997) show that our proposed RL algorithm reliably learns near-optimal policies and consistently outperforms standard benchmark methods, demonstrating superior solution quality and effective scalability to large-scale network instances.

Figures & tables

 S∪TJ={0=s0<s1<⋯<sM=T},\mathbb{S}\cup\mathcal{T}_{J}=\{0=s_{0}<s_{1}<\cdots<s_{M}=T\}, 
Algorithm 1 Episodic qq -Learning Algorithm
Itinerary jjλj0\lambda_{j}^{0}ϵj0\epsilon_{j}^{0}pj0p_{j}^{0}
1304.020
2154.030
3504.045
Table 1: Itinerary and demand parameters for the example in Section 5.1
Figure 1: Average revenue of Algorithm 1 over episodes for the example in Section 5.1
MethodAverage revenue99%-CI ( ±\pm )Gap from DP
DP709.883–0.00%
Algorithm 1692.9510.8112.39%
FP636.5192.59910.33%
FP-BL582.7632.39217.90%
Table 2: Numerical results for the example in Section 5.1
Figure 2: Airline network for the example in Section 5.2
Itinerary jjO–D pairPathϵj0\epsilon_{j}^{0}pj0p_{j}^{0}
111 – 2211 – 221.0220
211 – 3311 – 331.2220
311 – 4411 – 22 – 442.0400
411 – 5511 – 33 – 551.0250
511 – 6611 – 660.8200
622 – 3322 – 331.0230
Table 3: Itinerary and demand parameters for example in Section 5.2
Figure 3: Average revenue of Algorithm 1 over episodes for the example in Section 5.2
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Average reward of Algorithm 1 over episodes for queue control
More in this topic

Markov Decision Processes

Markov decision processes model sequential choices under uncertainty, linking actions, state transitions, and rewards to planning and reinforcement learning.

See all 157 papers

Explore similar work

CardsList

  1. Policy Gradient for Continuous-Time Robust Markov Decision Processes

    Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaRobust Markov Decision ProcessesMarkov Decision Processes

    Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR) · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR) · College of Computing and Data Science, Nanyang Technological University

  2. Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs

    Sep 19, 2026Kenny Guo, Valentio Iverson, Sahan Wijetunga +1Markov Decision ProcessesContinuous-Time RL

    Department of Economics, Yale University, New Haven, Connecticut, USA · Department of Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA · Department of Mathematics, University of California, Los Angeles, Los Angeles, California, USA

  3. Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence

    Mar 3, 2026Shengbo WangReinforcement LearningQ-Learning

    Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California