Rule-based and reinforcement-learning policies for resource allocation in a pediatric echo lab: a simulation study

Bozhi Sun; Seda Tierney; Jeffrey A. Feinstein; Frederick Damen; Alison L. Marsden; Daniele E. Schiavazzi

cs.LGMay 17, 2025

Rule-based and reinforcement-learning policies for resource allocation in a pediatric echo lab: a simulation study

Authors: Bozhi Sun, Seda Tierney, Jeffrey A. Feinstein, Frederick Damen, Alison L. Marsden, Daniele E. Schiavazzi

Organizations: Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN, USA. · Department of Pediatrics (Cardiology), Stanford University, CA 94305, US. · Department of Bioengineering, Stanford University, CA 94305, US.

Abstract

Scheduling echocardiographic examinations in a hospital setting presents significant challenges due to non-deterministic factors such as patient no-shows, variable arrival times, and heterogeneous exam durations, as well as asymmetric resource constraints between fetal and non-fetal patient streams. Fetal cases involve specialized ultrasound examinations performed during pregnancy, whereas non-fetal cases correspond to general diagnostic echocardiography services. Using one week (184 patients) of operational data from the Echo Laboratory at Stanford University's Lucile Packard Children's Hospital, we estimate patient no-show probabilities and derive empirical distributions of arrival times and exam durations, and use them to build a discrete-event stochastic simulation model using SimPy, exposed through an open-source Gymnasium interface (https://github.com/desResLab/echo_rl). As a baseline for policy optimization, we systematically compare two scheduling paradigms: on-the-fly allocation, in which patients are served according to their arrival time without prior reservations, and reservation-based allocation, in which varying proportions of resources are preassigned. We consider a hospital with seven ultrasound rooms (one equipped for both fetal and non-fetal exams, six for non-fetal exams only) and six sonographers (four qualified for both exam types, two for non-fetal exams only). In this setting, on-the-fly allocation markedly reduces the average patient waiting time compared with every reservation-based configuration tested. Building on this foundation, we train a Dueling Double Deep Q-Network (DQN) with prioritized experience replay to approximate an optimal dynamic on-the-fly allocation policy. The learned RL policy is comparable to, but does not outperform, the best on-the-fly rule-based policy.

Figures & tables

QTY	TYPE	QTY	TYPE
patient ID	int	echo time in	time
subspecialty	string	echo time out	time
chemotherapy	yes/no	echo duration	time
fetal	yes/no	met echo cycle time	yes/no
research	yes/no	echo start late $>$ 10 min	yes/no
expected check-in/arrival time	time	sonographer echo room number	int

Table 1: List of features in the dataset.

Panel of Figure 1 — Figure 1: Echo visit schedules for both fetal and non-fetal patients.

Panel of Figure 2 — Figure 2: Fitted distribution of arrival delays (left) and example of arrival dynamics (right).

Panel of Figure 3 — Figure 3: Fitted distribution for the echo test duration and the number of echo resources.

Breaks remaining	Outcome	Probability
2	Take both breaks together (30 min)	20%
2	Take one break (15 min)	30%
2	No break taken	50%
1	Take the remaining break (15 min)	30%
1	No break taken	70%
0	No break taken (none remaining)	—

Table 2: Sonographer break-taking probabilities after completing a test, conditional on the number of breaks remaining that day.

Figure 4: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 1.

Figure 5: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing predefined policy 2.

Figure 6: Average results in terms of echo resources and waiting patients (mean count and standard deviation) over 365 independent replications of a representative day, for the stochastic hospital process enforcing policy 3.

Figure 7: Average results in terms of echo resources and waiting patients (average counts and standard deviation) over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 4.

Case	$\boldsymbol{\alpha}$	$\boldsymbol{\beta}$	Case	$\boldsymbol{\alpha}$	$\boldsymbol{\beta}$
Case 1	0%	0%	Case 6	100%	0%
Case 2	0%	25%	Case 7	100%	25%
Case 3	0%	50%	Case 8	100%	50%
Case 4	0%	75%	Case 9	100%	75%
Case 5	0%	100%	Case 10	100%	100%

Table 3: List of analyzed parameters for policy 5.

Figure 8: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=100\%

. Additional results are reported in the Appendix A.2.1 .

Figure 9: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day, for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=100\%

. Additional results are reported in Section A.2.2 .

Figure 10: Violin plot of average patient waiting times.

Figure 11: Violin plot of average sonographer quotas.

Figure 17: Fetal versus non-fetal mean weighted waiting time under each policy, averaged over 365 independent replications of a representative day. The dashed line marks equal waiting times.

Panel of Figure 18 — Figure 18: Sensitivity of waiting times to varying no-show rates for both fetal and non-fetal patients.

Panel of Figure 19 — Figure 19: Sensitivity of $W$ to parameters $\theta$ and $k$ for both policy 2 and the RL policy.

Panel of Figure 20 — Figure 20: Sensitivity of patient waiting times to varying standard deviations for both fetal and non-fetal populations.

Figure 21: Comparison of average daily waiting time produced by various policies under abundant echo resources.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 26: Pie charts for patient’s absence rate.

Metric	Fetal	Non-Fetal
Mean	6.72	12.81
Median	5.00	10.50
Standard Deviation	8.38	16.89
Minimum	-5.00	-45.00
Maximum	25.00	75.00
Q1 (25th percentile)	0.00	1.25

Appendix

Table 4: Descriptive statistics of patient delay times (minutes).

Test	Statistic	p-value	Interpretation (at $\alpha=0.05$ )
Mann–Whitney U	$U=404.50$	0.0765	Not significant (no median difference)
Kolmogorov–Smirnov	$D=0.2401$	0.3355	Not significant (shapes similar)
Levene’s Test	$W=2.4709$	0.1200	Not significant (variances similar)
Cohen’s $d$	$d=-0.3941$	—	Small-to-medium effect (non-fetal larger delays)

Appendix

Table 5: Non-parametric test results comparing fetal vs. non-fetal delay distributions and effect size.

Figure 27: Distribution of patients’ arrival delay (difference between actual and scheduled arrival time)

Metric	Fetal	Non-Fetal
Mean	55.78	48.44
Median	57.50	45.00
Standard Deviation	23.01	18.06
Minimum	28.00	15.00
Maximum	120.00	99.00
Q1 (25th percentile)	41.25	36.00

Appendix

Table 6: Descriptive statistics of echo duration times (minutes).

Test	Statistic	p-value	Interpretation (at $\alpha=0.05$ )
Mann–Whitney U	$U=586.50$	0.1927	Not significant (no median difference)
Kolmogorov–Smirnov	$D=0.2593$	0.3012	Not significant (shapes similar)
Levene’s Test	$W=0.2922$	0.5905	Not significant (variances similar)
Cohen’s $d$	$d=0.3784$	—	Small-to-medium effect (fetal durations slightly longer)

Appendix

Table 7: Non-parametric test results comparing fetal vs. non-fetal echo duration distributions and effect size.

Figure 28: Distribution of patients’ echo duration.

Figure 29: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=0\%

and

\beta=0\%

Figure 30: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=0\%

and

\beta=25\%

Figure 31: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=0\%

and

\beta=50\%

Figure 32: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=0\%

and

\beta=75\%

Figure 33: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=0\%

and

\beta=100\%

Figure 34: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=0\%

Figure 35: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=25\%

Figure 36: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=50\%

Figure 37: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=75\%

Figure 38: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with

\alpha=100\%

and

\beta=100\%

Figure 39: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=0\%

and

\beta=0\%

Figure 40: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=0\%

and

\beta=25\%

Figure 41: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=0\%

and

\beta=50\%

Figure 42: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=0\%

and

\beta=75\%

Figure 43: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=0\%

and

\beta=100\%

Figure 44: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=0\%

Figure 45: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=25\%

Figure 46: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=50\%

Figure 47: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=75\%

Figure 48: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with

\alpha=100\%

and

\beta=100\%

Hyperparameter	Value
Network architecture	Dueling DQN; shared trunk $13\to 128\to 128$ ; value stream $128\to 64\to 1$ ; advantage stream $128\to 64\to 729$
Normalization / activation	Layer Normalization; ReLU
Weight initialization	Kaiming (He et al.)
Optimizer	Adam (default PyTorch betas $(0.9,0.999)$ , $\epsilon=10^{-8}$ )
Learning rate	Initial $0.001$ ; $\times 0.95$ after 500 consecutive steps without loss improvement; floor $0.00001$
Discount factor $\gamma$	$0.99$

Appendix

Table 8: Hyperparameters used to train the RL policy.

Explore similar work

Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning
Mar 7, 2026Ninda Nurseha Amalina, Heungjo AnHealthcare Multi-Objective Reinforcement Learning
Department of Industrial Engineering, Kumoh National Institute of Technology, Gumi City, South Korea
Differentiating Through Dual Prices: End-to-End Policy Learning Under Capacity Constraints
Aug 5, 2026Mohammadsaeed Haghi, Mahdi Salmani, Nima KelidariConstrained Optimization Differentiable Optimization
University of Southern California
Multi-Robot Multi-Queue Control via Exhaustive Assignment Actor-Critic Learning
Apr 4, 2026Mohammad Merati, H. M. Sabbir Ahmad, Wenchao Li +1Multi-Robot Task Allocation Robot Policy Learning
Divisions of Systems Engineering, Boston University, 8 St Mary’s St, Boston, MA 02215, United States. · Department of Electrical and Computer Engineering, Boston University, 8 St Mary’s St, Boston, MA 02215, United States.

Mar 7, 2026J/K move · Enter open · S save

Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning

Ninda Nurseha Amalina, Heungjo An

Department of Industrial Engineering, Kumoh National Institute of Technology, Gumi City, South Korea

Healthcare Multi-Objective Reinforcement Learning Proximal Policy Optimization

Open paper AskPDF arXiv

Patient no-shows disrupt outpatient clinic operations, reduce productivity, and may delay necessary care. Clinics often adopt overbooking or double-booking to mitigate these effects. However, poorly calibrated policies can increase congestion and waiting times. Most existing methods rely on fixed heuristics and fail to adapt to real-time scheduling conditions or patient-specific no-show risk. To address these limitations, we propose an adaptive outpatient double-booking framework that integrates individualized no-show prediction with multi-objective reinforcement learning. The scheduling problem is formulated as a Markov decision process, and patient-level no-show probabilities estimated by a Multi-Head Attention Soft Random Forest model are incorporated in the reinforcement learning state. We develop a Multi-Policy Proximal Policy Optimization method equipped with a Multi-Policy Co-Evolution Mechanism. Under this mechanism, we propose a novel τ rule based on Kullback-Leibler divergence that enables selective knowledge transfer among behaviorally similar policies, improving convergence and expanding the diversity of trade-offs. In addition, SHapley Additive exPlanations is used to interpret both the predicted no-show risk and the agent's scheduling decisions. The proposed framework determines when to single-book, double-book, or reject appointment requests, providing a dynamic and data-driven alternative to conventional outpatient scheduling policies.

Abstract

Explore similar work

Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning

Differentiating Through Dual Prices: End-to-End Policy Learning Under Capacity Constraints

Multi-Robot Multi-Queue Control via Exhaustive Assignment Actor-Critic Learning