Rule-based and reinforcement-learning policies for resource allocation in a pediatric echo lab: a simulation study
Authors: Bozhi Sun, Seda Tierney, Jeffrey A. Feinstein, Frederick Damen, Alison L. Marsden, Daniele E. Schiavazzi
Organizations: Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN, USA. · Department of Pediatrics (Cardiology), Stanford University, CA 94305, US. · Department of Bioengineering, Stanford University, CA 94305, US.
Scheduling echocardiographic examinations in a hospital setting presents significant challenges due to non-deterministic factors such as patient no-shows, variable arrival times, and heterogeneous exam durations, as well as asymmetric resource constraints between fetal and non-fetal patient streams. Fetal cases involve specialized ultrasound examinations performed during pregnancy, whereas non-fetal cases correspond to general diagnostic echocardiography services. Using one week (184 patients) of operational data from the Echo Laboratory at Stanford University's Lucile Packard Children's Hospital, we estimate patient no-show probabilities and derive empirical distributions of arrival times and exam durations, and use them to build a discrete-event stochastic simulation model using SimPy, exposed through an open-source Gymnasium interface (https://github.com/desResLab/echo_rl). As a baseline for policy optimization, we systematically compare two scheduling paradigms: on-the-fly allocation, in which patients are served according to their arrival time without prior reservations, and reservation-based allocation, in which varying proportions of resources are preassigned. We consider a hospital with seven ultrasound rooms (one equipped for both fetal and non-fetal exams, six for non-fetal exams only) and six sonographers (four qualified for both exam types, two for non-fetal exams only). In this setting, on-the-fly allocation markedly reduces the average patient waiting time compared with every reservation-based configuration tested. Building on this foundation, we train a Dueling Double Deep Q-Network (DQN) with prioritized experience replay to approximate an optimal dynamic on-the-fly allocation policy. The learned RL policy is comparable to, but does not outperform, the best on-the-fly rule-based policy.
Figures & tables
QTY
TYPE
QTY
TYPE
patient ID
int
echo time in
time
subspecialty
string
echo time out
time
chemotherapy
yes/no
echo duration
time
fetal
yes/no
met echo cycle time
yes/no
research
yes/no
echo start late > 10 min
yes/no
expected check-in/arrival time
time
sonographer echo room number
int
Table 1: List of features in the dataset.
Figure 1: Echo visit schedules for both fetal and non-fetal patients.
Figure 2: Fitted distribution of arrival delays (left) and example of arrival dynamics (right).
Figure 3: Fitted distribution for the echo test duration and the number of echo resources.
Breaks remaining
Outcome
Probability
2
Take both breaks together (30 min)
20%
2
Take one break (15 min)
30%
2
No break taken
50%
1
Take the remaining break (15 min)
30%
1
No break taken
70%
0
No break taken (none remaining)
—
Table 2: Sonographer break-taking probabilities after completing a test, conditional on the number of breaks remaining that day.
Figure 4: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 1.
Figure 5: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing predefined policy 2.
Figure 6: Average results in terms of echo resources and waiting patients (mean count and standard deviation) over 365 independent replications of a representative day, for the stochastic hospital process enforcing policy 3.
Figure 7: Average results in terms of echo resources and waiting patients (average counts and standard deviation) over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 4.
Case
α
β
Case
α
β
Case 1
0%
0%
Case 6
100%
0%
Case 2
0%
25%
Case 7
100%
25%
Case 3
0%
50%
Case 8
100%
50%
Case 4
0%
75%
Case 9
100%
75%
Case 5
0%
100%
Case 10
100%
100%
Table 3: List of analyzed parameters for policy 5.
Figure 8: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=100% . Additional results are reported in the Appendix A.2.1 .
Figure 9: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day, for the stochastic hospital process enforcing policy 6 with α=100% and β=100% . Additional results are reported in Section A.2.2 .
Figure 10: Violin plot of average patient waiting times.
Figure 11: Violin plot of average sonographer quotas.
Figure 15Figure 16
Figure 16: State-by-state comparison between policy 2 and the RL policy. (a) At 08:00 AM, policy 2 accommodates non-fetal on-time patient ( An,t ) and fetal early patient ( Af,e ). However, the RL policy only accommodates non-fetal on-time patient ( An,t ). (b) At 11:19 AM, policy 2 accommodates non-fetal on-time patient ( An,t ) while the RL policy accommodates the fetal on-time patient ( Af,t ). (c) At 10:14 AM, policy 2 accommodates both the non-fetal late patient ( An,l ) and the non-fetal early patient ( An,e ). In contrast, the RL policy accommodates the non-fetal late patient ( An,l ) only. (d) At 10:32 AM, policy 2 accommodates the non-fetal late patient ( An,l ) and the non-fetal on-time patient ( An,t ); while the RL policy only accommodates the non-fetal on-time patient ( An,t ). At 12:52 PM, policy 2 accommodates both non-fetal patients that arrived late ( An,l ), while the RL policy only accommodates one.
Figure 17: Fetal versus non-fetal mean weighted waiting time under each policy, averaged over 365 independent replications of a representative day. The dashed line marks equal waiting times.
Figure 18: Sensitivity of waiting times to varying no-show rates for both fetal and non-fetal patients.
Figure 19: Sensitivity of W to parameters θ and k for both policy 2 and the RL policy.
Figure 20: Sensitivity of patient waiting times to varying standard deviations for both fetal and non-fetal populations.
Figure 21: Comparison of average daily waiting time produced by various policies under abundant echo resources.
Figure 23Figure 24
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 26: Pie charts for patient’s absence rate.
Metric
Fetal
Non-Fetal
Mean
6.72
12.81
Median
5.00
10.50
Standard Deviation
8.38
16.89
Minimum
-5.00
-45.00
Maximum
25.00
75.00
Q1 (25th percentile)
0.00
1.25
Appendix
Table 4: Descriptive statistics of patient delay times (minutes).
Test
Statistic
p-value
Interpretation (at α=0.05 )
Mann–Whitney U
U=404.50
0.0765
Not significant (no median difference)
Kolmogorov–Smirnov
D=0.2401
0.3355
Not significant (shapes similar)
Levene’s Test
W=2.4709
0.1200
Not significant (variances similar)
Cohen’s d
d=−0.3941
—
Small-to-medium effect (non-fetal larger delays)
Appendix
Table 5: Non-parametric test results comparing fetal vs. non-fetal delay distributions and effect size.
Figure 27: Distribution of patients’ arrival delay (difference between actual and scheduled arrival time)
Metric
Fetal
Non-Fetal
Mean
55.78
48.44
Median
57.50
45.00
Standard Deviation
23.01
18.06
Minimum
28.00
15.00
Maximum
120.00
99.00
Q1 (25th percentile)
41.25
36.00
Appendix
Table 6: Descriptive statistics of echo duration times (minutes).
Table 7: Non-parametric test results comparing fetal vs. non-fetal echo duration distributions and effect size.
Figure 28: Distribution of patients’ echo duration.
Figure 29: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=0% and β=0% .
Figure 30: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=0% and β=25%
Figure 31: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=0% and β=50%
Figure 32: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=0% and β=75%
Figure 33: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=0% and β=100%
Figure 34: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=0%
Figure 35: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=25%
Figure 36: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=50%
Figure 37: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=75%
Figure 38: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 5 with α=100% and β=100%
Figure 39: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=0% and β=0% .
Figure 40: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=0% and β=25%
Figure 41: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=0% and β=50%
Figure 42: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=0% and β=75%
Figure 43: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=0% and β=100%
Figure 44: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=100% and β=0%
Figure 45: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=100% and β=25%
Figure 46: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=100% and β=50%
Figure 47: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=100% and β=75%
Figure 48: Average results in terms of echo resources and waiting patients over 365 independent replications of a representative day for the stochastic hospital process enforcing policy 6 with α=100% and β=100%
Patient no-shows disrupt outpatient clinic operations, reduce productivity, and may delay necessary care. Clinics often adopt overbooking or double-booking to mitigate these effects. However, poorly calibrated policies can increase congestion and waiting times. Most existing methods rely on fixed heuristics and fail to adapt to real-time scheduling conditions or patient-specific no-show risk. To address these limitations, we propose an adaptive outpatient double-booking framework that integrates individualized no-show prediction with multi-objective reinforcement learning. The scheduling problem is formulated as a Markov decision process, and patient-level no-show probabilities estimated by a Multi-Head Attention Soft Random Forest model are incorporated in the reinforcement learning state. We develop a Multi-Policy Proximal Policy Optimization method equipped with a Multi-Policy Co-Evolution Mechanism. Under this mechanism, we propose a novel τ rule based on Kullback-Leibler divergence that enables selective knowledge transfer among behaviorally similar policies, improving convergence and expanding the diversity of trade-offs. In addition, SHapley Additive exPlanations is used to interpret both the predicted no-show risk and the agent's scheduling decisions. The proposed framework determines when to single-book, double-book, or reject appointment requests, providing a dynamic and data-driven alternative to conventional outpatient scheduling policies.
Ninda Nurseha Amalina, Heungjo An
Department of Industrial Engineering, Kumoh National Institute of Technology, Gumi City, South Korea
Many social services assign scarce resources, such as housing assistance or hospital interventions, to people who arrive one at a time: each arrival must receive a decision immediately, and the long-run usage of every resource must stay within its capacity. We study how to learn such an assignment policy from logged observational data. The standard pipeline is decision-blind: fit one outcome model per arm by regression, price each capacitated resource from the fitted models, and assign each arrival the arm whose predicted outcome minus price is largest. We instead train the outcome models end-to-end, differentiating an off-policy estimate of the deployed policy's value through the dual prices themselves. We study two formulations: an exact nonconvex one, and a convex relaxation whose optimum always satisfies the capacity constraints in expectation and which is suboptimal by at most a term linear in the smoothing temperature and logarithmic in the number of arms. Every method is evaluated in a queueing simulation with resources replenished at their capacity rates. Across six datasets, the two end-to-end variants take the top slots on a deployment-adjusted value index at every delay cost, including zero; when capacities are binding, decision-blind baselines frequently violate them and incur much longer queueing delays. On the largest dataset, a hospital cohort of seventy thousand patients, end-to-end training also achieves significantly higher policy value, a margin that survives a capacity-matched neural baseline. Flexible decision-blind regression remains the stronger pure predictor where ground truth is measurable; end-to-end training is best suited to settings where resources are genuinely scarce and feasibility matters.
We study online task allocation for multi-robot, multi-queue systems with asymmetric stochastic arrivals and switching delays. We formulate the problem in discrete time: each location can host at most one robot per slot, servicing a task consumes one slot, switching between locations incurs a one-slot travel delay, and arrivals at locations are independent Bernoulli processes with heterogeneous rates. Building on our previous structural result that optimal policies are of exhaustive type, we formulate a discounted-cost Markov decision process and develop an exhaustive-assignment actor-critic policy architecture that enforces exhaustive service by construction and learns only the next-queue allocation for idle robots. Unlike the exhaustive-serve-longest (ESL) queue rule, whose optimality is known only under symmetry, the proposed policy adapts to asymmetry in arrival rates. Across different server-location ratios, loads, and asymmetric arrival profiles, the proposed policy consistently achieves lower discounted holding cost and smaller mean queue length than the ESL baseline, while remaining near-optimal on instances where an optimal benchmark is available. These results show that structure-aware actor-critic methods provide an effective approach for real-time multi-robot scheduling.
Mohammad Merati, H. M. Sabbir Ahmad, Wenchao Li +1
Divisions of Systems Engineering, Boston University, 8 St Mary’s St, Boston, MA 02215, United States. · Department of Electrical and Computer Engineering, Boston University, 8 St Mary’s St, Boston, MA 02215, United States.