Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning

Scheduling echocardiographic examinations in a hospital setting presents significant challenges due to non-deterministic factors such as patient no-shows, variable arrival times, and heterogeneous exam durations, as well as asymmetric resource constraints between fetal and non-fetal patient streams. Fetal cases involve specialized ultrasound examinations performed during pregnancy, whereas non-fetal cases correspond to general diagnostic echocardiography services. Using one week (184 patients) of operational data from the Echo Laboratory at Stanford University's Lucile Packard Children's Hospital, we estimate patient no-show probabilities and derive empirical distributions of arrival times and exam durations, and use them to build a discrete-event stochastic simulation model using SimPy, exposed through an open-source Gymnasium interface (https://github.com/desResLab/echo_rl). As a baseline for policy optimization, we systematically compare two scheduling paradigms: on-the-fly allocation, in which patients are served according to their arrival time without prior reservations, and reservation-based allocation, in which varying proportions of resources are preassigned. We consider a hospital with seven ultrasound rooms (one equipped for both fetal and non-fetal exams, six for non-fetal exams only) and six sonographers (four qualified for both exam types, two for non-fetal exams only). In this setting, on-the-fly allocation markedly reduces the average patient waiting time compared with every reservation-based configuration tested. Building on this foundation, we train a Dueling Double Deep Q-Network (DQN) with prioritized experience replay to approximate an optimal dynamic on-the-fly allocation policy. The learned RL policy is comparable to, but does not outperform, the best on-the-fly rule-based policy.

Dispatch in three-sided marketplaces provides a natural setting for reinforcement learning from world feedback: decisions are evaluated by delayed operational outcomes such as delivery speed, courier utilization, and merchant congestion. We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Rather than replacing the combinatorial assignment optimizer, a store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. We train a shared value function using centralized offline data and decentralized store-level execution, with Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. In a production switchback experiment, the offline-trained policy increases batching and reduces courier-side time costs without degrading customer-facing delivery quality. Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online.

Efficient patient flow coordination across autonomous hospital departments is critical for mitigating overcrowding and balancing resource utilization. While classical queueing theory, specifically open Baskett--Chandy--Muntz--Palacios (BCMP) networks, provides an interpretable mathematical topology for healthcare operations, analytical models rely on stationary assumptions and fixed routing matrices that degrade under state-dependent real-world dynamics. Conversely, centralized reinforcement learning approaches struggle to accommodate the decentralized structure of hospital governance, where individual clinical departments function with localized observations, heterogeneous resources, and divergent operational objectives. In this paper, we present a Multi-Agent Systems (MAS) framework titled Physics-Informed Multi-Agent Coordination, which embeds empirically calibrated BCMP queueing topologies as physical priors within a decentralized multi-agent reinforcement learning architecture. Formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) under coupled resource constraints, our method enables autonomous departmental agents to cooperatively negotiate patient routing and dynamic service scaling. To mitigate environmental non-stationarity without inducing excessive communication overhead, agents exchange localized action fingerprints along network edges and optimize a spatially decomposed reward structure. Empirical evaluations driven by real-world MIMIC-IV patient trajectories indicate that this cooperative multi-agent approach substantially reduces cumulative system delay compared to static Markovian approximations, heuristic dispatching, and independent multi-agent baselines, while maintaining clinical safety constraints.