Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data
Authors: Fateme Golivand Darvishvand, Saurabh Mathur, Nikhilesh Prabhakar, Michael A. Skinner, Ameet Soni, Neel Shah, Ethan L Sanford, Lakshmi Raman, Sriraam Natarajan
Organizations: University of Texas at Dallas · Technische Universität Darmstadt · Swarthmore College · Washington University in St. Louis · UT Southwestern Medical Center
Managing children on ECMO (Extracorporeal Membrane Oxygenation) requires balancing short-term physiological stability with the long-term goal of preventing neurological injury. Standard reinforcement learning can propose risky, out-of-distribution actions, while imitation learning is limited to reproducing clinician behavior. We introduce Support-Constrained Fitted Q-Iteration (FQI-SC), an offline reinforcement learning framework that cautiously improves upon expert behavior by restricting optimization to empirically supported state-action pairs and defaulting to the clinician baseline in low-data regions. On 78 pediatric trajectories totaling over 6,000 hours of clinical data, we evaluate our framework using Fitted Q-Evaluation (FQE) and a behavioral disagreement. Additionally, we formalize the performance-safety trade-off by introducing a unified Balance Score (S(π)). Our empirical evaluation shows that support-constrained policy improves the Balance Score by 38% relative to FQI, improves estimated value by 1.28 FQE units over imitation learning, and reduces disagreement with clinician actions by 71% relative to FQI.
Figures & tables
Figure 1: Balancing Safety and Performance in Pediatric ECMO. (a) We frame management of children on ECMO life support as an offline policy learning problem using logged clinician trajectories. (b) FQI attains high estimated value but deviates substantially from clinician behavior, whereas IL remains close to clinicians but has lower estimated value. FQI-SC seeks an intermediate operating point by restricting improvement to empirically supported actions.
Variable
Target / Rule
Unit
Weight
BP
(0–1)
age-norm.
2.0
HR
(0–1)
age-norm.
1.5
SpO 2
92–100
%
1.5
PaO 2
80–150
mmHg
1.5
PaCO 2
35–45
mmHg
2.0
Arterial pH
7.30–7.45
–
1.5
Table 1: Clinician-defined target ranges and importance weights used in the stability component of the reward. Weights are assigned per clinical category.
Figure 2: Per-head and all-head trade-off between estimated value and behavioral conservatism. Each point shows a policy’s mean FQE and disagreement rate, with error bars denoting variability across folds. Plain FQI occupies the high-value but high-disagreement regime, imitation learning occupies the low-disagreement but lower-value regime, and support-constrained FQI (FQI-SC) lies between these extremes. Different marker shapes indicate the support threshold N∧∈{5,10,15} , and the shaded upper-left region highlights the desirable operating regime of higher FQE and lower disagreement.
Policy
Avg FQE ↑
Disagreement ↓
Score S↑
FQI
-0.18 ± 0.28
0.68
0.47
Imitation Learning
-4.32 ± 0.56
0.06
0.50
FQI–SC N∧=5
-3.04 ± 0.20
0.20
0.65
Table 2: Overall performance across all heads. Higher FQE indicates better estimated long-term reward, while lower disagreement indicates closer adherence to historical clinician behavior. The score S summarizes the balance between these two objectives. Green cells indicate more desirable values, red cells indicate less desirable values. FQE values are reported as mean ± standard deviation across head–fold evaluations.
N∧
Avg FQE ↑
Disagreement ↓
Score S↑
15
-3.56 ± 0.14
0.09
0.63
10
-3.56 ± 0.13
0.11
0.63
5
-3.04 ± 0.20
0.20
0.65
Table 3: Effect of the support threshold N∧ on support-constrained FQI (FQI-SC) across all heads. The score S summarizes the balance between these two objectives using the same normalization and harmonic-mean definition as in Table 2 .