Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data
Authors: Fateme Golivand Darvishvand, Saurabh Mathur, Nikhilesh Prabhakar, Michael A. Skinner, Ameet Soni, Neel Shah, Ethan L Sanford, Lakshmi Raman, Sriraam Natarajan
Organizations: University of Texas at Dallas · Technische Universität Darmstadt · Swarthmore College · Washington University in St. Louis · UT Southwestern Medical Center
Managing children on ECMO (Extracorporeal Membrane Oxygenation) requires balancing short-term physiological stability with the long-term goal of preventing neurological injury. Standard reinforcement learning can propose risky, out-of-distribution actions, while imitation learning is limited to reproducing clinician behavior. We introduce Support-Constrained Fitted Q-Iteration (FQI-SC), an offline reinforcement learning framework that cautiously improves upon expert behavior by restricting optimization to empirically supported state-action pairs and defaulting to the clinician baseline in low-data regions. On 78 pediatric trajectories totaling over 6,000 hours of clinical data, we evaluate our framework using Fitted Q-Evaluation (FQE) and a behavioral disagreement. Additionally, we formalize the performance-safety trade-off by introducing a unified Balance Score (S(π)). Our empirical evaluation shows that support-constrained policy improves the Balance Score by 38% relative to FQI, improves estimated value by 1.28 FQE units over imitation learning, and reduces disagreement with clinician actions by 71% relative to FQI.
Figures & tables
Figure 1: Balancing Safety and Performance in Pediatric ECMO. (a) We frame management of children on ECMO life support as an offline policy learning problem using logged clinician trajectories. (b) FQI attains high estimated value but deviates substantially from clinician behavior, whereas IL remains close to clinicians but has lower estimated value. FQI-SC seeks an intermediate operating point by restricting improvement to empirically supported actions.
Variable
Target / Rule
Unit
Weight
BP
(0–1)
age-norm.
2.0
HR
(0–1)
age-norm.
1.5
SpO 2
92–100
%
1.5
PaO 2
80–150
mmHg
1.5
PaCO 2
35–45
mmHg
2.0
Arterial pH
7.30–7.45
–
1.5
Table 1: Clinician-defined target ranges and importance weights used in the stability component of the reward. Weights are assigned per clinical category.
Figure 2: Per-head and all-head trade-off between estimated value and behavioral conservatism. Each point shows a policy’s mean FQE and disagreement rate, with error bars denoting variability across folds. Plain FQI occupies the high-value but high-disagreement regime, imitation learning occupies the low-disagreement but lower-value regime, and support-constrained FQI (FQI-SC) lies between these extremes. Different marker shapes indicate the support threshold N∧∈{5,10,15} , and the shaded upper-left region highlights the desirable operating regime of higher FQE and lower disagreement.
Policy
Avg FQE ↑
Disagreement ↓
Score S↑
FQI
-0.18 ± 0.28
0.68
0.47
Imitation Learning
-4.32 ± 0.56
0.06
0.50
FQI–SC N∧=5
-3.04 ± 0.20
0.20
0.65
Table 2: Overall performance across all heads. Higher FQE indicates better estimated long-term reward, while lower disagreement indicates closer adherence to historical clinician behavior. The score S summarizes the balance between these two objectives. Green cells indicate more desirable values, red cells indicate less desirable values. FQE values are reported as mean ± standard deviation across head–fold evaluations.
N∧
Avg FQE ↑
Disagreement ↓
Score S↑
15
-3.56 ± 0.14
0.09
0.63
10
-3.56 ± 0.13
0.11
0.63
5
-3.04 ± 0.20
0.20
0.65
Table 3: Effect of the support threshold N∧ on support-constrained FQI (FQI-SC) across all heads. The score S summarizes the balance between these two objectives using the same normalization and harmonic-mean definition as in Table 2 .
Pediatric critical care is a dynamic, high-stakes process involving constant monitoring and adjustments in life-saving treatments. Modeling these interventions is crucial for effective decision support. To address the challenges of high complexity and data scarcity in pediatric Extracorporeal Membrane Oxygenation (ECMO), we frame clinical decision-making as learning to act from trajectories, i.e., imitation learning that learns action models from observational data, with a key feature that actions are not directly observed. We consider TabPFN, a recent transformer-based approach for tabular data, and traditional baselines including XGBoost and Multi-Layer Perceptrons(MLPs) on real-world pediatric ECMO data to learn the action models. We find that the TabPFN-based approach consistently outperforms these classical baselines, supporting its use as a strong clinician-behavior baseline for pediatric ECMO decision support.
Fateme Golivand, Michael Skinner, Saurabh Mathur +5
University of Texas at Dallas · Technische Universität Darmstadt · Swarthmore College +1
Offline reinforcement learning seeks optimal decision rules from previously collected data. In some applications, a decision can be an entire function, such as a fluence map in radiation therapy or a smooth movement trajectory in robotics. In this paper, we study the finite-sample theory for fitted Q-iteration (FQI) with functional actions in a discounted infinite-horizon setting. Three major difficulties arise in this setting: first, the absence of a Lebesgue probability density for functional actions complicates coverage descriptions; second, conventional coverage requirements can be restrictive; and third, the large functional action space makes greedy optimization in FQI challenging. To address these difficulties, we study smoothness-regularized policy search under a critic-relative coverage condition. This condition measures how well logged data distinguish relevant action-value differences without requiring an action density. Our main theorem gives finite-sample guarantees for learned-policy regret relative to the best value within a fixed smooth class of functional-action policies. The results allow trajectory lengths to be either bounded or growing and Q-functions to be fitted by either functional-input kernel ridge regression or adaptive functional neural networks. For a few examples, we can obtain polynomially decaying regret bounds in the number of logged transitions, up to logarithmic factors, with logarithmically many FQI iterations. Numerical experiments show gains of learned functional-action policies over constant-action policies and support our adoption of a critic-relative coverage condition.
Gefei Lin, Rui Miao, Xiaoke Zhang
Department of Statistics, The George Washington University · Department of Mathematical Sciences, The University of Texas at Dallas
Offline reinforcement learning methods typically enforce strict constraints to ensure safety; yet this rigidity often prevents the discovery of optimal behaviors outside the immediate support of the behavior policy. To address this, we propose Implicit Support Expansion via stochastic Policy optimization (ISEP), which leverages a value function interpolated between in-distribution data and policy samples to implicitly expand the feasible action support. This mechanism "densifies" high-reward regions, creating a navigable path for policy improvement while theoretically guaranteeing bounded value error. However, optimizing against this expanded support creates a multimodal landscape where standard deterministic averaging leads to mode collapse and invalid actions. ISEP mitigates this via a stochastic action selection strategy, optimizing the policy by stochastically alternating between conservative cloning and optimistic expansion signals. We instantiate this framework as ISEP-FM using Conditional Flow Matching utilizing classifier-free guidance to effectively capture the interpolated value signal.