Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation

Marc Pérez-Roig; David Fernández-Narro; Carlos Sáez

cs.AIAug 17, 2026

Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation

Authors: Marc Pérez-Roig, David Fernández-Narro, Carlos Sáez

Organizations: Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, València, València, Spain

Abstract

The dosing of intravenous fluids and vasopressors in sepsis is a sequential decision made under uncertainty and guided largely by clinical judgment, which makes it a natural target for reinforcement learning from historical care. Because a learned policy cannot be trialed on patients, its value must be estimated off-policy, and such estimates can be fragile and optimistic. This work advances the reliable evaluation of sepsis treatment policies by combining off-policy estimation, reliability diagnostics, and clinician-agreement analyses in a transparent validation framework. We modeled fluid and vasopressor dosing on a cohort of 36,872 septic ICU stays drawn from the MIMIC-IV critical-care database, as a discretized Markov decision process with 1,000 states and 25 actions, defined by a five-by-five grid of fluid and vasopressor levels and solved by policy iteration. The clinicians' behavior policy was estimated with a random forest, which mitigated the collapse of the Effective Sample Size (ESS 50.1 against 4.0 with smoothed counts) that otherwise destabilizes the importance-sampling estimate. The learned policy was evaluated with two estimators, weighted importance sampling (WIS) and fitted Q evaluation (FQE), with the ESS and clinician agreement as reliability checks. An empirical variable selection found that the composition of the state matters more than its size. Both estimators place the learned policy above the clinicians' return (WIS 50.8 and FQE 46.8 against 38.2, ESS 50.1), yet it departs only modestly from observed practice (total variation 0.18), favoring less intravenous fluid. These retrospective single-center off-policy results support the learned policy as a clinically plausible refinement of observed practice and motivate its further evaluation as a discordance-based clinical decision-support approach.

Figures & tables

Indicator	MIMIC-IV (ours)	MIMIC-III	eRI
ICU mortality	10.7	7.4	9.8
In-hospital mortality	16.2	8.9	16.4
28-day mortality	19.8	11.3	n/a
90-day mortality	27.3	18.9	n/a

Table 1: Descriptive mortality of the MIMIC-IV cohort against the development (MIMIC-III) and external validation (eRI) cohorts of the AI Clinician [ 11 ] . In-hospital mortality ( hospital_expire_flag ) governs the terminal reward of the MDP; the remaining indicators are descriptive. All values are percentages.

Configuration
Component	Value
State-variable set	Main set ( core_hi75 ), 22 variables
Number of states $K$	$1000$ plus 2 absorbing states
Support threshold $M$	25
Discount $\gamma$	0.95
Dose penalty $P$	0.05

Table 2: Final configuration selected by the decision rule, with the sizing diagnostics of the trained model on the validation split. The 22 state variables are listed in Supplementary Table S2.

Figure 1: Modal action over the $5\times 5$ grid of fluid (horizontal, 0 none to 4 maximal) and vasopressor (vertical) levels, as the percentage of states. (a) Clinicians. (b) Learned policy. (c) Difference in percentage points, learned minus clinicians (red, actions the policy favors more than clinicians; blue, less).

Figure 2: Off-policy evaluation of the final configuration. (a) Estimated return of the clinicians’ policy (empirical) and of the learned policy by weighted importance sampling (WIS) and fitted Q evaluation (FQE), on the terminal-reward scale, with bootstrap $95\text{\,}\mathrm{\%}$ confidence intervals. (b) The decision-rule funnel applied to the $1200$ candidates of the sweep: an ESS floor of 50 ( $1200$ $\to$ $77$ ), a plausibility gate on the non-intervention rate ( $77$ $\to$ $65$ ), triangulation requiring both WIS and FQE above the clinicians’ return ( $65$ $\to$ $26$ ), and the confidence-interval robustness layer of Section 4.5 ( $26$ $\to$ $3$ ) that resolves the final choice among $K=$1000$$ , $M=25$ survivors. — Figure 2: Off-policy evaluation of the final configuration. (a) Estimated return of the clinicians’ policy (empirical) and of the learned policy by weighted importance sampling (WIS) and fitted Q evaluation (FQE), on the terminal-reward scale, with bootstrap $95\text{\,}\mathrm{\%}$ confidence intervals. (b) The decision-rule funnel applied to the $1200$ candidates of the sweep: an ESS floor of 50 ( $1200$ $\to$ $77$ ), a plausibility gate on the non-intervention rate ( $77$ $\to$ $65$ ), triangulation requiring both WIS and FQE above the clinicians’ return ( $65$ $\to$ $26$ ), and the confidence-interval robustness layer of Section 4.5 ( $26$ $\to$ $3$ ) that resolves the final choice among $K=$ 1000$$ , $M=25$ survivors.

Figure 3: (a) Agreement between the learned policy and the clinicians, by treatment dimension (global, fluids, vasopressors and with a one-level tolerance), by whether the clinician intervenes, and by outcome (discharge versus death). (b, c) Observed in-hospital mortality on the validation split against the dose-level difference between clinician and policy (clinician minus policy; 0 is exact match) for fluids (b) and vasopressors (c); gray bars are patients per bin, error bars are $95\text{\,}\mathrm{\%}$ Wilson intervals, the horizontal dashed line is the cohort mortality and the vertical line the exact match.

Figure 4: One illustrative discharged patient’s trajectory, near the mean agreement of her outcome group: SOFA score over the stay; the clinician’s and the learned policy’s vasopressor and fluid doses at each $4\text{\,}\mathrm{h}$ step; and a band marking step-by-step agreement (exact match, tolerant within one bin, or mismatch).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Cohort selection flow. From the $94\,458$ ICU stays in MIMIC-IV, the adult Sepsis-3 criteria ( sepsis3 derived view) delimit $41\,295$ stays, to which two cumulative exclusions are applied, treatment withdrawal and the absence of documented intravenous fluids. The final cohort comprises $36\,872$ ICU stays ( $28\,605$ unique patients), $89.3\text{\,}\mathrm{\%}$ of the Sepsis-3 stays.

Figure S2: Overview of the preprocessing and MDP-construction pipeline: extraction, discretization into $4\text{\,}\mathrm{h}$ windows, imputation, scaling, state clustering, and estimation of the transition and reward models.

Category	Variable (unit)	Median [IQR]	Mean $\pm$ SD
Vital signs
	Heart rate (bpm)	86.2 [75.0, 100.0]	88.2 $\pm$ 18.6
	Systolic blood pressure (mmHg)	114.7 [103.8, 128.7]	117.4 $\pm$ 19.5
	Mean arterial pressure (mmHg)	77.0 [69.2, 86.0]	78.5 $\pm$ 13.6
	Respiratory rate (min -1 )	19.0 [16.0, 22.8]	19.8 $\pm$ 5.1
	Temperature ( ∘ C)	36.8 [36.4, 37.2]	36.8 $\pm$ 0.9

Appendix

Table S1: Baseline clinical state of the cohort in the initial

4\text{\,}\mathrm{h}

block

[0,4)

after onset. Each variable is summarized as median with interquartile range [IQR] and as mean

\pm

standard deviation (SD), computed over observed values without imputation.

Variable	Clinical role	Transf.
Vital signs
heart_rate	Heart rate	z
mbp	Mean arterial pressure	z
resp_rate	Respiratory rate	z
spo2	Peripheral oxygen saturation	z
temperature	Temperature	z

Appendix

Table S2: State variables of the final configuration (22 variables). The transformation applied before clustering is coded as: log ,

\log(1+x)

followed by standardization; z , standardization only; bin , centered binary. The empirical selection of this set is described in the main-text variable selection.

Level	0	1	2	3	4
IV fluids per $4\text{\,}\mathrm{h}$ ( $\mathrm{ml}$ )	$0$	$(0,100]$	$(100,259.7]$	$(259.7,650]$	$>650$
Vasopressor ( $\mathrm{\SIUnitSymbolMicro g}\text{\,}{\mathrm{kg}}^{-1}\text{\,}{\mathrm{min}}^{-1}$ )	$0$	$(0,0.0503]$	$(0.0503,0.1002]$	$(0.1002,0.2198]$	$>0.2198$

Appendix

Table S3: Action discretization. Each lever is split into five levels by the quartiles of strictly positive doses on the training split. Vasopressor doses are norepinephrine equivalents.

Set	$n$	Inclusion rule	Role
Sepsis-3 core ( sepsis3_min )	16	Treatment, Sepsis-3/SOFA axes and core vital signs	Minimal clinical floor
Main set ( core_hi75 )	22	Coverage $\geq$ 75\text{,}\mathrm{%}$$ , treatment and Sepsis-3 override, no redundancies	Primary candidate
Extended labs ( core_labs30 )	29	Coverage $\geq$ 30\text{,}\mathrm{%}$$ (adds imputed labs) and override, no redundancies	Whether sparse labs help
Komorowski replica ( komorowski46 )	46	Pool variables present in [ 11 ]	Replicative control
Full set ( all50 )	50	All pool variables	Negative control

Appendix

Table S4: The five candidate variable sets carried into the final sweep. The two controls (full set and Komorowski replica) reproduce the non-intervention collapse; the curated sets do not.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 292 papers

Oct 18, 2026

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

Offline sepsis treatment aims to learn treatment policies from retrospective ICU trajectories. Decision Transformer-based methods provide a natural sequence modeling framework for this task, but they mainly rely on raw physiological states and historical clinician actions. Such raw trajectory tokens may not explicitly capture clinical semantics that clinicians use in practice, such as patient severity, recent treatment response, and current risk priorities. To address this limitation, we propose ClinSDT, an LLM-assisted Decision Transformer framework for offline sepsis treatment. ClinSDT uses a frozen LLM as a semantic bridge rather than a treatment decision maker. It constructs three clinical semantic views from ICU trajectories, namely Profile, Context, and Advisory, and inserts their encoded representations as independent tokens into a Decision Transformer policy. To support policy refinement and rollout, ClinSDT further trains a Treatment-Response Forecaster to estimate the next physiological state, SAPS-II change, and predictive uncertainty from state-action inputs. In the second training stage, the forecaster screens local counterfactual treatment candidates, and only low-uncertainty candidates with lower predicted SAPS-II increase are distilled into the policy after semantic regeneration. During inference, ClinSDT performs closed-loop treatment rollout by refreshing semantic inputs from the currently available patient state. Experiments on MIMIC-III and MIMIC-IV show that ClinSDT achieves consistent relative improvements over representative imitation learning, offline reinforcement learning, and Decision Transformer baselines under the same rollout protocol. Code and data are available at https://github.com/KAG778/ClinSDT.

Yanyan Wu, Guanyu Zhu, Changbin Zhao +7

South China Agricultural University, Guangzhou, China · The Ohio State University, Columbus, Ohio, United States

Healthcare Clinical Decision-Making Offline RL LLM-Guided RL Decision Transformer

Ask PDF

Jul 7, 2026

EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins

Sepsis is a leading cause of mortality, yet optimal treatment policies remain contested. Existing reinforcement learning (RL) approaches learn fixed strategies for sepsis treatment, limiting adaptability to changing clinical objectives during inference. We propose EHRMPC, a framework that decouples learning patient dynamics from optimizing treatment by training a patient digital twin in the form of a generative electronic health record (EHR) model. The digital twin predicts clinical trajectories under interventions and enables model predictive control (MPC) to optimize treatments via inference-time planning over simulations. We evaluate EHR-MPC on a multicenter ICU sepsis cohort spanning 8 hospitals in the Mass General Brigham health system using both off-policy importance sampling and on-policy simulation-based evaluation. Relative to RL baselines, EHR-MPC achieves comparable off-policy performance and improved simulation performance. Unlike RL, this work frames sepsis treatment optimization as inference-time control over learned patient dynamics, establishing a general framework for decision making with generative clinical models.

Joshua Pickard, Wei Qi, Na Li +4

Broad Institute of MIT and Harvard · Harvard University · Brigham and Women’s Hospital +2

Inference-Time Optimization Model Predictive Control

Ask PDF

Aug 11, 2026

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE) assess only behavioral imitation and cannot detect Toxic Mimicry, a failure mode in which agents replicate harmful patterns such as treatment withdrawal during comfort-care transitions. Using the MIMIC-III database, we propose the Counterfactual Clinical Audit (CCA) framework, which stress-tests RL agents through physiological perturbations anchored in Surviving Sepsis Campaign (SSC) guidelines. We audit a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), the latter employing Causal Action Shielding, propensity-based importance weighting, and Conservative Q-Learning. CCA reveals that MedDT paradoxically reduces vasopressor dosage as lactate escalates, contradicting resuscitation guidelines, while HCT-RL maintains physiologically consistent responses. These findings expose a systemic misalignment between statistical fit and clinical safety, supporting counterfactual audits as a necessary evaluation standard for medical RL.

Hangqi Ren, Junyi Liao

School of Engineering, Vanderbilt University, Nashville, TN, USA · Pratt School of Engineering, Duke University, Durham, NC, USA

Counterfactual Evaluation Offline RL Safe RL

Ask PDF

Abstract

Offline RL

Explore similar work

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits