Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment

Daniel J. Tan; Jayne Hui Zhen Chan; Kai Wen Hwang; Arturo Yong Yao Neo; Kay Choong See; Mengling Feng

cs.AIApr 12, 2026

Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment

Authors: Daniel J. Tan, Jayne Hui Zhen Chan, Kai Wen Hwang, Arturo Yong Yao Neo, Kay Choong See, Mengling Feng

Organizations: Institute of Data Science, National University of Singapore, Singapore · National University Hospital, Singapore · Saw Swee Hock School of Public Health, National University of Singapore, Singapore

Abstract

Designing reward functions for reinforcement learning (RL) in healthcare remains challenging because clinically meaningful outcomes are sparse, delayed, and difficult to explicitly specify. Although structured clinical data capture physiologic states, they often fail to reflect broader aspects of patient trajectories such as treatment response, recovery dynamics, and intervention burden. Clinical narratives, by contrast, encode longitudinal clinician assessments of disease progression, treatment effectiveness, and recovery, providing a potential source of trajectory-level supervision beyond predefined outcome metrics. We propose Clinical Narrative-informed Preference Rewards (CN-PR), a framework that learns reward functions directly from discharge summaries by treating clinical narratives as scalable supervision for trajectory-level preferences. Using a large language model, we derive trajectory quality scores and construct pairwise preferences between patient trajectories to learn rewards through preference-based optimization. To account for variability in narrative informativeness, we incorporate a task relevance signal that weights supervision according to its relevance to the downstream decision-making task. We evaluate CN-PR in dynamic sepsis treatment using offline RL. The learned reward demonstrated strong monotonic alignment with trajectory quality scores and produced policies associated with improved recovery-related outcomes, including increased organ support-free days and faster shock resolution, while maintaining mortality performance comparable to outcome-based reward baselines. These findings were preserved under external validation. Our results suggest that clinical narratives provide a scalable and expressive source of supervision for reward learning in dynamic treatment regimes.

Figures & tables

Figure 1: Overview of the CN-PR framework.

Category	Variables
Demographics/Scores (8)	Age, Gender, Weight, ICU readmission, GCS, Elixhauser s, SOFA, SIRS
Vitals (11)	HR, SBP, MBP, DBP, Resp. rate, Temperature, PaCO 2 , PaO 2 , PF ratio, SpO 2 , Shock index
Labs (24)	Albumin, pH, Calcium, Glucose, Hemoglobin, Magnesium, WBC, Creatinine, Bicarbonate, Sodium, Lactate, Chloride, Platelets, Potassium, PTT, PT, AST, ALT, BUN, INR, Ionised calcium, Total bilirubin, Base excess, Phosphate
Treatments and Interventions (2)	Mechanical ventilation, FiO 2
Others (3)	Urine output (4-hourly), Total output, Time since sepsis start

Table 1: State Space Features

Figure 2: Distribution of TQS (1–5) derived from clinical narratives across the full study cohort. Bars indicate the number of trajectories per score category, with percentages shown below each bar. Bar color encodes the mean task relevance score within each category.

Figure 3: Per-trajectory mean learned reward stratified by TQS (1 = lowest, 5 = highest) on the test set. Red markers denote group means.

Reward	OSFD-7	TSR	Mortality	IV Fluid Burden	Vasopressor Burden	Discharge Score
CN-PR	0.48	-0.22	-0.57	-0.18	-0.30	0.59
CN-PR (No TR)	0.44	-0.19	-0.55	-0.16	-0.26	0.55
Mortality-only	0.38	-0.08	-0.64	-0.02	-0.15	0.41
SOFA-Lac	0.30	-0.03	-0.63	-0.06	-0.08	0.40
NEWS2	0.22	-0.12	-0.48	-0.07	-0.17	0.41
Permuted-TQS	-0.06	0.10	0.03	-0.02	0.12	-0.02

Table 2: Spearman correlation between trajectory-level cumulative rewards and clinical outcomes for different reward formulations on the internal MIMIC-IV dataset.

Figure 4: Counterfactual joint treatment reward surfaces across severity strata. Each panel shows the expected reward as a function of IV fluid and vasopressor dosing levels for different patient groups: (A) low severity (SOFA < 5), (B) moderate severity (SOFA 5–15), and (C) high severity (SOFA > 15). Heatmap values represent the predicted reward for each joint treatment combination, illustrating how optimal dosing strategies vary with patient severity.

Figure 5: Relationship between policy–clinician discrepancy (measured as joint action distance) and clinical outcomes across multiple endpoints. Each subplot shows model-adjusted outcomes as a function of standardized joint action distance: A) OSFD-7, B) hospital mortality risk, C) TSR, D) discharge disposition score, E) mean vasopressor dose, and F) mean IV fluid dose. Curves represent regression-adjusted estimates for each policy. Shaded confidence intervals are shown for the CN-PR policy only to improve visual clarity; full regression results are provided in Supplementary Table S5.

Figure 6: Joint action distributions for IV fluids and vasopressors under clinician and CN-PR policies, stratified by severity (Low: SOFA $<8$ ; High: SOFA $\geq 8$ ). Cells show action proportions; only values $\geq 0.04$ are annotated.

Dataset	Cohort	% Female	Age (years)	ICU Stay (hours)	Total (n)
			Median (IQR)	Median (IQR)
MIMIC-IV	Overall	42.5	68 (57–79)	63.4 (34.7–123.2)	25,370
	Non-Survivors	44.0	72 (61–82)	97.1 (46.2–189.4)	3,877
	Survivors	42.2	68 (56–78)	58.4 (33.2–113.3)	21,493
AmsterdamUMCdb	Overall	40.1	65 (55–75) *	99.8 (39.3–270.8)	4,404
	Non-Survivors	41.0	65 (55–75) *	94.9 (39.3–278.5)	966

Table S1 : Cohort characteristics for the MIMIC-IV and AmsterdamUMCdb sepsis cohorts.

Quantile	0	1	2	3	4
IV Fluids (ml)	0	0.00–50.05	50.05–213.33	213.33–520.0	$>$ 520.0
Vasopressors (mcg/kg)	0	0.00–7.20	7.20–17.41	17.41–40.06	$>$ 40.06

Table S2 : Action Quantile Thresholds (4-hourly doses)

Task	Comparison	Quadratic-weighted $\kappa$	Spearman $\rho$	Exact Agreement	Within-1 Agreement
TQS	Clinician–clinician pairwise	$0.917\pm 0.007$	$0.931\pm 0.006$	$0.736$	$0.998$
	LLM–clinician pairwise	$0.910\pm 0.023$	$0.923\pm 0.011$	$0.722$	$0.989$
	LLM–clinician consensus	$0.915$	$0.926$	$0.747$	$0.987$
TR	Clinician–clinician pairwise	$0.684\pm 0.041$	$0.742\pm 0.028$	$0.518$	$0.821$
	LLM–clinician pairwise	$0.661\pm 0.052$	$0.718\pm 0.036$	$0.487$	$0.803$
	LLM–clinician consensus	$0.701$	$0.761$	$0.536$	$0.834$

Table S3 : Agreement between clinician-assigned and LLM-derived trajectory quality scores (TQS) and task relevance (TR) scores on a subset of 150 discharge summaries.

Reward	OSFD-7	TSR	Mortality	IV Fluid Burden	Vasopressor Burden
CN-PR	0.74	-0.36	-0.69	-0.17	-0.45
CN-PR (No TR)	0.72	-0.31	-0.68	-0.15	-0.43
Mortality-only	0.68	-0.27	-0.76	-0.04	-0.11
SOFA-Lac	0.67	-0.20	-0.72	-0.04	-0.11
NEWS2	0.60	-0.26	-0.56	-0.11	-0.34
Permuted-TQS	-0.04	0.04	0.06	-0.06	0.06

Table S4 : External validation on AmsterdamUMCdb: Spearman correlation between trajectory-level cumulative rewards and clinical outcomes for different reward formulations.

A. Hospital Mortality
	CN-PR	SOFA-Lac policy	Mortality policy	NEWS2 policy	Random policy
Distance	0.48 (0.4, 0.55)***	0.5 (0.42, 0.58)***	0.5 (0.42, 0.58)***	0.31 (0.24, 0.38)***	-0.01 (-0.09, 0.06)
Age	0.02 (0.02, 0.03)***	0.02 (0.02, 0.03)***	0.02 (0.02, 0.03)***	0.02 (0.02, 0.03)***	0.02 (0.02, 0.03)***
SOFA	0.11 (0.08, 0.14)***	0.11 (0.08, 0.13)***	0.11 (0.08, 0.14)***	0.12 (0.1, 0.15)***	0.15 (0.12, 0.17)***
MechVent	0.18 (-0.03, 0.39)	0.1 (-0.11, 0.31)	0.09 (-0.12, 0.3)	0.14 (-0.07, 0.34)	0.11 (-0.1, 0.31)
S.I.	0.75 (0.44, 1.06)***	0.73 (0.41, 1.04)***	0.66 (0.34, 0.97)***	0.88 (0.58, 1.19)***	0.99 (0.69, 1.29)***

Table S5 : Multivariable regression: MIMIC-IV. Coefficients shown as

\beta

(95% CI).

A. Hospital Mortality
	CN-PR	SOFA-Lac policy	Mortality policy	NEWS2 policy	Random policy
Distance	0.54 (0.42, 0.65)***	0.6 (0.47, 0.72)***	0.58 (0.47, 0.7)***	0.31 (0.2, 0.43)***	0.24 (0.12, 0.37)***
Age	0.03 (0.02, 0.04)***	0.03 (0.02, 0.04)***	0.03 (0.02, 0.04)***	0.03 (0.02, 0.04)***	0.03 (0.02, 0.04)***
SOFA	0.22 (0.18, 0.26)***	0.21 (0.17, 0.25)***	0.22 (0.18, 0.26)***	0.22 (0.17, 0.26)***	0.24 (0.19, 0.28)***
MechVent	0.33 (0.23, 0.42)***	0.32 (0.22, 0.42)***	0.31 (0.21, 0.4)***	0.33 (0.23, 0.42)***	0.32 (0.22, 0.41)***
S.I.	0.42 (0.06, 0.78)*	0.41 (0.05, 0.78)*	0.42 (0.05, 0.79)*	0.49 (0.13, 0.86)**	0.48 (0.12, 0.83)**

Table S6 : Multivariable regression: AmsterdamUMCdb (External Validation). Coefficients shown as

\beta

(95% CI).

Figure S1 : Permutation feature importance for predicting discretized treatment actions under clinician and CN-PR policies. Separate models were trained for (a) IV fluids and (b) vasopressors, with importance computed via permutation. The top 10 features for each policy are shown to facilitate comparison of key drivers of treatment decisions.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 259 papers

Oct 18, 2026

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

Offline sepsis treatment aims to learn treatment policies from retrospective ICU trajectories. Decision Transformer-based methods provide a natural sequence modeling framework for this task, but they mainly rely on raw physiological states and historical clinician actions. Such raw trajectory tokens may not explicitly capture clinical semantics that clinicians use in practice, such as patient severity, recent treatment response, and current risk priorities. To address this limitation, we propose ClinSDT, an LLM-assisted Decision Transformer framework for offline sepsis treatment. ClinSDT uses a frozen LLM as a semantic bridge rather than a treatment decision maker. It constructs three clinical semantic views from ICU trajectories, namely Profile, Context, and Advisory, and inserts their encoded representations as independent tokens into a Decision Transformer policy. To support policy refinement and rollout, ClinSDT further trains a Treatment-Response Forecaster to estimate the next physiological state, SAPS-II change, and predictive uncertainty from state-action inputs. In the second training stage, the forecaster screens local counterfactual treatment candidates, and only low-uncertainty candidates with lower predicted SAPS-II increase are distilled into the policy after semantic regeneration. During inference, ClinSDT performs closed-loop treatment rollout by refreshing semantic inputs from the currently available patient state. Experiments on MIMIC-III and MIMIC-IV show that ClinSDT achieves consistent relative improvements over representative imitation learning, offline reinforcement learning, and Decision Transformer baselines under the same rollout protocol. Code and data are available at https://github.com/KAG778/ClinSDT.

Yanyan Wu, Guanyu Zhu, Changbin Zhao +7

South China Agricultural University, Guangzhou, China · The Ohio State University, Columbus, Ohio, United States

Healthcare Clinical Decision-Making Offline RL LLM-Guided RL Decision Transformer

Ask PDF

Jul 7, 2026

EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins

Sepsis is a leading cause of mortality, yet optimal treatment policies remain contested. Existing reinforcement learning (RL) approaches learn fixed strategies for sepsis treatment, limiting adaptability to changing clinical objectives during inference. We propose EHRMPC, a framework that decouples learning patient dynamics from optimizing treatment by training a patient digital twin in the form of a generative electronic health record (EHR) model. The digital twin predicts clinical trajectories under interventions and enables model predictive control (MPC) to optimize treatments via inference-time planning over simulations. We evaluate EHR-MPC on a multicenter ICU sepsis cohort spanning 8 hospitals in the Mass General Brigham health system using both off-policy importance sampling and on-policy simulation-based evaluation. Relative to RL baselines, EHR-MPC achieves comparable off-policy performance and improved simulation performance. Unlike RL, this work frames sepsis treatment optimization as inference-time control over learned patient dynamics, establishing a general framework for decision making with generative clinical models.

Joshua Pickard, Wei Qi, Na Li +4

Broad Institute of MIT and Harvard · Harvard University · Brigham and Women’s Hospital +2

Inference-Time Optimization Model Predictive Control

Ask PDF

Oct 2, 2026

RASPER: Reward-Aligned Summarization of Clinical Notes for EHR Outcome Prediction

Unstructured discharge notes in Electronic Health Records (EHRs) often carry signal complementary to structured medical codes, holding patient-specific evidence that standardized cohort-level codes alone cannot capture. However, this evidence in notes is frequently buried in lengthy, noisy text that is not intentionally written with any specific clinical prediction in mind. Summarization is an obvious mitigation, but generic summaries, tuned for fluency rather than the outcome, routinely omit decisive evidence while retaining plausible but uninformative detail. To this end, we propose RASPER, a Reward-Aligned Summarizer for Prediction in EHR, that optimizes note summarization directly against the downstream clinical task. RASPER employs a tunable LLM-based summarizer to extract task-relevant evidence from discharge notes and trains it via reinforcement learning from prediction feedback, using a reward derived from the downstream predictor's loss. To ground the summarizer, a longitudinal encoder converts structured codes into soft prompts that incorporate each patient's clinical context into note summarization. By rewarding the quality of the resulting multimodal prediction, RASPER encourages the summarizer to retain patient-specific evidence that complements, rather than duplicates, information captured by structured codes. RASPER consistently outperforms strong baselines on both readmission prediction and medication recommendation across MIMIC-III and MIMIC-IV.

Arya Hadizadeh Moghaddam, Mohsen Nayebi Kerdabadi, Chen Chen +2

Electrical Engineering and Computer Science, University of Kansas, USA

Medication Recommendation Text Summarization Clinical Outcome Prediction Clinical Decision Support Longitudinal EHR Modeling

Ask PDF

Abstract

Reward Design for RL

Explore similar work

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins

RASPER: Reward-Aligned Summarization of Clinical Notes for EHR Outcome Prediction