Offline sepsis treatment aims to learn treatment policies from retrospective ICU trajectories. Decision Transformer-based methods provide a natural sequence modeling framework for this task, but they mainly rely on raw physiological states and historical clinician actions. Such raw trajectory tokens may not explicitly capture clinical semantics that clinicians use in practice, such as patient severity, recent treatment response, and current risk priorities. To address this limitation, we propose ClinSDT, an LLM-assisted Decision Transformer framework for offline sepsis treatment. ClinSDT uses a frozen LLM as a semantic bridge rather than a treatment decision maker. It constructs three clinical semantic views from ICU trajectories, namely Profile, Context, and Advisory, and inserts their encoded representations as independent tokens into a Decision Transformer policy. To support policy refinement and rollout, ClinSDT further trains a Treatment-Response Forecaster to estimate the next physiological state, SAPS-II change, and predictive uncertainty from state-action inputs. In the second training stage, the forecaster screens local counterfactual treatment candidates, and only low-uncertainty candidates with lower predicted SAPS-II increase are distilled into the policy after semantic regeneration. During inference, ClinSDT performs closed-loop treatment rollout by refreshing semantic inputs from the currently available patient state. Experiments on MIMIC-III and MIMIC-IV show that ClinSDT achieves consistent relative improvements over representative imitation learning, offline reinforcement learning, and Decision Transformer baselines under the same rollout protocol. Code and data are available at https://github.com/KAG778/ClinSDT.
Figures & tables
Figure 1. Motivation of ClinSDT. Raw ICU trajectories leave clinical decision semantics implicit in numerical states and actions. ClinSDT uses a frozen LLM to encode structured clinical context, while a Decision Transformer remains responsible for treatment action generation. Motivation diagram of ClinSDT showing that raw ICU trajectories contain implicit clinical semantics, which are encoded by a frozen LLM as structured context for a Decision Transformer policy.
Figure 2. Overview of ClinSDT. ClinSDT encodes Profile , Context , and Advisory semantics from ICU trajectories with a frozen LLM and injects them as independent tokens into a Decision Transformer policy. Training consists of Stage I semantic-token policy and response-forecaster learning, followed by Stage II uncertainty-constrained counterfactual semantic refinement. During closed-loop rollout, the policy refreshes semantic inputs from each predicted patient state while retaining the retrospectively initialized target-return condition. The LLM provides semantic representations only; treatment actions are generated by the policy model. In the schematic, Ω≡Fϕ and Ls≡Lfrfact . Overview of ClinSDT, showing how Profile, Context, and Advisory semantics are encoded by a frozen LLM, injected as independent tokens into a Decision Transformer policy, and used with treatment-response forecasting and uncertainty-constrained conservative counterfactual distillation for model-based treatment rollout.
Dataset
Model
5-step rollout
10-step rollout
HIGH
MID
LOW
ALL
HIGH
MID
LOW
ALL
MIMIC-III
BC
3.24
-1.51
-5.59
-0.59
5.39
-1.55
-7.80
-0.22
BCQ
3.17
-1.30
-3.88
-0.35
5.99
-1.17
-5.59
0.34
DQN
6.42
-1.05
-5.03
0.57
13.17
0.10
-6.74
2.94
DT
7.01
-1.57
-5.93
0.30
12.96
-1.40
-8.98
1.72
IQL
2.86
-2.13
-6.13
-1.14
4.39
-2.58
-9.25
-1.28
Table 1. Overall model-based rollout results on MIMIC-III and MIMIC-IV. We compare ClinSDT with representative baselines from imitation learning, offline RL, value-based RL, and Decision Transformer-style policy learning. The metric is negative cumulative forecaster-predicted SAPS-II change. All methods follow the same rollout procedure, while each uses its own learned forecaster; the values are therefore protocol-relative point estimates rather than scores on a shared independent scale. Bold and underlining indicate only the numerical ordering within the table.
Figure 3. Overall model-based rollout results on MIMIC-III and MIMIC-IV. ClinSDT has the highest ALL-patient point estimate in all four reported dataset–horizon settings under the implemented rollout protocol. A bar chart showing method-specific rollout estimates for ClinSDT and the baseline methods on MIMIC-III and MIMIC-IV under 5-step and 10-step settings. The chart summarizes negative cumulative forecaster-predicted SAPS-II changes for all patients.
Group
Variant
5-step rollout
10-step rollout
HIGH
MID
LOW
ALL
HIGH
MID
LOW
ALL
Removal
w/o Profile
3.14
-0.54
-4.96
0.09
2.52
0.30
-0.03
0.84
w/o Context
5.66
-0.43
-4.97
0.80
5.92
-0.14
-0.68
1.37
w/o Advisory
5.97
-1.30
-7.94
0.09
4.26
0.05
-0.28
1.10
w/o Semantics
7.06
-2.48
-9.69
-0.56
4.82
0.41
0.39
1.53
Single-view
only Profile
8.45
-0.79
-3.67
1.36
1.68
-0.94
-0.66
-0.25
Table 2. Semantic-view variants on MIMIC-III under 5-step and 10-step rollouts. Results are method-specific negative cumulative forecaster-predicted SAPS-II changes. Bold and underlining indicate only the highest and second-highest numerical point estimates.
Variant
Design
HIGH
MID
LOW
ALL
Mean
zero-param pooling
0.49
-0.24
-1.78
-0.16
Concat
one projection
0.62
-0.35
-7.43
-0.60
Gated
two projections
4.08
-1.19
-21.50
-1.28
Residual
residual projection
-0.02
-1.34
-6.78
-1.38
Cross-attn
MHA + LN
-0.94
-1.48
-5.89
-1.65
Ours
6-token block (3 semantic)
17.74
5.16
10.77
8.75
Table 3. Comparison of token-fusion designs on MIMIC-III under the 10-step rollout. Higher values indicate a more favorable estimate under the corresponding learned forecaster.
Figure 4. Token-fusion variants on MIMIC-III. The separated-token design has the highest 10 -step point estimate among the complete fusion configurations. A bar chart comparing different semantic token-fusion designs on MIMIC-III under the 10-step rollout. The separated-token design has the highest numerical estimate across the reported groups under its learned forecaster.
Figure 5. Stage I-to-Stage II change in the method-relative rollout estimate on MIMIC-III. The y-axis is defined as the negative cumulative SAPS-II change predicted by each method’s own forecaster; higher is more favorable under that forecaster. Line plot showing the Stage I-to-Stage II change in method-relative rollout estimates on MIMIC-III. The y-axis is the negative cumulative SAPS-II change predicted by each method's own forecaster, so higher values are more favorable under that forecaster.
Model
5-step estimate change
10-step estimate change
HIGH
MID
LOW
ALL
HIGH
MID
LOW
ALL
DQN
2.93
0.45
0.28
1.06
6.76
1.41
1.07
2.75
DT
3.79
0.76
1.28
1.56
7.03
1.29
1.86
2.79
BCQ
1.21
1.38
1.71
1.36
2.70
2.15
2.58
2.32
BC
0.08
0.49
0.46
0.39
0.79
0.86
0.62
0.83
IQL
0.11
0.43
0.24
0.33
0.74
0.87
0.68
0.83
Table 4. Stage I-to-Stage II change in each method’s forecaster-based estimate on MIMIC-III.
Stratum
Profile
Context
Advisory
Sem.
RTG
State
HIGH
+0.0031
−0.0016
+0.0014
+0.0029
+0.0013
−0.0043
MID
−0.0012
+0.0007
−0.0005
−0.0010
−0.0007
+0.0016
LOW
−0.0021
−0.0003
−0.0018
−0.0042
+0.0016
+0.0025
Table 5. Stratum-wise relative attention scores of the Stage II policy on MIMIC-III. Positive values indicate higher relative attention to the corresponding token group, while negative values indicate lower relative attention.
Figure 6. Stratum-wise relative attention heatmap of the Stage II policy on MIMIC-III. Warmer and cooler colors indicate higher and lower relative attention, respectively. Attention heatmap showing stratum-wise relative attention of the Stage II policy on MIMIC-III across Profile, Context, Advisory, semantic, RTG, and State token groups.
Group
n
Top1
σbeh
σpred
OOD%
KL
High
8362
0.458
0.910
0.907
4.87
0.662
Mid
18646
0.483
0.943
0.942
4.93
0.712
Low
1914
0.399
1.010
1.012
5.17
0.883
Overall
28922
0.470
0.938
0.937
4.84
0.685
Table 6. Detailed behavior-support statistics of the Stage II policy on MIMIC-III.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Implementation
Cohort
Adult Sepsis-3 trajectories following the AI Clinician pipeline.
Sepsis management in the ICU requires sequential treatment decisions under rapidly evolving patient physiology. Although large language models (LLMs) encode broad clinical knowledge and can reason over guidelines, they are not inherently grounded in action-conditioned patient dynamics. We introduce SepsisAgent, a world model-augmented LLM agent for sepsis treatment recommendation. SepsisAgent uses a learned Clinical World Model to simulate patient responses under candidate fluid--vasopressor interventions, and follows a propose--simulate--refine workflow before committing to a prescription. We first show that world-model access alone yields inconsistent LLM decision performance, motivating agent-specific training. We then train SepsisAgent through a three-stage curriculum: patient-dynamics supervised fine-tuning, propose--simulate--refine behavior cloning, and world-model-based agentic reinforcement learning. On MIMIC-IV sepsis trajectories, SepsisAgent outperforms all traditional RL and LLM-based baselines in off-policy value while achieving the best safety profile under guideline adherence and unsafe-action metrics. Further analysis shows that repeated interaction with the Clinical World Model enables the agent to learn regularities in patient evolution, which remain useful even when simulator access is removed.
Minghao Wu, Yuting Yan, Zhenyang Cai +9
The Chinese University of Hong Kong, Shenzhen · Beijing Hospital
The dosing of intravenous fluids and vasopressors in sepsis is a sequential decision made under uncertainty and guided largely by clinical judgment, which makes it a natural target for reinforcement learning from historical care. Because a learned policy cannot be trialed on patients, its value must be estimated off-policy, and such estimates can be fragile and optimistic. This work advances the reliable evaluation of sepsis treatment policies by combining off-policy estimation, reliability diagnostics, and clinician-agreement analyses in a transparent validation framework. We modeled fluid and vasopressor dosing on a cohort of 36,872 septic ICU stays drawn from the MIMIC-IV critical-care database, as a discretized Markov decision process with 1,000 states and 25 actions, defined by a five-by-five grid of fluid and vasopressor levels and solved by policy iteration. The clinicians' behavior policy was estimated with a random forest, which mitigated the collapse of the Effective Sample Size (ESS 50.1 against 4.0 with smoothed counts) that otherwise destabilizes the importance-sampling estimate. The learned policy was evaluated with two estimators, weighted importance sampling (WIS) and fitted Q evaluation (FQE), with the ESS and clinician agreement as reliability checks. An empirical variable selection found that the composition of the state matters more than its size. Both estimators place the learned policy above the clinicians' return (WIS 50.8 and FQE 46.8 against 38.2, ESS 50.1), yet it departs only modestly from observed practice (total variation 0.18), favoring less intravenous fluid. These retrospective single-center off-policy results support the learned policy as a clinically plausible refinement of observed practice and motivate its further evaluation as a discordance-based clinical decision-support approach.
Marc Pérez-Roig, David Fernández-Narro, Carlos Sáez
Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, València, València, Spain
Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE) assess only behavioral imitation and cannot detect Toxic Mimicry, a failure mode in which agents replicate harmful patterns such as treatment withdrawal during comfort-care transitions. Using the MIMIC-III database, we propose the Counterfactual Clinical Audit (CCA) framework, which stress-tests RL agents through physiological perturbations anchored in Surviving Sepsis Campaign (SSC) guidelines. We audit a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), the latter employing Causal Action Shielding, propensity-based importance weighting, and Conservative Q-Learning. CCA reveals that MedDT paradoxically reduces vasopressor dosage as lactate escalates, contradicting resuscitation guidelines, while HCT-RL maintains physiologically consistent responses. These findings expose a systemic misalignment between statistical fit and clinical safety, supporting counterfactual audits as a necessary evaluation standard for medical RL.
Hangqi Ren, Junyi Liao
School of Engineering, Vanderbilt University, Nashville, TN, USA · Pratt School of Engineering, Duke University, Durham, NC, USA