ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

Yanyan Wu; Guanyu Zhu; Changbin Zhao; Jin Zhang; Dexu Yu; Tianmin Sun; Jining Luan; Chunxiao Li

cs.LGOct 18, 2026

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

Authors: Yanyan Wu, Guanyu Zhu, Changbin Zhao, Jin Zhang, Dexu Yu, Tianmin Sun, Jining Luan, Chunxiao Li,

Organizations: South China Agricultural University, Guangzhou, China · The Ohio State University, Columbus, Ohio, United States

Abstract

Offline sepsis treatment aims to learn treatment policies from retrospective ICU trajectories. Decision Transformer-based methods provide a natural sequence modeling framework for this task, but they mainly rely on raw physiological states and historical clinician actions. Such raw trajectory tokens may not explicitly capture clinical semantics that clinicians use in practice, such as patient severity, recent treatment response, and current risk priorities. To address this limitation, we propose ClinSDT, an LLM-assisted Decision Transformer framework for offline sepsis treatment. ClinSDT uses a frozen LLM as a semantic bridge rather than a treatment decision maker. It constructs three clinical semantic views from ICU trajectories, namely Profile, Context, and Advisory, and inserts their encoded representations as independent tokens into a Decision Transformer policy. To support policy refinement and rollout, ClinSDT further trains a Treatment-Response Forecaster to estimate the next physiological state, SAPS-II change, and predictive uncertainty from state-action inputs. In the second training stage, the forecaster screens local counterfactual treatment candidates, and only low-uncertainty candidates with lower predicted SAPS-II increase are distilled into the policy after semantic regeneration. During inference, ClinSDT performs closed-loop treatment rollout by refreshing semantic inputs from the currently available patient state. Experiments on MIMIC-III and MIMIC-IV show that ClinSDT achieves consistent relative improvements over representative imitation learning, offline reinforcement learning, and Decision Transformer baselines under the same rollout protocol. Code and data are available at https://github.com/KAG778/ClinSDT.

Figures & tables

Figure 1. Motivation of ClinSDT. Raw ICU trajectories leave clinical decision semantics implicit in numerical states and actions. ClinSDT uses a frozen LLM to encode structured clinical context, while a Decision Transformer remains responsible for treatment action generation. Motivation diagram of ClinSDT showing that raw ICU trajectories contain implicit clinical semantics, which are encoded by a frozen LLM as structured context for a Decision Transformer policy.

Figure 2. Overview of ClinSDT. ClinSDT encodes Profile , Context , and Advisory semantics from ICU trajectories with a frozen LLM and injects them as independent tokens into a Decision Transformer policy. Training consists of Stage I semantic-token policy and response-forecaster learning, followed by Stage II uncertainty-constrained counterfactual semantic refinement. During closed-loop rollout, the policy refreshes semantic inputs from each predicted patient state while retaining the retrospectively initialized target-return condition. The LLM provides semantic representations only; treatment actions are generated by the policy model. In the schematic,

\Omega\equiv F_{\phi}

and

\mathcal{L}_{s}\equiv\mathcal{L}_{\mathrm{fr}}^{\mathrm{fact}}

. Overview of ClinSDT, showing how Profile, Context, and Advisory semantics are encoded by a frozen LLM, injected as independent tokens into a Decision Transformer policy, and used with treatment-response forecasting and uncertainty-constrained conservative counterfactual distillation for model-based treatment rollout.

Dataset	Model	5-step rollout				10-step rollout
Dataset	Model	HIGH	MID	LOW	ALL	HIGH	MID	LOW	ALL
MIMIC-III	BC	3.24	-1.51	-5.59	-0.59	5.39	-1.55	-7.80	-0.22
	BCQ	3.17	-1.30	-3.88	-0.35	5.99	-1.17	-5.59	0.34
	DQN	6.42	-1.05	-5.03	0.57	13.17	0.10	-6.74	2.94
	DT	7.01	-1.57	-5.93	0.30	12.96	-1.40	-8.98	1.72
	IQL	2.86	-2.13	-6.13	-1.14	4.39	-2.58	-9.25	-1.28

Table 1. Overall model-based rollout results on MIMIC-III and MIMIC-IV. We compare ClinSDT with representative baselines from imitation learning, offline RL, value-based RL, and Decision Transformer-style policy learning. The metric is negative cumulative forecaster-predicted SAPS-II change. All methods follow the same rollout procedure, while each uses its own learned forecaster; the values are therefore protocol-relative point estimates rather than scores on a shared independent scale. Bold and underlining indicate only the numerical ordering within the table.

Figure 3. Overall model-based rollout results on MIMIC-III and MIMIC-IV. ClinSDT has the highest ALL-patient point estimate in all four reported dataset–horizon settings under the implemented rollout protocol. A bar chart showing method-specific rollout estimates for ClinSDT and the baseline methods on MIMIC-III and MIMIC-IV under 5-step and 10-step settings. The chart summarizes negative cumulative forecaster-predicted SAPS-II changes for all patients.

Group	Variant	5-step rollout				10-step rollout
Group	Variant	HIGH	MID	LOW	ALL	HIGH	MID	LOW	ALL
Removal	w/o Profile	3.14	-0.54	-4.96	0.09	2.52	0.30	-0.03	0.84
	w/o Context	5.66	-0.43	-4.97	0.80	5.92	-0.14	-0.68	1.37
	w/o Advisory	5.97	-1.30	-7.94	0.09	4.26	0.05	-0.28	1.10
	w/o Semantics	7.06	-2.48	-9.69	-0.56	4.82	0.41	0.39	1.53
Single-view	only Profile	8.45	-0.79	-3.67	1.36	1.68	-0.94	-0.66	-0.25

Table 2. Semantic-view variants on MIMIC-III under 5-step and 10-step rollouts. Results are method-specific negative cumulative forecaster-predicted SAPS-II changes. Bold and underlining indicate only the highest and second-highest numerical point estimates.

Variant	Design	HIGH	MID	LOW	ALL
Mean	zero-param pooling	0.49	-0.24	-1.78	-0.16
Concat	one projection	0.62	-0.35	-7.43	-0.60
Gated	two projections	4.08	-1.19	-21.50	-1.28
Residual	residual projection	-0.02	-1.34	-6.78	-1.38
Cross-attn	MHA + LN	-0.94	-1.48	-5.89	-1.65
Ours	6-token block (3 semantic)	17.74	5.16	10.77	8.75

Table 3. Comparison of token-fusion designs on MIMIC-III under the 10-step rollout. Higher values indicate a more favorable estimate under the corresponding learned forecaster.

Figure 4. Token-fusion variants on MIMIC-III. The separated-token design has the highest $10$ -step point estimate among the complete fusion configurations. A bar chart comparing different semantic token-fusion designs on MIMIC-III under the 10-step rollout. The separated-token design has the highest numerical estimate across the reported groups under its learned forecaster.

Figure 5. Stage I-to-Stage II change in the method-relative rollout estimate on MIMIC-III. The y-axis is defined as the negative cumulative SAPS-II change predicted by each method’s own forecaster; higher is more favorable under that forecaster. Line plot showing the Stage I-to-Stage II change in method-relative rollout estimates on MIMIC-III. The y-axis is the negative cumulative SAPS-II change predicted by each method's own forecaster, so higher values are more favorable under that forecaster.

Model	5-step estimate change				10-step estimate change
Model	HIGH	MID	LOW	ALL	HIGH	MID	LOW	ALL
DQN	$2.93$	$0.45$	$0.28$	$1.06$	$6.76$	$1.41$	$1.07$	$2.75$
DT	$3.79$	$0.76$	$1.28$	$1.56$	$7.03$	$1.29$	$1.86$	$2.79$
BCQ	$1.21$	$1.38$	$1.71$	$1.36$	$2.70$	$2.15$	$2.58$	$2.32$
BC	$0.08$	$0.49$	$0.46$	$0.39$	$0.79$	$0.86$	$0.62$	$0.83$
IQL	$0.11$	$0.43$	$0.24$	$0.33$	$0.74$	$0.87$	$0.68$	$0.83$

Table 4. Stage I-to-Stage II change in each method’s forecaster-based estimate on MIMIC-III.

Stratum	Profile	Context	Advisory	Sem.	RTG	State
HIGH	$+0.0031$	$-0.0016$	$+0.0014$	$+0.0029$	$+0.0013$	$-0.0043$
MID	$-0.0012$	$+0.0007$	$-0.0005$	$-0.0010$	$-0.0007$	$+0.0016$
LOW	$-0.0021$	$-0.0003$	$-0.0018$	$-0.0042$	$+0.0016$	$+0.0025$

Table 5. Stratum-wise relative attention scores of the Stage II policy on MIMIC-III. Positive values indicate higher relative attention to the corresponding token group, while negative values indicate lower relative attention.

Figure 6. Stratum-wise relative attention heatmap of the Stage II policy on MIMIC-III. Warmer and cooler colors indicate higher and lower relative attention, respectively. Attention heatmap showing stratum-wise relative attention of the Stage II policy on MIMIC-III across Profile, Context, Advisory, semantic, RTG, and State token groups.

Group	n	Top1	$\sigma_{\text{beh}}$	$\sigma_{\text{pred}}$	OOD%	KL
High	8362	0.458	0.910	0.907	4.87	0.662
Mid	18646	0.483	0.943	0.942	4.93	0.712
Low	1914	0.399	1.010	1.012	5.17	0.883
Overall	28922	0.470	0.938	0.937	4.84	0.685

Table 6. Detailed behavior-support statistics of the Stage II policy on MIMIC-III.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Implementation
Cohort	Adult Sepsis-3 trajectories following the AI Clinician pipeline.
Temporal unit	Non-overlapping 4-hour bins.
State	45 variables covering demographics, vital signs, laboratory tests, fluid input/output, and ventilation.
Missing values	Last observation carried forward, followed by population-median imputation.
Normalization	Z-normalization using training-set statistics.
Action space	$5\times 5$ discretization of intravenous fluid and vasopressor doses.

Appendix

Table 7. Preprocessing and trajectory construction.

Item	MIMIC-III	MIMIC-IV
Train trajectories	13,011	52,380
Validation trajectories	2,781	11,225
Test trajectories	2,786	11,224
10-step eligible test trajectories	1,966	8,153
Trajectory length	2–20, mean 13.4	6–768, mean 26.9

Appendix

Table 8. Dataset statistics and rollout eligibility. A trajectory is 10-step eligible when it contains at least 11 observations.

Item	MIMIC-III	MIMIC-IV
Initial SAPS-II mean	81.8	53.5
Initial SAPS-II range	65.5–91.5	37–86
Initial SAPS-II q10/q50/q90	79.5 / 80.5 / 91.5	44 / 53 / 64
Low threshold	$<75$	$<50$
Mid threshold	$[75,85)$	$[50,57)$
High threshold	$\geq 85$	$\geq 57$

Appendix

Table 9. Initial SAPS-II distribution and acuity stratification. Counts are computed on the 10-step eligible test subset.

Component	Available input	Use
Profile	Factual / Stage II: $(s_{t},m_{t},R_{t})$ ; rollout $1+$ : $\hat{s}_{k}$	Global condition
Context	Factual or generated incoming transition	Treatment response
Advisory	Factual: $(s_{t},m_{t})$ ; predicted state: $\hat{s}$	Current risk priorities
Acuity-enriched	Factual views; Stage II Profile ; rollout step 0	Recorded information only
Predicted-state	Stage II Advisory ; rollout Profile / Advisory	No recorded acuity

Appendix

Table 10. Semantic inputs and leakage-control regimes.

View	Example description
Profile	Severe sepsis with cardiovascular and renal involvement; mechanically ventilated; high acuity.
Context	After vasopressor titration, blood pressure is stable, lactate is decreasing, and perfusion is improving.
Advisory	Persistent hyperlactatemia indicates high hypoperfusion risk; prioritize cardiovascular protection.

Appendix

Table 11. Representative semantic descriptions produced by the templates.

Item	Value
Policy layers / heads / hidden size	$4$ / $8$ / $128$
Context / tokens / block size	$20$ / $6$ / $120$
LLM embedding dimension	$896$ per view
Forecaster type / hidden size	3-layer MLP / $256$
MC Dropout rate / samples	$0.2$ / $10$
Policy / forecaster learning rate	$6\times 10^{-4}$ / $10^{-3}$

Appendix

Table 12. Model architecture and main hyperparameters. The same settings are used for MIMIC-III and MIMIC-IV unless otherwise stated.

Stage	Protocol
Stage I policy	Behavior cloning on documented clinician actions.
Stage I forecaster	Factual state–action next-state and SAPS-II supervision.
Candidate set	Radius- $2$ grid neighborhood plus the policy proposal.
Screening	Require uncertainty below $2.0$ and lower predicted one-step SAPS-II change than the documented action.
Distillation	Regenerate next-step hindsight semantics and apply MSE-to-one-hot on accepted actions.
Stage II forecaster	Retain factual anchoring and confidence-gated bootstrap regularization; no BC or KL term.

Appendix

Table 13. Compact training and evaluation protocol.

Phase	Available information
Stage I semantics	Factual current/incoming transition and, where specified, recorded acuity.
Stage II screening	Current factual state, documented action, local candidates, and forecaster predictions.
Stage II regeneration	Current factual state, accepted candidate, and its predicted counterfactual next state; training only.
Rollout step 0	Held-out initial state and recorded initial information; no subsequent observation.
Rollout steps $1+$	Previously generated state and incoming generated transition; state-only semantic regime.
Scoring	Predicted SAPS-II changes from the corresponding method-specific updated forecaster.

Appendix

Table 14. Information available across training and evaluation phases.

Healthcare

Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.

See all 608 papers

Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model
May 14, 2026Minghao Wu, Yuting Yan, Zhenyang Cai +9Agentic RL Clinical Decision Support
The Chinese University of Hong Kong, Shenzhen · Beijing Hospital
Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
Aug 17, 2026Marc Pérez-Roig, David Fernández-Narro, Carlos SáezOffline RL Clinical Decision-Making
Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, València, València, Spain
Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits
Aug 11, 2026Hangqi Ren, Junyi LiaoCounterfactual Evaluation Offline RL
School of Engineering, Vanderbilt University, Nashville, TN, USA · Pratt School of Engineering, Duke University, Durham, NC, USA

May 14, 2026J/K move · Enter open · S save

Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model

Minghao Wu, Yuting Yan, Zhenyang Cai, Ke Ji+8

The Chinese University of Hong Kong, Shenzhen · Beijing Hospital

Agentic RL Clinical Decision Support LLM Agent Training Medical LLMs

Open paper AskPDF arXiv

Sepsis management in the ICU requires sequential treatment decisions under rapidly evolving patient physiology. Although large language models (LLMs) encode broad clinical knowledge and can reason over guidelines, they are not inherently grounded in action-conditioned patient dynamics. We introduce SepsisAgent, a world model-augmented LLM agent for sepsis treatment recommendation. SepsisAgent uses a learned Clinical World Model to simulate patient responses under candidate fluid--vasopressor interventions, and follows a propose--simulate--refine workflow before committing to a prescription. We first show that world-model access alone yields inconsistent LLM decision performance, motivating agent-specific training. We then train SepsisAgent through a three-stage curriculum: patient-dynamics supervised fine-tuning, propose--simulate--refine behavior cloning, and world-model-based agentic reinforcement learning. On MIMIC-IV sepsis trajectories, SepsisAgent outperforms all traditional RL and LLM-based baselines in off-policy value while achieving the best safety profile under guideline adherence and unsafe-action metrics. Further analysis shows that repeated interaction with the Clinical World Model enables the agent to learn regularities in patient evolution, which remain useful even when simulator access is removed.

Abstract

Healthcare

Explore similar work

Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model

Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits