cs.LGMar 11, 2026

Ensuring Safety in Automated Mechanical Ventilation through Offline Reinforcement Learning and Digital Twin Verification

Authors: Hang Yu, Huidong Liu, Qingchen Zhang, William Joy, Kateryna Nikulina, Andreas A. Schuppert, Sina Saffaran, Declan Bates

Organizations: School of Engineering, University of Warwick, Coventry, UK · School of Computer Science, Chongqing University, China · School of Computer Science and Technology, Hainan University, China · Institute for Computational Biomedicine, RWTH Aachen University, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mechanical ventilation (MV) is a life-saving intervention for patients with acute respiratory failure (ARF) in the ICU. However, inappropriate ventilator settings could cause ventilator-induced lung injury (VILI). Also, clinicians workload is shown to be directly linked to patient outcomes. Hence, MV should be personalized and automated to improve patient outcomes. Previous attempts to incorporate personalization and automation in MV include traditional supervised learning and offline reinforcement learning (RL) approaches, which often neglect temporal dependencies and rely excessively on mortality-based rewards. As a result, early stage physiological deterioration and the risk of VILI are not adequately captured. To address these limitations, we propose Transformer-based Conservative Q-Learning (T-CQL), a novel offline RL framework that integrates a Transformer encoder for effective temporal modeling of patient dynamics, conservative adaptive regularization based on uncertainty quantification to ensure safety, and consistency regularization for robust decision-making. We build a clinically informed reward function that incorporates indicators of VILI and a score for severity of patients illness. Also, previous work predominantly uses Fitted Q-Evaluation (FQE) for RL policy evaluation on static offline data, which is less responsive to dynamic environmental changes and susceptible to distribution shifts. To overcome these evaluation limitations, interactive digital twins of ARF patients were used for online "at the bedside" evaluation. Our results demonstrate that T-CQL consistently outperforms existing state-of-the-art offline RL methodologies, providing safer and more effective ventilatory adjustments. Our framework demonstrates the potential of Transformer-based models combined with conservative RL strategies as a decision support tool in critical care.

Figures & tables

Figure 1: A simplified, diagrammatic representation of the digital twin cardiopulmonary system.
Figure 2: Overview of the proposed method. (A) Preparation of the structured offline dataset. (B) The training procedure of our proposed T-CQL method, which integrates a Transformer encoder for effective temporal modeling of patient dynamics, state-dependent conservative adaptive regularization based on uncertainty quantification to ensure safety, and consistency regularization for robust decision-making. (C) Evaluation methods: off-policy FQE and online interactive digital twin evaluation.
 S1:L=(st−L+1,…,st),S′1:L=(st−L+2,…,st+1)\mathbf{S}^{1:L}=\bigl(s_{t-L+1},\dots,s_{t}\bigr),\quad\mathbf{S}^{\prime}{}^{1:L}=\bigl(s_{t-L+2},\dots,s_{t+1}\bigr) 
Algorithm 1 Training Procedure for Transformer‐based CQL (T‐CQL)
Figure 3: Comparison of average initial Q-values for in-distribution (ID) versus out-of-distribution (OOD) cases. Error bars denote the variance, and the horizontal line marks the maximum achievable expected return.
ModelFQE ScoreMortality Score
Clinician0.46±0.010.46\pm 0.010.27±0.050.27\pm 0.05
BCQ0.63±0.090.63\pm 0.090.24±0.050.24\pm 0.05
Deepvent (CQL)0.79±0.130.79\pm 0.130.19±0.030.19\pm 0.03
T-CQL(This study)0.87±0.050.87\pm 0.050.16±0.020.16\pm 0.02
Table 1: Comparison of expected return and estimated mortality score for various policies. The values are presented as means ±\pm standard deviations across five runs. Higher FQE scores reflect greater expected cumulative return under the policy (i.e. better long‐term patient outcome trajectories), whereas lower estimated mortality rates correspond to higher predicted survival likelihood
ModelPearson Correlation Coefficient
BCQ−0.30±0.07-0.30\pm 0.07
Deepvent (CQL)−0.38±0.03-0.38\pm 0.03
T-CQL(This study)−0.49±0.04-0.49\pm 0.04
Table 2: The Pearson correlation coefficients between the expected return and estimated mortality (p << 0.0001).
Figure 4: Distribution of ventilator settings chosen by different policies on the MIMIC test set. ”Clinicians” refer to the actions taken by clinicians in the original MIMIC dataset.
ModelSafety Constraints Rate (%)Reduced DP Rate (%)
Clinician43.88±0.1543.88\pm 0.1537.76±0.0237.76\pm 0.02
DDQN22.45±2.5822.45\pm 2.5820.41±1.3620.41\pm 1.36
BCQ41.83±1.4741.83\pm 1.4740.82±0.9540.82\pm 0.95
Deepvent (CQL)34.69±1.6334.69\pm 1.6333.67±1.0533.67\pm 1.05
T-CQL(This study)47.96±1.1547.96\pm 1.1544.90±0.9244.90\pm 0.92
Table 3: Compliance rates for safety constraints, and reduction in driving pressure (DP), for each policy. Values are presented as means ± standard deviations across five runs. Safety constraints compliance rate indicates the proportion of times all safety constraints were met. The Reduced DP rate indicates the proportion of times the driving pressure was reduced compared to the baseline value (initial state).
Figure 5: Distribution of ventilator settings selected by different policies across digital twins of real patients. ”Physician” refers to clinician-derived actions from the MIMIC dataset, while ”RWTH Intensivists” represent ventilator settings applied by clinicians at the University Hospital of Aachen.
More in this topic

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Explore similar work

CardsList
Jun 3, 2026

VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis. However, current data-driven methods, especially those imitating retrospective Electronic Health Records (EHR), often suffer from imitation bias. They may capture superficial correlations from inconsistent clinical demonstrations, such as associating passive ventilator settings with survival because such settings are common in stable patients, and thus fail to generalize to volatile or out-of-distribution phenotypes. Standard Reinforcement Learning (RL) methods also struggle with the adversarial trade-offs of critical care and often produce opaque policies with limited clinical interpretability. To address these limitations, we introduce VentAgent, a hierarchical framework in which Large Language Models (LLMs) act as transparent arbitrators for mechanical ventilation. We reformulate ventilation control as a dynamic Multi-Objective Arbitration process rather than single-objective optimization. VentAgent decomposes decision-making into three interpretable stages: Perception, Planning, and Orchestration. By leveraging the semantic reasoning capabilities of LLMs, it synthesizes strategies from heterogeneous experts and resolves conflicting clinical priorities through an explicit coordination mechanism. Evaluations on a high-fidelity physiological simulator show that VentAgent outperforms state-of-the-art RL and classical control baselines. Moreover, it converts control decisions into human-readable reasoning chains, offering a safer, more interpretable, and adaptable paradigm for critical care automation.

Teqi Hao, Yuxuan Fu, Xiaoyu Tan +4

Tencent Youtu Lab Shanghai, Shanghai, China · Artificial Intelligence Innovation and Incubation Institute, Fudan University Shanghai, Shanghai, China · School of Electronic and Electrical Engineering, Shanghai University of Engineering Science Shanghai, Shanghai, China

Language Model-Based ControlFeedback Control
AskPDF
Aug 11, 2026

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE) assess only behavioral imitation and cannot detect Toxic Mimicry, a failure mode in which agents replicate harmful patterns such as treatment withdrawal during comfort-care transitions. Using the MIMIC-III database, we propose the Counterfactual Clinical Audit (CCA) framework, which stress-tests RL agents through physiological perturbations anchored in Surviving Sepsis Campaign (SSC) guidelines. We audit a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), the latter employing Causal Action Shielding, propensity-based importance weighting, and Conservative Q-Learning. CCA reveals that MedDT paradoxically reduces vasopressor dosage as lactate escalates, contradicting resuscitation guidelines, while HCT-RL maintains physiologically consistent responses. These findings expose a systemic misalignment between statistical fit and clinical safety, supporting counterfactual audits as a necessary evaluation standard for medical RL.

Hangqi Ren, Junyi Liao

School of Engineering, Vanderbilt University, Nashville, TN, USA · Pratt School of Engineering, Duke University, Durham, NC, USA

Counterfactual EvaluationOffline RLSafe RL
AskPDF
Aug 14, 2026

Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. While reinforcement learning (RL) offers a promising framework for optimizing these sequential decisions, standard approaches rely primarily on structured electronic health record (EHR) data, missing crucial clinical context recorded in free-text notes. Integrating longitudinal clinical notes into RL state spaces is challenging because notes are heavily inflated by temporal redundancy, such as copy-forward text, templating, and repetitive documentation, which dilutes time-local updates and degrades state representation quality. To address this, we propose a redundancy-aware multimodal state representation framework that explicitly removes duplicated note text over time before policy learning. We evaluate two computationally efficient temporal decomposition strategies for removing duplicated note text: (1) an embedding-space decomposition using singular value decomposition on local history subspaces, and (2) an interpretable sentence-level diff operation that filters out previously documented sentences before text encoding. Using real-world ICU data, we demonstrate that state representations constructed by stripping temporal note redundancy significantly outperform both structured-only and raw-note baselines across multiple off-policy evaluation methods (Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation). Our findings show that explicitly isolating new clinical information from repeated note text yields higher-quality state representations and directly improves RL performance for clinical decision support.

Chenran Weng, Joo Seung Lee, Malini Mahendra +1

University of California, Berkeley Berkeley, CA, USA · University of California, San Francisco San Francisco, CA, USA

Reinforcement LearningRepresentation Learning for RLHealthcareLongitudinal EHR ModelingOffline RL
AskPDF