Advances in electronic health records (EHR) and large-scale critical care databases have enabled the development of machine learning models for the early detection of patient deterioration. While these models often achieve strong predictive performance for outcomes such as sepsis and mortality, most studies evaluate them purely on discrimination metrics. This leaves a critical gap: how to systematically translate probabilistic clinical risk estimates into downstream operational decisions under real-world constraints, such as shift rigidity and hiring delays. This study proposes an uncertainty-aware anticipatory staffing framework that bridges short-term patient risk prediction and staff planning without altering the upstream predictive model. Instead of relying on naive deterministic aggregation or fixed safety buffers, the framework explicitly models dynamic predictive uncertainty and evaluates extreme shortage exposure using Conditional Value-at-Risk (CVaR). These aggregated, risk-calibrated demand signals are then embedded into an hourly rolling-horizon scheduling policy to generate delay-constrained staffing decisions. Evaluated on realistic trajectories from the MIMIC-IV database, the proposed framework demonstrates a favorable cost-shortage trade-off compared to deterministic and purely reactive baselines. Pareto frontier analysis illustrates how uncertainty-aware demand construction changes resource allocation under varying safety preferences, while stress testing confirms the framework's operational robustness even when subjected to systematic predictive bias in the form of calibration drift. By formalizing this translation layer, this work provides a principled, transparent pathway for safely integrating probabilistic AI outputs into high-stakes healthcare operational systems.
Figures & tables
Figure 1: Overview of the uncertainty-aware risk aggregation framework
Model
AUROC
AUPRC
Cost (Mean)
Shortage (Mean)
Logistic Regression
0.8605
0.6809
23.4
15.0
LSTM
0.8419
0.6612
23.4
16.6
Transformer
0.8646
0.6890
23.1
16.2
Table 1: Backbone sensitivity analysis
Method
Cost
Shortage
Overstaff
Volatility
CVaR (90%)
Deterministic
19.9 ± 3.2
22.9 ± 13.9
26.4 ± 23.1
8.7 ± 1.7
6.49
Reactive
33.9 ± 3.1
14.4 ± 4.5
120.6 ± 32.9
13.7 ± 1.4
0.60
Variance-only
27.2 ± 3.4
14.8 ± 5.0
72.4 ± 32.4
11.1 ± 1.5
0.62
Proposed (Full)
23.4 ± 3.3
16.6 ± 7.4
46.3 ± 29.8
9.8 ± 1.4
4.44
Oracle
19.3 ± 5.8
0.0 ± 0.0
10.6 ± 5.8
2.4 ± 1.6
0.00
Table 2: Performance comparison across staffing policies
Ablation Variant
Cost (Mean)
Shortage (Mean)
P90 Shortage
P95 Shortage
1. Mean-only ( μ )
19.9
22.9
45.1
46.5
2. Static Margin ( μ+σstat )
26.9
14.7
21.2
23.5
3. Dynamic Margin ( μ+σdyn )
22.9
16.9
29.1
30.0
4. Full Framework ( μ+σdyn )
23.4
16.6
28.3
31.5
Table 3: Mechanism ablation and tail-risk analysis
Figure 5
Scenario
Cost (Mean)
Shortage (Mean)
P90 Shortage
Original
23.4
16.6
28.3
Overestimation (1.3x)
28.9
14.5
21.1
Underestimation (0.7x)
17.3
30.3
59.3
Noisy (+N)
23.7
16.4
28.2
Table 4: Systematic bias robustness stress test
Figure 4: Impact of systematic probability drift on shortage risk
Figure 5: 24-hour micro-level decision dynamics under MIMIC-IV-derived ICU demand
Safe predictions are a crucial requirement for integrating predictive models into clinical decision support systems. One approach to improving trustworthiness is to enable models to express uncertainty about individual predictions. However, current machine learning models frequently lack reliable uncertainty estimation, hindering real-world deployment. This limitation is particularly evident in multimodal settings, where models must effectively integrate heterogeneous information. In this work, we propose MedCertAIn, a predictive uncertainty framework that leverages multimodal clinical data to improve model performance and reliability for in-hospital mortality risk prediction as an indicator of patient deterioration. We design data-driven priors over neural network parameters using a hybrid strategy that considers cross-modal similarity in self-supervised latent representations and modality-specific data corruptions. We train and evaluate the models with such priors using clinical time-series and chest X-ray images from the publicly available datasets MIMIC-IV and MIMIC-CXR. Our results show that MedCertAIn achieves competitive predictive performance and substantial gains in selective prediction compared with the evaluated deterministic and stochastic baselines. These findings highlight the promise of data-driven priors in advancing robust, uncertainty-aware AI tools for high-stakes clinical applications. Our implementation is publicly available at: https://github.com/jlaitue/medcertain.
L. Julián Lechuga López, Tim G. J. Rudner, Farah E. Shamout
Tandon School of Engineering, New York University · New York University Abu Dhabi · University of Toronto
Clinical language models (LMs) are increasingly applied to support clinical risk prediction from free-text notes, yet their uncertainty estimates often remain poorly calibrated and clinically unreliable. In this work, we propose Clinical Uncertainty Risk Alignment (CURA), a framework that aligns clinical LM-based risk estimates and uncertainty with both individual error likelihoods and cohort-level ambiguities. CURA first fine-tunes domain-specific clinical LMs to obtain task-adapted patient embeddings, and then performs uncertainty fine-tuning of a multi-head classifier using a bi-level uncertainty objective. Specifically, an individual-level calibration term aligns predictive uncertainty with each patient's likelihood of error, while a cohort-aware regularizer pulls risk estimates toward event rates in their local neighborhoods in the embedding space and places extra weight on ambiguous cohorts near the decision boundary. We further show that this cohort-aware term can be interpreted as a cross-entropy loss with neighborhood-informed soft labels, providing a label-smoothing view of our method. Extensive experiments on MIMIC-IV clinical risk prediction tasks across various clinical LMs show that CURA consistently improves calibration metrics without substantially compromising discrimination. Further analysis illustrates that CURA reduces overconfident false reassurance and yields more trustworthy uncertainty estimates for downstream clinical decision support.
Forecasting evolving clinical risks relies on intrinsic pathological dependencies rather than mere chronological proximity, yet current methods struggle with coarse binary supervision and physical timestamps. To align predictive modeling with clinical logic, we propose the Medical-semantics Aware Time-ALiBi Transformer (MATA-Former), utilizing event semantics to dynamically parameterize attention weights to prioritize causal validity over time lags. Furthermore, we introduce Plateau-Gaussian Soft Labeling (PSL), reformulating binary classification into continuous multi-horizon regression for full-trajectory risk modeling. Evaluated on SIICU -- a newly constructed dataset featuring over 506k events with rigorous expert-verified, fine-grained annotations -- and the MIMIC-IV dataset, our framework demonstrates superior efficacy and robust generalization in capturing risks from text-intensive, irregular clinical time series.
Zhichong Zheng, Xiaohang Nie, Xueqi Wang +3
Tongji University · Shanghai Innovation Institute · Harbin Institute of Technology +1