Ensemble Machine Learning and Statistical Procedures for Dynamic Predictions of Time-to-Event Outcomes
Authors: Nina van Gerwen, Sten Willemsen, Bettina E. Hansen, Christophe Corpechot, Marco Carbone, Cynthia Levy, Maria-Carlota Londõno, Atsushi Tanaka, Palak Trivedi, Alejandra Villamil, Gideon Hirschfield, Dimitris Rizopoulos
Organizations: Department of Biostatistics, Erasmus University Medical Center, the Netherlands · Department of Epidemiology, Erasmus University Medical Center, the Netherlands · Reference Center for Inflammatory Biliary Diseases and Autoimmune Hepatitis, European Reference Network on Hepatological Diseases (ERN Rare-Liver), Saint-Antoine Hospital, Assistance Publique - Hˆopitaux de Paris; Inserm UMR S938, Saint-Antoine Research Center, Sorbonne University, Paris, France · Department of Medicine and Surgery, University of Milan Bicocca, Italy · Miller School of Medicine, University of Miami, United States of America · Viral, Toxic and Metabolic Hepatopathies, Hospital Cl´ınic de Barcelona, Spain · Department of Medicine, Teikyo University School of Medicine, Japan · Biomedical Research Unit & Centre for Liver Research, University of Birmingham, United Kingdom · Division of Gastroenterology & Hepatology, Hospital Italiano de Buenos Aires, Argentina · Centre for Liver Disease, Toronto General Hospital, Canada
Dynamic predictions for longitudinal and time-to-event outcomes have become a versatile tool in precision medicine. Our work is motivated by the application of dynamic predictions in the decision-making process for primary biliary cholangitis patients. For these patients, serial biomarker measurements (e.g., bilirubin and alkaline phosphatase levels) are routinely collected to inform treating physicians of the risk of liver failure and guide clinical decision-making. Two popular statistical approaches to derive dynamic predictions are joint modelling and landmarking. However, recently, machine learning techniques have also been proposed. Each approach has its merits, and no single method exists to outperform all others. Consequently, obtaining the best possible survival estimates is challenging. Therefore, we extend the Super Learner framework to combine dynamic predictions from different models and procedures. Super Learner is an ensemble learning technique that allows users to combine different prediction algorithms to improve predictive accuracy and flexibility. It uses cross-validation and different objective functions of performance (e.g., squared loss) that suit specific applications to build the optimally weighted combination of predictions from a library of candidate algorithms. In our work, we pay special attention to appropriate objective functions for Super Learner to obtain the most optimal weighted combination of dynamic predictions. In our primary biliary cholangitis application, Super Learner presented unique benefits due to its ability to flexibly combine outputs from a diverse set of models with varying assumptions for equal or better predictive performance than any model fit separately.
Figures & tables
Loss function
Model
BS
IBS
tv -AUC
1
0.00
0.00
0.00
2
0.00
0.00
0.07
3
0.00
0.00
0.02
4
0.47
0.23
0.15
5
0.00
0.00
0.00
Table 1: Super Learner weight estimates at prediction window {6, 9} (left) and {9, 12} (right) under the three different loss functions described in Section 2.1 for predicting the composite endpoint in the Global PBC Data. The boldfaced numbers denote the discrete SL for a given prediction window and loss function. Models M1 to M9 are denoted by 1 to 9.
Figure 1: Metric estimates of the ensemble SL (eSL) and discrete SL (dSL) for predicting the composite endpoint in the Global PBC Data. Models M1 to M9 are denoted by 1 to 9 . The blue dashed line serves as reference point for the performance of the eSL. The textboxes in each graph denote the performance of a Kaplan-Meier (KM) model that contained no covariates.
Figure 2: Metric estimates of the ensemble SL (eSL) and discrete SL (dSL) and oracle model (OM) under no censoring (Scenario 1), random censoring (Scenario 2) and informative censoring (Scenario 3) over 100 simulated datasets. The box plots under Train Error denote the metric values in the dataset the model was trained on, the box plots under Test Error show the metric values in the holdout data set.
Scenario 1
Scenario 2
Scenario 3
Model
BS
IBS
tv -AUC
BS
IBS
tv -AUC
BS
IBS
tv -AUC
1
0.30 (0.32)
0.47 (0.28)
0.34 (0.20)
0.23 (0.29)
0.35 (0.31)
0.37 (0.23)
0.31 (0.33)
0.56 (0.30)
0.36 (0.24)
2
0.00 (0.00)
0.04 (0.05)
0.01 (0.04)
0.01 (0.02)
0.04 (0.06)
0.01 (0.05)
0.00 (0.01)
0.02 (0.04)
0.01 (0.05)
3
0.79 (0.32)
0.50 (0.28)
0.64 (0.20)
0.76 (0.29)
0.61 (0.32)
0.62 (0.23)
0.69 (0.33)
0.42 (0.29)
0.63 (0.23)
Table 2: Average model weight and its standard deviation in brackets under no censoring (Scenario 1), random censoring (Scenario 2) and informative censoring (Scenario 3). Model 1 is the true one-stage landmark model, model 2 is a two-stage landmark model and model 3 is the true joint model.
Details
Base learner
Package
Version
Hyperparameters
Landmark: Standard Cox
survival
3.6-4
-
Landmark: Boosted Cox
mboost
2.9-10
bstop , ν , bopt
Joint Model
JMbayes2
0.5-0
priors, MCMC samples, chains, burn in
Table 3: Details of the implementation of the different type of base learners and their hyperparameters.
Details
Functional forms
Base learner
Type
Algorithm
Longitudinal model
TB
ALP
AST
M1
One-stage LM
Standard Cox
-
-
-
-
M2
One-stage LM
Boosted Cox
-
-
-
-
M3
Two-stage LM
Standard Cox
linear
-
-
-
M4
Two-stage LM
Boosted Cox
linear
-
-
-
M5
Two-stage LM
Standard Cox
splines (df = 3)
-
-
-
Table 4: Summary table for the 9 base learners included in the Super Learner ensemble. The column Type details the type of base learner, Algorithm defines how the base learner is estimated, Longitudinal model details the time specification for the base learner, and Functional form shows the association between per longitudinal outcome and the hazard function. A dash implies non-applicability. The abbreviations are LM = landmark; JM = joint model; df = degrees of freedom; CV = current value; S = slope; A = area.
The Super Learner (SL) is a widely used ensemble method that combines point predictions from a library of learners based on their predictive performance. Interval predictions are of considerable practical interest because they allow uncertainty in predictions produced by an individual learner or an ensemble to be quantified. Several methods have been proposed for constructing interval predictions based on the SL, however, these approaches are typically justified using asymptotic arguments or rely on computationally intensive procedures such as the bootstrap. Conformal prediction (CP) is a machine learning framework for constructing prediction intervals with finite-sample and asymptotic coverage guarantees under mild conditions. We propose coupling CP with the SL through a natural construction that mirrors the original SL framework, using individual learner weights and combining learner-specific conformity scores via a weighted majority vote. We characterize the properties of the resulting SL-based prediction intervals for continuous outcomes. We cover settings under exchangeability, potential violations of exchangeability, and data-generating mechanisms exhibiting heteroscedasticity, sparsity, and other forms of distributional heterogeneity. A comprehensive simulation study shows that the conformalized SL achieves valid finite-sample coverage with competitive performance relative to the true data-generating mechanism. A central contribution of this work is an application to predicting creatinine levels using socio-demographic, biometric, and laboratory measurements. This example demonstrates the benefits of an ensemble with carefully selected learners designed to capture key aspects of complex regression functions, including non-linear effects, interactions, sparsity, heteroscedasticity, and robustness to outliers.
Department of Mathematics King’s College London London, UK · Department of Statistics and Operations Research University of Granada Granada, Spain · Department of Statistical Science University College London London, UK
There are many complex issues in the world of artificial intelligence. Some of these problems are solved using other artificial intelligence methods, which are called artificial intelligence for artificial intelligence. Finding an appropriate classifier algorithm is a time-consuming task. For this reason, an algorithm that can automatically learn the choice of classification algorithms is very important. Classification algorithms are useful in predicting various diseases. Also, Primary Biliary Cirrhosis is one of the most well-known diseases that have been predicted by classification algorithms. This research's most significant achievement and novelty is the automatic increase in learning through a scoring method of reinforcement learning is called square learning (SL). In this research, an algorithm is presented that learns to automatically select the appropriate classification algorithm to predict Primary Biliary Cirrhosis. In this article, with inspiration from four evaluation metrics in classification algorithms, a new reinforcement learning method by the name of Fourth Degree Learning has been presented. In this research, we increased the performance of the classification algorithms used in this method from 63% of accuracy and achieved 98% accuracy.
Hasan Samadbin, Arman Daliri
Department of Computer Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran
Longitudinal clinical notes contain rich evidence of how patients evolve over time, but converting this signal into training supervision for clinical prediction remains challenging. We extend Foresight Learning to clinical prediction by converting time-ordered MIMIC-III notes into examples consisting of past patient context, a natural-language question about a possible future event, and a label resolved from later documentation. This process yields 6,900 prediction examples from 702 admissions across medications, procedures, organ support, microbiology, and mortality. A small LoRA adapter trained on these examples improves over the prompted base model, reducing expected calibration error from 0.1269 to 0.0398 and Brier score from 0.199 to 0.145, while slightly outperforming GPT-5 point estimates on held-out questions. The approach enables reusable clinical prediction supervision from longitudinal notes without hand-engineered structured features or endpoint-specific classifiers.