cs.LGApr 10, 2026
SaveA Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics
Organizations: Applied Data Science Program, Eastern University, St. Davids, PA, USA.
Abstract
Student dropout is a persistent concern in Learning Analytics, yet comparative studies frequently evaluate predictive models under heterogeneous protocols, prioritizing discrimination over temporal interpretability and calibration. This study introduces a survival-oriented benchmark for temporal dropout risk modelling using the Open University Learning Analytics Dataset (OULAD). Two arms are compared: Family A: Dynamic Weekly, with models in person-period representation, and Family B: Static Early-Window, with an expanded roster of families: tree-based survival, parametric, and neural models. The evaluation protocol integrates four analytical layers: predictive performance, ablation, explainability, and calibration. Results are reported within each family separately, because a single numerical cross-family ranking would conflate genuine model differences with artifacts of temporal representation, to which survival metrics are known to be sensitive. Within Family B, Random Survival Forest showed the highest point estimates for time-dependent concordance and the lowest Brier scores across all three horizons; within Family A, Poisson Piecewise-Exponential showed the lowest point estimate for integrated Brier score within a tight five-model cluster. No-refit bootstrap resampling qualifies these positions as directional signals, not claims of strict superiority. Ablation and explainability analyses converged, across all models, on a shared finding: the dominant predictive signal was not primarily demographic or structural, but temporal and behavioral. Calibration corroborated this pattern in the better-discriminating models, except for XGBoost AFT, the sole outlier (analyzed in the Discussion). These results support unified, multi-dimensional benchmarking in Learning Analytics and situate dropout risk as a temporal-behavioral process rather than a function of static background attributes.
Figures & tables
| Study | D1 | D2 | D3 | D4 | D5 | D6 |
| Gutierrez-Pachas et al. (2023) ( Gutierrez-Pachas et al., 2023 ) | – | ✓ | – | – | ||
| Pan et al. (2022) ( Pan et al., 2022 ) | – | ✓ | – | – | – | – |
| Adnan et al. (2021) ( Adnan et al., 2021 ) | – | – | – | – | – | |
| He et al. (2020) ( He et al., 2020 ) | – | – | ✓ | – | – | – |
| Villar & de Andrade (2024) ( Villar and de Andrade, 2024 ) | – | – | – | – | – | |
| Coussement et al. (2020) ( Coussement et al., 2020 ) | – | – | – | – | – |
Table 1: Positioning of representative prior studies against six methodological dimensions (D1–D6) of the present benchmark. D1 : dual representation families as experimental control; D2 : survival metrics as primary protocol; D3 : OULAD with weekly/daily VLE granularity; D4 : benchmark design as primary contribution; D5 : block-level ablation with survival metrics; D6 : equity/fairness analysis by subgroup. Symbols: ✓ = present and central; = partial; – = absent.
| Shared (both families) | Family A only (dynamic weekly) | Family B only (static early-window) |
| Static structural block: demographics and prior academic record (OULAD studentInfo ). | Dynamic temporal-behavioral block: week-by-week engagement signals on the person-period panel. | Early-window behavioral block: first-weeks engagement compressed into enrollment-level scalars. |
Table 2: Feature-block membership across the two families. The overlap is intentional and confined to the shared static structural block; the exclusive block is what creates each family’s representational regime. Because both families derive from the same OULAD source, agreement across them is not confounded by differing data. Full variable lists are in Table 3 .
Figure 1: Overview of the full benchmark pipeline, organized into four color-coded phases. The Data phase (blue) covers raw data ingestion and feature engineering from OULAD sources into static, temporal-behavioral, and early-window feature blocks. The Modeling phase (teal) encompasses the enrollment-level train/test partition and the training of 14 tuned models across the two representational arms (Family A: Dynamic Weekly; Family B: Static Early-Window). The Evaluation phase (orange) applies the unified assessment protocol — IBS, time-dependent concordance, Brier scores at three horizons, and calibration gaps — followed by ablation and explainability analyses. The Outputs phase (purple) consolidates within-family rankings, calibration diagnostics, and bootstrap uncertainty estimates as the final benchmark deliverables.
Figure 2: Data preparation pipeline from raw OULAD sources to benchmark-ready datasets for the discrete-time and static early-window families.
| Block / Variable | Description | Computation |
| Static structural covariates — all models | ||
| gender | Student gender | Direct from studentInfo |
| region | UK region of residence | Direct from studentInfo |
| highest_education | Highest prior qualification | Direct from studentInfo |
| imd_band | Index of Multiple Deprivation decile | Direct from studentInfo |
| age_band | Age group at registration | Direct from studentInfo |
Table 3: Feature blocks used in the benchmark, with computation method. Static structural covariates are drawn directly from the OULAD studentInfo table. Dynamic temporal-behavioral features are aggregated from studentVle at the enrollment week level; derived features ( italic ) are computed on the person-period panel. Early-window behavioral summaries collapse the first four weeks of the dynamic panel into enrollment-level scalars. Block membership governs the ablation and grouped permutation importance analyses.
| Group | Model type | Model | Inductive bias |
| Family A | Linear | Linear Discrete-Time Hazard | Linear |
| Poisson Piecewise-Exponential | Linear (log-rate) | ||
| Neural | Neural Discrete-Time Survival | Neural | |
| Gradient-Boosted | GB Weekly Hazard | Tree ensemble | |
| CatBoost Weekly Hazard | Tree ensemble | ||
| Family B | Cox | Cox Comparable | Linear (PH) |
Table 4: Overview of the 14 models organized by representational group (Family A/B), model type, and inductive bias. Models within each group are ordered from linear to neural, with tree-based and parametric families in between.
| Metric | Role | Better when | What it measures |
| Primary performance metrics | |||
| IPCW weight | Censoring correction | — | Inverse-probability-of-censoring weight, estimated non-parametrically via Kaplan-Meier on the censoring process. Upweights uncensored observations to correct for the selective removal of censored students from the risk set as follow-up progresses. Applied internally to Brier@h, IBS, and Calib@h; not a standalone evaluation metric. |
| Brier@10, @20, @30 | Error | Lower | Horizon-specific probabilistic error: squared difference between predicted survival probability and observed event status at a fixed week, averaged over enrollments with IPCW adjustment. Analogous to MSE at a snapshot in time. |
| Integrated Brier Score (IBS) | Error | Lower | Global probabilistic error integrated over the full evaluation window. Survival analogue of MSE: where MSE averages squared errors on a fixed outcome, IBS averages squared errors on the predicted survival curve over time, adjusted for censoring. |
| TD Concordance (C-index) | Discrimination | Higher | Time-dependent concordance (Antolini): probability that a student who drops out earlier receives a higher predicted risk than one who drops out later or does not drop out. Measures ranking quality across the full event-time trajectory, not just at a fixed horizon. |
| Calibration metrics | |||
Table 5: Reference summary of all evaluation metrics used in the benchmark. IBS is the survival analogue of the Mean Squared Error (MSE) used in classical ML regression: where MSE averages squared prediction errors on a fixed outcome, IBS averages squared errors on the predicted survival probability over the full event-time axis, adjusted for censoring via IPCW. For error-based metrics, lower scores correspond to stronger performance, whereas discrimination metrics improve as scores increase. Calibration gaps nearer zero reflect closer agreement between predicted risk and observed dropout rates.
Figure 3: Unified benchmark evaluation flow used to compare the models under a common survival-oriented protocol.
Figure 4: Overview of the explainability and calibration layers used as complementary post-benchmark analyses across the models.
| Model | IBS | TD Concordance | Brier@10 | Brier@20 | Brier@30 |
| Family A: Dynamic Weekly | |||||
| Poisson Piecewise-Exponential | 0.1399 | 0.5019 | 0.1213 | 0.1621 | 0.1931 |
| Linear Discrete-Time Hazard | 0.1400 | 0.5013 | 0.1195 | 0.1637 | 0.1971 |
| CatBoost Weekly Hazard | 0.1408 | 0.5011 | 0.1203 | 0.1644 | 0.1980 |
| Neural Discrete-Time Survival | 0.1410 | 0.4992 | 0.1199 | 0.1648 | 0.1990 |
| Gradient-Boosted Weekly Hazard | 0.1410 | 0.5018 | 0.1206 | 0.1646 | 0.1981 |
Table 6: Main benchmark comparison across all 14 models, grouped by family. For Family A, Brier-horizon scores are computed from enrollment-level survival curves reconstructed from accumulated weekly hazards, using the same IPCW methodology as Family B; both families are evaluated on the full metric set. Lower IBS and Brier scores indicate better probabilistic accuracy (lower mean squared survival error); higher time-dependent concordance indicates better discrimination.
Figure 5: Benchmark comparison of all 14 models under the unified survival-oriented protocol, grouped by family. Within Family B, RSF leads on time-dependent concordance and all Brier horizons; XGBoost AFT is a visible outlier. Within Family A, all five models cluster tightly on IBS and horizon-specific Brier; Brier values for Family A are reconstructed from accumulated weekly hazards.
| Model | IBS static | IBS temporal | TD static | TD temporal | IBS ratio |
| Cox Comparable | 0.0020 | 0.0049 | 0.0288 | 0.0598 | 2.46 |
| DeepSurv | 0.0044 | 0.0082 | 0.0361 | 0.0711 | 1.86 |
| Neural-MTLR | 0.0050 | 0.0095 | 0.0385 | 0.0747 | 1.90 |
| Random Survival Forest | 0.0089 | 0.0106 | 0.0510 | 0.0764 | 1.20 |
Table 7: Ablation results: Family B (early-window continuous-time survival). IBS ratios above 1.0 indicate greater dependence on temporal-behavioral signal than on static covariates.
| Model | TD static | TD temporal |
| Gradient-Boosted Weekly Hazard | 0.0513 | 0.1023 |
| Linear Discrete-Time Hazard | 0.0434 | 0.0976 |
| Neural Discrete-Time Survival | 0.0178 | 0.0812 |
| Poisson Piecewise-Exponential | 0.0389 | 0.0997 |
Table 8: Ablation results: Family A (weekly discrete-time hazard). Only concordance-based evidence is reported ( TD concordance). IBS-based ablation was not computed for Family A models: the per-person-period hazard evaluation surface differs in temporal footprint from the enrollment-level survival curve on which IBS is defined, and converting the ablation to IBS would require re-running the full curve-reconstruction pipeline under each feature-removal variant. Concordance-based ablation is therefore the methodologically consistent metric for this family.
Figure 6: Ablation impact across eight models from both families, contrasting the loss associated with removing static versus temporal-behavioral signal. Across all families, removing temporal-behavioral signal is more damaging than removing static covariates. The IBS panel (left) shows only the four Family B models because IBS-based ablation requires enrollment-level survival curves, which are directly available for the early-window Family B but not for Family A, where predictions are weekly hazard scores. The TD concordance panel (right) includes all eight models.
| Model | Top driver | Dominant block |
| Family B | ||
| Cox Comparable | clicks_first_4_weeks | early_window_behavior |
| DeepSurv | active_weeks_first_4 | early_window_behavior |
| Random Survival Forest | active_weeks_first_4 | early_window_behavior |
| Neural-MTLR | active_weeks_first_4 | early_window_behavior |
| Family A | ||
Table 9: Cross-family explainability summary for eight models across both families. Early-window behavioral : compressed early-course engagement summaries. Dynamic temporal-behavioral : week-by-week activity features. Discrete time index : the time-step covariate (week number).
Figure 7: Normalized explainability block dominance across eight models from both families included in the paper-facing export. Values are normalized within model to highlight relative block importance; the figure should be read as a within-model comparison of dominant signal blocks, not as an absolute cross-model magnitude comparison.
| Model | Calib@10 | Calib@20 | Calib@30 |
| Family B | |||
| Random Survival Forest | 0.0121 | 0.0162 | 0.0138 |
| DeepSurv | 0.0320 | 0.0145 | 0.0119 |
| DeepHit | 0.0333 | 0.0322 | 0.0270 |
| Weibull AFT | 0.0369 | 0.0219 | 0.0123 |
| Neural-MTLR | 0.0384 | 0.0503 | 0.0521 |
Table 10: Calibration gap summary for Family A and Family B models. Values are IPCW-adjusted mean absolute gaps between predicted risk and observed event rate across quantile bins; lower values indicate better calibration.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Status/value | Details |
| Unit of analysis | defined: enrollment | All final benchmark comparisons are reported at the enrollment level. |
| Event definition | defined: withdrawn with valid date (unregistration) | Observed withdrawal event with a valid time stamp (unregistration with a recorded date). |
| Official horizons | defined: 10, 20, 30 | Shared benchmark horizons used for Brier, IBS, and calibration reporting. |
| Primary discrimination metric | defined: TD concordance | The canonical discrimination metric is time-dependent concordance rather than a static concordance proxy. |
| Censoring treatment | defined: IPCW with KM | Brier score and IBS use inverse-probability-of-censoring weighting with the Kaplan–Meier censoring estimator. |
| Dynamic versus comparable rule | defined: dynamic weekly vs early-window comparable | Weekly discrete-time arms update predictions over person-period rows, whereas comparable continuous-time arms use early-window enrollment representations; cross-family comparison occurs only after shared enrollment-level horizon reporting. |
Table 11: Evaluation protocol audit for the unified benchmark.
| Model | Input level | Validation strategy | Tuning and complexity control |
| Cox Comparable | enrollment early window | Enrollment-level split with event stratification when possible (20%). | 24 candidates. Selection by highest validation C-index. Restricted regularization grid over penalizer and -ratio. No early stopping. |
| DeepSurv | enrollment early window | Internal validation fraction on training rows (20%). | 24 candidates. Selection by lowest validation loss. Early stopping (patience 10). Complexity controls: architecture grid, dropout, weight decay, best-epoch refit. |
| Random Survival Forest | enrollment early window | Enrollment-level split with event stratification when possible (20%). | 24 candidates. Selection by lowest validation IBS. No early stopping. Complexity controls: forest size, minimum leaf size, maximum depth, feature subsampling fraction. |
| Neural-MTLR | enrollment early window | Enrollment-level split with deterministic survival-time discretization (10%). | 24 candidates. Selection by lowest validation IBS. Early stopping (patience 8). Complexity controls: discretization grid, architecture, dropout, weight decay. |
Table 12: Preprocessing and tuning audit for the four Family B models retained in the focused comparative export. All families used median numeric imputation, constant-missing categorical imputation, one-hot encoding, and standard scaling fit on training rows only. Tuning was deliberately controlled, not exhaustive.
| Model | IBS [95% CI] | TD concordance [95% CI] | Brier@10 [95% CI] | Brier@20 [95% CI] | Brier@30 [95% CI] |
| RSF | 0.1118 [0.109, 0.115] | 0.6735 [0.663, 0.684] | 0.0947 [0.091, 0.099] | 0.1329 [0.129, 0.137] | 0.1574 [0.153, 0.161] |
| DeepSurv | 0.1145 [0.111, 0.118] | 0.6619 [0.651, 0.673] | 0.0978 [0.094, 0.102] | 0.1343 [0.131, 0.138] | 0.1589 [0.155, 0.163] |
| Cox Comparable | 0.1176 [0.114, 0.121] | 0.6512 [0.640, 0.663] | 0.1012 [0.097, 0.105] | 0.1378 [0.133, 0.142] | 0.1623 [0.158, 0.166] |
| Neural-MTLR | 0.1131 [0.109, 0.117] | 0.6569 [0.647, 0.668] | 0.0954 [0.091, 0.100] | 0.1340 [0.129, 0.139] | 0.1588 [0.154, 0.163] |
Table 13: Bootstrap uncertainty for the four Family B models (200 enrollment-level resamples, no refit, frozen survival predictions). Lower IBS and Brier scores indicate better probabilistic accuracy (lower mean squared survival error); higher time-dependent concordance indicates better discrimination.
| Metric | Direction | Leading model | Claim status | Conservative takeaway |
| Integrated Brier Score | lower is better | RSF | directional signal only | IBS intervals overlap substantially; RSF leads narrowly but the ranking is not stable enough for a strong dominance claim. |
| Time-dependent concordance | higher is better | RSF | largely stable bootstrap leadership | RSF held rank 1 in 90% of resamples; largely stable but not absolute leadership. |
Table 14: Bootstrap inferential scope for the main benchmark leadership claims (Family B, 200 enrollment-level resamples, no refit).
| Split unit | Stratification | Total | Train | Test | Train event rate | Test event rate |
| enrollment | event status + coarse event-time bucket | 32,593 | 22,815 | 9,778 | 0.2266 | 0.2266 |
Table 15: Split and contextual overlap audit for the main benchmark.
| Model | Representation | Candi- dates | Rows per enrollment | Row event rate | IBS rank | TD rank |
| Neural Discrete-Time | dynamic weekly person-period | 24 | 23.77 | 0.0095 | 4 | 1 |
| Linear Discrete-Time | dynamic weekly person-period | 24 | 23.77 | 0.0095 | 1 | 2 |
Table 16: Discrete-time diagnostic summary for Family A models.
| Hypothesis | Status | Conservative takeaway |
| Weak tuning as primary driver | tested, not supported as primary explanation | Family A models were explicitly tuned; weak tuning alone is not the best-supported explanation for the final benchmark gap. |
| Absent weekly signal | tested, not supported | Weekly rows contain predictive signal; the underperformance emerges downstream when those weekly hazards must support enrollment-level survival ranking and calibration. |
| Sparse low-base-rate aggregation burden | tested, partially supported | The dynamic representation faces a sparse low-base-rate aggregation problem that is consistent with harder survival reconstruction and calibration, but this remains evidence-consistent rather than causal proof. |
| Neural weekly instability or overdispersion | tested, partially supported | Family A neural models appear more expressive locally, but that extra flexibility does not translate into a stronger final ranking profile under the benchmark contract. |
| Representational inefficiency under the current contract | residual interpretation best supported | Family A captures signal, but less efficiently than the early-window comparable design under the present benchmark contract. |
Table 17: Hypothesis audit for the discrete-time underperformance reading.
| Model | Covariates tested | Flagged | Global interpretation |
| Cox Comparable | 38 | 10 (26.3%) | Broad departure: approximate comparable benchmark, not assumption-clean Cox specification |
Table 18: Global proportional-hazards audit summary for the comparable Cox model.
| Model | Assumption class | Formal PH audit | Paper claim boundary |
| Cox Comparable | PH (linear) | yes | broad departure from proportionality; all comparable Cox results should be narrated as from an approximate benchmark specification, not from a fully assumption-clean Cox model. |
| DeepSurv | Cox-type neural (nonlinear) | no | Classical Schoenfeld diagnostics require per-subject partial-likelihood score contributions, which are not computable in closed form when the linear predictor is replaced by a neural network. PH diagnostics are therefore methodologically inapplicable (not merely omitted). Evaluated via IBS, concordance, calibration gaps, and bootstrap rank stability. |
| Random Survival Forest | tree-survival (non-parametric) | N/A | No PH assumption is made; PH audit is not applicable. |
| Gradient-Boosted Cox | PH (partial likelihood) | in principle; not run | Partial-likelihood objective implies PH; a Schoenfeld audit would be applicable but was not executed in this benchmark. |
| Weibull AFT | AFT (parametric) | N/A | AFT model; PH assumption does not apply; accelerated time parametrization is used instead. |
| Royston-Parmar | PH spline (relaxed PH) | no | Spline-based relaxation of PH; dedicated diagnostics exist but were not applied. Narrate as relaxed-PH, not audited-PH. |
Table 19: Proportional-hazards scope boundary across all nine Family B models. Formal classical PH diagnostics (Schoenfeld residuals) require a linear partial-likelihood estimator; this condition is met only by the Cox anchor, which was formally audited.
| Model | IBS | TD Concordance |
| Random Survival Forest | 0.1118 | 0.6735 |
| Neural-MTLR | 0.1131 | 0.6569 |
| DeepSurv | 0.1145 | 0.6619 |
| Gradient-Boosted Cox | 0.1148 | 0.6621 |
| DeepHit | 0.1151 | 0.5835 |
| Cox Comparable | 0.1176 | 0.6512 |
Table 20: Early-window sensitivity for Family B. IBS (lower is better) and time-dependent concordance (higher is better) for each model, reported at the canonical window ( weeks). Performance generally improves with longer observation windows; full sensitivity results are available in the pipeline outputs.
Figure 8: Reliability diagrams for Family B (panel 1 of 3). This panel covers the first subset of static early-window families; most points remain close to the diagonal, consistent with the strong calibration profile of the leading static early-window models.
Figure 9: Reliability diagrams for Family B (panel 2 of 3). The middle subset preserves the same three-horizon calibration comparison while avoiding the page overflow of the original full-width grid.
Figure 10: Reliability diagrams for Family B (panel 3 of 3). This final panel includes the remaining static early-window families and retains the visible calibration departure of XGBoost AFT relative to the better-calibrated alternatives.
Figure 11: Reliability diagrams for Family A. Rows correspond to horizons 10, 20, and 30 weeks; columns correspond to models. Family A calibration gaps remain wider than those of the strongest Family B models, but the panels still show coherent horizon-wise probability structure.