Student dropout is a persistent concern in Learning Analytics, yet comparative studies frequently evaluate predictive models under heterogeneous protocols, prioritizing discrimination over temporal interpretability and calibration. This study introduces a survival-oriented benchmark for temporal dropout risk modelling using the Open University Learning Analytics Dataset (OULAD). Two arms are compared: Family A: Dynamic Weekly, with models in person-period representation, and Family B: Static Early-Window, with an expanded roster of families: tree-based survival, parametric, and neural models. The evaluation protocol integrates four analytical layers: predictive performance, ablation, explainability, and calibration. Results are reported within each family separately, because a single numerical cross-family ranking would conflate genuine model differences with artifacts of temporal representation, to which survival metrics are known to be sensitive. Within Family B, Random Survival Forest showed the highest point estimates for time-dependent concordance and the lowest Brier scores across all three horizons; within Family A, Poisson Piecewise-Exponential showed the lowest point estimate for integrated Brier score within a tight five-model cluster. No-refit bootstrap resampling qualifies these positions as directional signals, not claims of strict superiority. Ablation and explainability analyses converged, across all models, on a shared finding: the dominant predictive signal was not primarily demographic or structural, but temporal and behavioral. Calibration corroborated this pattern in the better-discriminating models, except for XGBoost AFT, the sole outlier (analyzed in the Discussion). These results support unified, multi-dimensional benchmarking in Learning Analytics and situate dropout risk as a temporal-behavioral process rather than a function of static background attributes.
Figures & tables
Study
D1
D2
D3
D4
D5
D6
Gutierrez-Pachas et al. (2023) ( Gutierrez-Pachas et al., 2023 )
–
✓
–
∼
–
∼
Pan et al. (2022) ( Pan et al., 2022 )
–
✓
–
–
–
–
Adnan et al. (2021) ( Adnan et al., 2021 )
–
–
∼
–
–
–
He et al. (2020) ( He et al., 2020 )
–
–
✓
–
–
–
Villar & de Andrade (2024) ( Villar and de Andrade, 2024 )
–
–
–
∼
–
–
Coussement et al. (2020) ( Coussement et al., 2020 )
–
–
–
∼
–
–
Table 1: Positioning of representative prior studies against six methodological dimensions (D1–D6) of the present benchmark. D1 : dual representation families as experimental control; D2 : survival metrics as primary protocol; D3 : OULAD with weekly/daily VLE granularity; D4 : benchmark design as primary contribution; D5 : block-level ablation with survival metrics; D6 : equity/fairness analysis by subgroup. Symbols: ✓ = present and central; ∼ = partial; – = absent.
Shared (both families)
Family A only (dynamic weekly)
Family B only (static early-window)
Static structural block: demographics and prior academic record (OULAD studentInfo ).
Dynamic temporal-behavioral block: week-by-week engagement signals on the person-period panel.
Early-window behavioral block: first-weeks engagement compressed into enrollment-level scalars.
Table 2: Feature-block membership across the two families. The overlap is intentional and confined to the shared static structural block; the exclusive block is what creates each family’s representational regime. Because both families derive from the same OULAD source, agreement across them is not confounded by differing data. Full variable lists are in Table 3 .
Figure 1: Overview of the full benchmark pipeline, organized into four color-coded phases. The Data phase (blue) covers raw data ingestion and feature engineering from OULAD sources into static, temporal-behavioral, and early-window feature blocks. The Modeling phase (teal) encompasses the enrollment-level train/test partition and the training of 14 tuned models across the two representational arms (Family A: Dynamic Weekly; Family B: Static Early-Window). The Evaluation phase (orange) applies the unified assessment protocol — IBS, time-dependent concordance, Brier scores at three horizons, and calibration gaps — followed by ablation and explainability analyses. The Outputs phase (purple) consolidates within-family rankings, calibration diagnostics, and bootstrap uncertainty estimates as the final benchmark deliverables.
Figure 2: Data preparation pipeline from raw OULAD sources to benchmark-ready datasets for the discrete-time and static early-window families.
Block / Variable
Description
Computation
Static structural covariates — all models
gender
Student gender
Direct from studentInfo
region
UK region of residence
Direct from studentInfo
highest_education
Highest prior qualification
Direct from studentInfo
imd_band
Index of Multiple Deprivation decile
Direct from studentInfo
age_band
Age group at registration
Direct from studentInfo
Table 3: Feature blocks used in the benchmark, with computation method. Static structural covariates are drawn directly from the OULAD studentInfo table. Dynamic temporal-behavioral features are aggregated from studentVle at the enrollment × week level; derived features ( italic ) are computed on the person-period panel. Early-window behavioral summaries collapse the first four weeks of the dynamic panel into enrollment-level scalars. Block membership governs the ablation and grouped permutation importance analyses.
Group
Model type
Model
Inductive bias
Family A
Linear
Linear Discrete-Time Hazard
Linear
Poisson Piecewise-Exponential
Linear (log-rate)
Neural
Neural Discrete-Time Survival
Neural
Gradient-Boosted
GB Weekly Hazard
Tree ensemble
CatBoost Weekly Hazard
Tree ensemble
Family B
Cox
Cox Comparable
Linear (PH)
Table 4: Overview of the 14 models organized by representational group (Family A/B), model type, and inductive bias. Models within each group are ordered from linear to neural, with tree-based and parametric families in between.
Metric
Role
Better when
What it measures
Primary performance metrics
IPCW weight G^(t)−1
Censoring correction
—
Inverse-probability-of-censoring weight, estimated non-parametrically via Kaplan-Meier on the censoring process. Upweights uncensored observations to correct for the selective removal of censored students from the risk set as follow-up progresses. Applied internally to Brier@h, IBS, and Calib@h; not a standalone evaluation metric.
Brier@10, @20, @30
Error
Lower ↓
Horizon-specific probabilistic error: squared difference between predicted survival probability and observed event status at a fixed week, averaged over enrollments with IPCW adjustment. Analogous to MSE at a snapshot in time.
Integrated Brier Score (IBS)
Error
Lower ↓
Global probabilistic error integrated over the full evaluation window. Survival analogue of MSE: where MSE averages squared errors on a fixed outcome, IBS averages squared errors on the predicted survival curve over time, adjusted for censoring.
TD Concordance (C-index)
Discrimination
Higher ↑
Time-dependent concordance (Antolini): probability that a student who drops out earlier receives a higher predicted risk than one who drops out later or does not drop out. Measures ranking quality across the full event-time trajectory, not just at a fixed horizon.
Calibration metrics
Table 5: Reference summary of all evaluation metrics used in the benchmark. IBS is the survival analogue of the Mean Squared Error (MSE) used in classical ML regression: where MSE averages squared prediction errors on a fixed outcome, IBS averages squared errors on the predicted survival probability over the full event-time axis, adjusted for censoring via IPCW. For error-based metrics, lower scores correspond to stronger performance, whereas discrimination metrics improve as scores increase. Calibration gaps nearer zero reflect closer agreement between predicted risk and observed dropout rates.
Figure 3: Unified benchmark evaluation flow used to compare the models under a common survival-oriented protocol.
Figure 4: Overview of the explainability and calibration layers used as complementary post-benchmark analyses across the models.
Model
IBS
TD Concordance
Brier@10
Brier@20
Brier@30
Family A: Dynamic Weekly
Poisson Piecewise-Exponential
0.1399
0.5019
0.1213
0.1621
0.1931
Linear Discrete-Time Hazard
0.1400
0.5013
0.1195
0.1637
0.1971
CatBoost Weekly Hazard
0.1408
0.5011
0.1203
0.1644
0.1980
Neural Discrete-Time Survival
0.1410
0.4992
0.1199
0.1648
0.1990
Gradient-Boosted Weekly Hazard
0.1410
0.5018
0.1206
0.1646
0.1981
Table 6: Main benchmark comparison across all 14 models, grouped by family. For Family A, Brier-horizon scores are computed from enrollment-level survival curves reconstructed from accumulated weekly hazards, using the same IPCW methodology as Family B; both families are evaluated on the full metric set. Lower IBS and Brier scores indicate better probabilistic accuracy (lower mean squared survival error); higher time-dependent concordance indicates better discrimination.
Figure 5: Benchmark comparison of all 14 models under the unified survival-oriented protocol, grouped by family. Within Family B, RSF leads on time-dependent concordance and all Brier horizons; XGBoost AFT is a visible outlier. Within Family A, all five models cluster tightly on IBS and horizon-specific Brier; Brier values for Family A are reconstructed from accumulated weekly hazards.
Model
Δ IBS static
Δ IBS temporal
Δ TD static
Δ TD temporal
IBS ratio
Cox Comparable
0.0020
0.0049
− 0.0288
− 0.0598
2.46
DeepSurv
0.0044
0.0082
− 0.0361
− 0.0711
1.86
Neural-MTLR
0.0050
0.0095
− 0.0385
− 0.0747
1.90
Random Survival Forest
0.0089
0.0106
− 0.0510
− 0.0764
1.20
Table 7: Ablation results: Family B (early-window continuous-time survival). IBS ratios above 1.0 indicate greater dependence on temporal-behavioral signal than on static covariates.
Model
Δ TD static
Δ TD temporal
Gradient-Boosted Weekly Hazard
− 0.0513
− 0.1023
Linear Discrete-Time Hazard
− 0.0434
− 0.0976
Neural Discrete-Time Survival
− 0.0178
− 0.0812
Poisson Piecewise-Exponential
− 0.0389
− 0.0997
Table 8: Ablation results: Family A (weekly discrete-time hazard). Only concordance-based evidence is reported ( Δ TD concordance). IBS-based ablation was not computed for Family A models: the per-person-period hazard evaluation surface differs in temporal footprint from the enrollment-level survival curve on which IBS is defined, and converting the ablation to IBS would require re-running the full curve-reconstruction pipeline under each feature-removal variant. Concordance-based ablation is therefore the methodologically consistent metric for this family.
Figure 6: Ablation impact across eight models from both families, contrasting the loss associated with removing static versus temporal-behavioral signal. Across all families, removing temporal-behavioral signal is more damaging than removing static covariates. The IBS panel (left) shows only the four Family B models because IBS-based ablation requires enrollment-level survival curves, which are directly available for the early-window Family B but not for Family A, where predictions are weekly hazard scores. The TD concordance panel (right) includes all eight models.
Model
Top driver
Dominant block
Family B
Cox Comparable
clicks_first_4_weeks
early_window_behavior
DeepSurv
active_weeks_first_4
early_window_behavior
Random Survival Forest
active_weeks_first_4
early_window_behavior
Neural-MTLR
active_weeks_first_4
early_window_behavior
Family A
Table 9: Cross-family explainability summary for eight models across both families. Early-window behavioral : compressed early-course engagement summaries. Dynamic temporal-behavioral : week-by-week activity features. Discrete time index : the time-step covariate (week number).
Figure 7: Normalized explainability block dominance across eight models from both families included in the paper-facing export. Values are normalized within model to highlight relative block importance; the figure should be read as a within-model comparison of dominant signal blocks, not as an absolute cross-model magnitude comparison.
Model
Calib@10
Calib@20
Calib@30
Family B
Random Survival Forest
0.0121
0.0162
0.0138
DeepSurv
0.0320
0.0145
0.0119
DeepHit
0.0333
0.0322
0.0270
Weibull AFT
0.0369
0.0219
0.0123
Neural-MTLR
0.0384
0.0503
0.0521
Table 10: Calibration gap summary for Family A and Family B models. Values are IPCW-adjusted mean absolute gaps between predicted risk and observed event rate across quantile bins; lower values indicate better calibration.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Status/value
Details
Unit of analysis
defined: enrollment
All final benchmark comparisons are reported at the enrollment level.
Event definition
defined: withdrawn with valid date (unregistration)
Observed withdrawal event with a valid time stamp (unregistration with a recorded date).
Official horizons
defined: 10, 20, 30
Shared benchmark horizons used for Brier, IBS, and calibration reporting.
Primary discrimination metric
defined: TD concordance
The canonical discrimination metric is time-dependent concordance rather than a static concordance proxy.
Censoring treatment
defined: IPCW with KM
Brier score and IBS use inverse-probability-of-censoring weighting with the Kaplan–Meier censoring estimator.
Dynamic versus comparable rule
defined: dynamic weekly vs early-window comparable
Weekly discrete-time arms update predictions over person-period rows, whereas comparable continuous-time arms use early-window enrollment representations; cross-family comparison occurs only after shared enrollment-level horizon reporting.
Appendix
Table 11: Evaluation protocol audit for the unified benchmark.
Model
Input level
Validation strategy
Tuning and complexity control
Cox Comparable
enrollment early window
Enrollment-level split with event stratification when possible (20%).
24 candidates. Selection by highest validation C-index. Restricted regularization grid over penalizer and l1 -ratio. No early stopping.
DeepSurv
enrollment early window
Internal validation fraction on training rows (20%).
24 candidates. Selection by lowest validation loss. Early stopping (patience 10). Complexity controls: architecture grid, dropout, weight decay, best-epoch refit.
Random Survival Forest
enrollment early window
Enrollment-level split with event stratification when possible (20%).
24 candidates. Selection by lowest validation IBS. No early stopping. Complexity controls: forest size, minimum leaf size, maximum depth, feature subsampling fraction.
Neural-MTLR
enrollment early window
Enrollment-level split with deterministic survival-time discretization (10%).
24 candidates. Selection by lowest validation IBS. Early stopping (patience 8). Complexity controls: discretization grid, architecture, dropout, weight decay.
Appendix
Table 12: Preprocessing and tuning audit for the four Family B models retained in the focused comparative export. All families used median numeric imputation, constant-missing categorical imputation, one-hot encoding, and standard scaling fit on training rows only. Tuning was deliberately controlled, not exhaustive.
Model
IBS [95% CI]
TD concordance [95% CI]
Brier@10 [95% CI]
Brier@20 [95% CI]
Brier@30 [95% CI]
RSF
0.1118 [0.109, 0.115]
0.6735 [0.663, 0.684]
0.0947 [0.091, 0.099]
0.1329 [0.129, 0.137]
0.1574 [0.153, 0.161]
DeepSurv
0.1145 [0.111, 0.118]
0.6619 [0.651, 0.673]
0.0978 [0.094, 0.102]
0.1343 [0.131, 0.138]
0.1589 [0.155, 0.163]
Cox Comparable
0.1176 [0.114, 0.121]
0.6512 [0.640, 0.663]
0.1012 [0.097, 0.105]
0.1378 [0.133, 0.142]
0.1623 [0.158, 0.166]
Neural-MTLR
0.1131 [0.109, 0.117]
0.6569 [0.647, 0.668]
0.0954 [0.091, 0.100]
0.1340 [0.129, 0.139]
0.1588 [0.154, 0.163]
Appendix
Table 13: Bootstrap uncertainty for the four Family B models (200 enrollment-level resamples, no refit, frozen survival predictions). Lower IBS and Brier scores indicate better probabilistic accuracy (lower mean squared survival error); higher time-dependent concordance indicates better discrimination.
Metric
Direction
Leading model
Claim status
Conservative takeaway
Integrated Brier Score
lower is better
RSF
directional signal only
IBS intervals overlap substantially; RSF leads narrowly but the ranking is not stable enough for a strong dominance claim.
Time-dependent concordance
higher is better
RSF
largely stable bootstrap leadership
RSF held rank 1 in 90% of resamples; largely stable but not absolute leadership.
Appendix
Table 14: Bootstrap inferential scope for the main benchmark leadership claims (Family B, 200 enrollment-level resamples, no refit).
Split unit
Stratification
Total
Train
Test
Train event rate
Test event rate
enrollment
event status + coarse event-time bucket
32,593
22,815
9,778
0.2266
0.2266
Appendix
Table 15: Split and contextual overlap audit for the main benchmark.
Model
Representation
Candi- dates
Rows per enrollment
Row event rate
IBS rank
TD rank
Neural Discrete-Time
dynamic weekly person-period
24
23.77
0.0095
4
1
Linear Discrete-Time
dynamic weekly person-period
24
23.77
0.0095
1
2
Appendix
Table 16: Discrete-time diagnostic summary for Family A models.
Hypothesis
Status
Conservative takeaway
Weak tuning as primary driver
tested, not supported as primary explanation
Family A models were explicitly tuned; weak tuning alone is not the best-supported explanation for the final benchmark gap.
Absent weekly signal
tested, not supported
Weekly rows contain predictive signal; the underperformance emerges downstream when those weekly hazards must support enrollment-level survival ranking and calibration.
Sparse low-base-rate aggregation burden
tested, partially supported
The dynamic representation faces a sparse low-base-rate aggregation problem that is consistent with harder survival reconstruction and calibration, but this remains evidence-consistent rather than causal proof.
Neural weekly instability or overdispersion
tested, partially supported
Family A neural models appear more expressive locally, but that extra flexibility does not translate into a stronger final ranking profile under the benchmark contract.
Representational inefficiency under the current contract
residual interpretation best supported
Family A captures signal, but less efficiently than the early-window comparable design under the present benchmark contract.
Appendix
Table 17: Hypothesis audit for the discrete-time underperformance reading.
Model
Covariates tested
Flagged
Global interpretation
Cox Comparable
38
10 (26.3%)
Broad departure: approximate comparable benchmark, not assumption-clean Cox specification
Appendix
Table 18: Global proportional-hazards audit summary for the comparable Cox model.
Model
Assumption class
Formal PH audit
Paper claim boundary
Cox Comparable
PH (linear)
yes
broad departure from proportionality; all comparable Cox results should be narrated as from an approximate benchmark specification, not from a fully assumption-clean Cox model.
DeepSurv
Cox-type neural (nonlinear)
no
Classical Schoenfeld diagnostics require per-subject partial-likelihood score contributions, which are not computable in closed form when the linear predictor is replaced by a neural network. PH diagnostics are therefore methodologically inapplicable (not merely omitted). Evaluated via IBS, concordance, calibration gaps, and bootstrap rank stability.
Random Survival Forest
tree-survival (non-parametric)
N/A
No PH assumption is made; PH audit is not applicable.
Gradient-Boosted Cox
PH (partial likelihood)
in principle; not run
Partial-likelihood objective implies PH; a Schoenfeld audit would be applicable but was not executed in this benchmark.
Weibull AFT
AFT (parametric)
N/A
AFT model; PH assumption does not apply; accelerated time parametrization is used instead.
Royston-Parmar
PH spline (relaxed PH)
no
Spline-based relaxation of PH; dedicated diagnostics exist but were not applied. Narrate as relaxed-PH, not audited-PH.
Appendix
Table 19: Proportional-hazards scope boundary across all nine Family B models. Formal classical PH diagnostics (Schoenfeld residuals) require a linear partial-likelihood estimator; this condition is met only by the Cox anchor, which was formally audited.
Model
IBS
TD Concordance
Random Survival Forest
0.1118
0.6735
Neural-MTLR
0.1131
0.6569
DeepSurv
0.1145
0.6619
Gradient-Boosted Cox
0.1148
0.6621
DeepHit
0.1151
0.5835
Cox Comparable
0.1176
0.6512
Appendix
Table 20: Early-window sensitivity for Family B. IBS (lower is better) and time-dependent concordance (higher is better) for each model, reported at the canonical window ( w=4 weeks). Performance generally improves with longer observation windows; full sensitivity results are available in the pipeline outputs.
Figure 8: Reliability diagrams for Family B (panel 1 of 3). This panel covers the first subset of static early-window families; most points remain close to the diagonal, consistent with the strong calibration profile of the leading static early-window models.
Figure 9: Reliability diagrams for Family B (panel 2 of 3). The middle subset preserves the same three-horizon calibration comparison while avoiding the page overflow of the original full-width grid.
Figure 10: Reliability diagrams for Family B (panel 3 of 3). This final panel includes the remaining static early-window families and retains the visible calibration departure of XGBoost AFT relative to the better-calibrated alternatives.
Figure 11: Reliability diagrams for Family A. Rows correspond to horizons 10, 20, and 30 weeks; columns correspond to models. Family A calibration gaps remain wider than those of the strongest Family B models, but the panels still show coherent horizon-wise probability structure.
This study proposes a temporal modeling framework with a counterfactual policy-simulation layer for student dropout in higher education, using LMS engagement data and administrative withdrawal records. Dropout is operationalized as a time-to-event outcome at the enrollment level; weekly risk is modeled in discrete time via penalized, class-balanced logistic regression over person--period rows. Under a late-event temporal holdout, the model attains row-level AUCs of 0.8350 (train) and 0.8405 (test), with aggregate calibration acceptable but sparsely supported in the highest-risk bins. Ablation analyses indicate performance is sensitive to feature set composition, underscoring the role of temporal engagement signals. A scenario-indexed policy layer produces survival contrasts ΔS(T) under an explicit trigger/schedule contract: positive contrasts are confined to the shock branch (Tpolicy=18: 0.0102, 0.0260, 0.0819), while the mechanism-aware branch is negative (ΔSmech(18)=−0.0078, ΔSmech(38)=−0.0134). A subgroup analysis by gender quantifies scenario-induced survival gaps via bootstrap; contrasts are directionally stable but small. Results are not causally identified; they demonstrate the framework's capacity for internal structural scenario comparison under observational data constraints.
Rafael da Silva, Jeff Eicher, Gregory Longo
Applied Data Science Program, Eastern University, St. Davids, PA, USA.
Early-warning models built from Learning Management System (LMS) logs aim to predict end-of-course outcomes early enough to enable timely learner support. However, reported "early" performance is often inflated by temporal leakage. This occurs when the pipeline uses information that would not yet be available at the time of prediction. We formalize cutoff-based early outcome prediction under a temporal availability constraint and introduce LEAP (Leakage-Excluded Early-Availability Protocol), which enforces cutoff-first truncation prior to joins and aggregation and audits feature provenance to prevent post-cutoff evidence from entering the benchmark. We instantiate LEAP on the public Open University Learning Analytics Dataset (OULAD) as a multi-step protocol for leakage-controlled evaluation across weekly cutoffs. Using several standard learning methods, we evaluate performance using ROC-AUC, PR-AUC, Brier score, and [email protected]. Results show improving performance as the observation window expands, with a marked gain around week~3; Random Forest performs best at the earliest cutoffs, while Gradient Boosting dominates thereafter. Leakage ablations further show that temporal violations, especially through assessment information, can inflate apparent "early" performance.
Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge
Gamaizer, 93340 Le Raincy, France · Université de technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne Cedex, France · Sorbonne Université, CNRS UMR 7585, LPMHE (Laboratoire de Physique Nucléaire et des Hautes Énergies), 75252 Paris cedex 05, France
Different predictors often excel on different inputs, so picking the best one per instance promises higher accuracy than committing to a single model. In practice, selectors trained from logged data routinely fail to beat the strongest single predictor. Three causes typically go unseparated before more tuning is applied: a mismatched learner, a state that does not predict which model wins, or buffer-to-deployment label shift. A three-stage diagnostic rules them out on a shared buffer. Stage1 estimates a local ceiling on oracle recovery from k-NN label consistency. Stage2 asks whether paired BC and offline-RL learners (BC, DQN, and CQL across penalty weights) reach that ceiling. Stage~3 ablates the selector state to test whether richer features would raise it. The combined verdict points to the most promising next step: tuning the learner, redesigning the state, or collecting new data. We apply it to selecting among five dropout-prediction models on edX clickstream data. Across 16 windows, the oracle beats the strongest single base model by 9.7 accuracy points on average, yet BC, DQN, and CQL land in the same test-accuracy band below it (robust to a tenfold buffer sweep and N=2,000 held-out examples). The bottleneck is local representational ambiguity: CQL closes the imitation gap without a deployment gain (not conservatism), regret clusters tightly across learners (not tie-breaking), and the three learners converge on test accuracy (not shift). The next iteration should change the state or collect new data, not tune the offline learner further.
Tyler Crosse, Alan Nadelsticher Ruvalcaba, Dustin Khang LeDuc +3