An Auditable Policy-Simulation Framework for Student Dropout in Intervention-Free Data

Rafael da Silva; Jeff Eicher; Gregory Longo

cs.LGApr 10, 2026

An Auditable Policy-Simulation Framework for Student Dropout in Intervention-Free Data

Authors: Rafael da Silva, Jeff Eicher, Gregory Longo

Organizations: Applied Data Science Program, Eastern University, St. Davids, PA, USA.

Abstract

This study proposes a temporal modeling framework with a counterfactual policy-simulation layer for student dropout in higher education, using LMS engagement data and administrative withdrawal records. Dropout is operationalized as a time-to-event outcome at the enrollment level; weekly risk is modeled in discrete time via penalized, class-balanced logistic regression over person--period rows. Under a late-event temporal holdout, the model attains row-level AUCs of 0.8350 (train) and 0.8405 (test), with aggregate calibration acceptable but sparsely supported in the highest-risk bins. Ablation analyses indicate performance is sensitive to feature set composition, underscoring the role of temporal engagement signals. A scenario-indexed policy layer produces survival contrasts $ΔS(T)$ under an explicit trigger/schedule contract: positive contrasts are confined to the shock branch ( $T_{\rm policy}=18$ : 0.0102, 0.0260, 0.0819), while the mechanism-aware branch is negative ( $ΔS_{\rm mech}(18)=-0.0078$ , $ΔS_{\rm mech}(38)=-0.0134$ ). A subgroup analysis by gender quantifies scenario-induced survival gaps via bootstrap; contrasts are directionally stable but small. Results are not causally identified; they demonstrate the framework's capacity for internal structural scenario comparison under observational data constraints.

Figures & tables

Figure 1: Framework overview: from raw LMS interaction logs and administrative records ( Input ) through person-period construction, discrete-time hazard modeling (RQ1), scenario-based policy simulation (RQ2), to subgroup fairness diagnostics and exportable policy specification (RQ3).

Symbol	Definition
$i$	enrollment (unit of analysis)
$t$	discrete week
$X_{it}$	observed covariates at time $t$
$E_{i}$	primary event indicator
$C_{i}$	censoring time
$\hat{h}_{it}$	predicted weekly hazard

Table 1: Main notation. For the reported OULAD run:

T_{policy}=18

T_{eval\_policy}=38

T_{eval\_metrics}=37

g_{min}=0.05

Quantity	Train	Test
Enrollments	22,815	9,778
Events (Withdrawn)	5,171	2,216
Person-period rows	542,878	232,417

Table 2: Cohort summary and train/test partition after stratified temporal split.

RQ	Direct answer
RQ1	Yes, for temporal risk ranking, with caution in the highest-risk tail. $AUC_{row,test}=0.8396$ ; $\text{ECE}_{15}=0.0012$ ; recalibrated mean-hazard $\text{AUC}_{IPCW}(T_{policy})=0.7748$ .
RQ2	The operator returns a monotone, sign-stable dose–response; a positive shock contrast is definitional given a hazard-reducing scenario, so the reportable content is the auditable response, not the sign. Structural survival contrasts at $T_{policy}=18$ : anchored ( $\delta=0.08$ ) $\Delta S=+0.0108$ ; hypothetical A ( $\delta=0.20$ ) $+0.0274$ ; hypothetical B ( $\delta=0.60$ ) $+0.0865$ ; mech-aware $+0.0121$ .
RQ3	A capability demonstration, not an equity finding: the monitor returns directionally stable but negligible gap changes ( $\|\Delta Gap\|\approx 5\times 10^{-4}$ ). Gender (F $-$ M): $\Delta Gap(18)=-0.000508$ CI $(-0.000662,-0.000329)$ . Disability (Y $-$ N): $\Delta Gap(18)=-0.000402$ CI $(-0.000661,-0.000106)$ . All four bootstrap CIs exclude zero; detectability at this magnitude does not imply an actionable equity change.

Table 3: Direct answers to RQ1–RQ3. Group coding: F=female, M=male; Y=disability declared, N=no declaration. Bootstrap:

B=500

, percentile 95% CI, seed 42. Horizons:

T_{policy}=18

;

T_{eval}\equiv T_{eval\_metrics}=37

Figure 2: Weekly hazard calibration on the test set (15 equal-width bins; 95% Wilson CI per bin; bin counts annotated). Bins 8–10 contain

\leq 9

rows each (Table 4 ). Aggregate calibration governed by Bin 0 (

n=230{,}157

;

\text{ECE}_{15}=0.0012

Bin	Risk interval	$n$	Events	Non-events
Bin 8	[0.533, 0.600)	9	3	6
Bin 9	[0.600, 0.667)	7	0	7
Bin 10	[0.667, 0.733)	3	3	0

Table 4: Support of highest-risk calibration bins on test rows.

Figure 3: Mean survival trajectories on the test set (

n=9{,}778

enrollments). Lines show baseline, shock (hypothetical A,

\delta_{shock}=0.20

, shown for visual clarity), and mechanism-aware scenario means. Vertical dotted lines mark

T_{policy}=18

and

T_{eval\_policy}=38

. Primary reported scenario is the anchored conservative (

\delta_{shock}=0.08

, reported numerically in Table 5 ).

Figure 4: Censoring survival

\hat{G}(t)

on the test set (

N=9{,}778

enrollments). Vertical lines mark the three evaluation horizons; horizontal line at

g_{min}=0.05

marks the IPCW stability cutoff.

Scenario	$\delta_{shock}$	$\Delta S_{shock}(18)$	$\Delta S_{shock}(38)$	$\Delta S_{mech}(18)$	$\Delta S_{mech}(38)$
Anchored conservative	0.08	0.0108	0.0105	+0.0121	+0.0083
Hypothetical A	0.20	0.0274	0.0266	+0.0121	+0.0083
Hypothetical B	0.60	0.0865	0.0834	+0.0121	+0.0083

Table 5: RQ2 summary by scenario at

T_{policy}=18

and

T_{eval\_policy}=38

(point estimates). Bootstrap 95% CIs (

B{=}500

, seed 42, percentile) exported to rq2_policy_bootstrap_ci.csv .

Model family	AUC ${}_{IPCW}(T_{policy})$	AUC ${}_{IPCW}(T_{eval\_metrics})$
Temporal hazard †	0.4768	0.4950
Temporal hazard + IPCW †	0.5220	0.5291
Enrollment-level LR (static)	0.6586	0.6479
Non-temporal RSF	0.6634	0.6520
DeepHit (static enrollment only)	0.6228	0.5947
Temporal hazard (recalibrated)	0.7748	0.7714

Table 6: Cross-family IPCW benchmark at reported horizons. † Product-form survival rank inversion (Proposition 1 ); AUC

<0.5

expected. Values are point estimates from a single stratified temporal holdout.

Attribute	Horizon	$\Delta Gap(T)$	95% bootstrap CI
Gender (F $-$ M)	$T_{policy}=18$	$-0.000508$	$(-0.000662,\,-0.000329)$
	$T_{eval\_metrics}=37$	$-0.000567$	$(-0.000715,\,-0.000407)$
Disability (Y $-$ N)	$T_{policy}=18$	$-0.000402$	$(-0.000661,\,-0.000106)$
	$T_{eval\_metrics}=37$	$-0.000446$	$(-0.000687,\,-0.000194)$

Table 7: RQ3 summary:

\Delta Gap(T)

with bootstrap uncertainty (

B=500

, seed 42, percentile 95% CI). Group1

-

Group0: gender Group0=M, Group1=F; disability Group0=N, Group1=Y.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Mechanism-aware (all scenarios): 0.01473 [0.01446, 0.01501]
Scenario	$\Delta S^{\dagger}_{shock}(18)$ mean	95% CI lo	95% CI hi
Anchored conservative	0.00974	0.00962	0.00986
Hypothetical A	0.02467	0.02437	0.02498
Hypothetical B	0.07753	0.07650	0.07854

Appendix

Table 8: Bootstrap uncertainty for RQ2 at

T_{policy}=18

(

B=500

, seed 42, percentile). † Bootstrap means differ from Table 5 point estimates. Full 19-scenario table: rq2_policy_bootstrap_ci.csv .

Attribute	Horizon	$\widehat{\Delta Gap}$	95% bootstrap CI	$P(\Delta Gap{<}0)$
Gender (F $-$ M)	$T_{policy}=18$	$-0.000505$	$(-0.000662,\,-0.000329)$	100.0%
	$T_{eval}=37$	$-0.000566$	$(-0.000715,\,-0.000407)$	100.0%
Disability (Y $-$ N)	$T_{policy}=18$	$-0.000391$	$(-0.000661,\,-0.000106)$	99.4%
	$T_{eval}=37$	$-0.000441$	$(-0.000687,\,-0.000194)$	99.8%

Appendix

Table 9: Bootstrap distribution summary of

\Delta Gap(T)

(

B=500

, seed 42).

P(\Delta Gap{<}0)

: fraction of replicates with negative gap reduction.

Group	AUC row	Brier	ECE
Female (F)	0.8391	0.0089	0.0018
Male (M)	0.8407	0.0098	0.0020
No disability (N)	0.8401	0.0090	0.0016
Disability (Y)	0.8376	0.0136	0.0059

Appendix

Table 10: By-group row-level diagnostics. Gender (top); disability (bottom).

Anchor variant	Cens. AUC (test)	Capped-weight share
Conservative ( anchor_-1 )	0.9163	0.5029
Default ( anchor_0 )	0.9299	0.6576

Appendix

Table 11: Censoring anchor sensitivity: censoring model test AUC and capped-weight share.

Steps	Script(s)	Role
1	A_01 , A_02	Load OULAD; build enrollment backbone
2–4	A_03	Endpoint, censoring anchor, $t^{final}$
5	A_04	Person-period weekly frame
6	A_05	Weekly covariate engineering
7	A_06	Stratified temporal split
8–9	B_00	Hazard model, predictions

Appendix

Table 12: Mapping of Algorithm 1 steps to pipeline scripts.

Explore similar work

A Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics
Apr 10, 2026Rafael da Silva, Jeff Eicher, Gregory LongoEducational Data Mining Survival Analysis
Applied Data Science Program, Eastern University, St. Davids, PA, USA.
From Student Risk Prediction to SC2R: Semantics-Constrained Counterfactual Recourse for Educational Decision Support
Aug 18, 2026Ngoc Luyen Le, Marie-Hélène Abel, Bertrand LaforgeAI-Assisted Decision Making Counterfactual Explanations
Gamaizer, 93340 Le Raincy, France. · Université de Technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne Cedex, France. · Sorbonne Université, CNRS UMR 7585, LPMHE (Laboratoire de Physique Nucléaire et des Hautes Énergies), 75252 Paris cedex 05, France
A Time-Synchronous Prequential Evaluation Framework for Next-Week Student Disengagement Prediction from Moodle LMS Logs
Oct 12, 2026David Otoo-Arthur, Terence L. van ZylEducational Data Mining Human Behavior Prediction
University of the Witwatersrand, Johannesburg, South Africa · University of Johannesburg, Johannesburg, South Africa

Apr 10, 2026J/K move · Enter open · S save

A Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics

Rafael da Silva, Jeff Eicher, Gregory Longo

Applied Data Science Program, Eastern University, St. Davids, PA, USA.

Educational Data Mining Survival Analysis

Open paper AskPDF arXiv

Student dropout is a persistent concern in Learning Analytics, yet comparative studies frequently evaluate predictive models under heterogeneous protocols, prioritizing discrimination over temporal interpretability and calibration. This study introduces a survival-oriented benchmark for temporal dropout risk modelling using the Open University Learning Analytics Dataset (OULAD). Two arms are compared: Family A: Dynamic Weekly, with models in person-period representation, and Family B: Static Early-Window, with an expanded roster of families: tree-based survival, parametric, and neural models. The evaluation protocol integrates four analytical layers: predictive performance, ablation, explainability, and calibration. Results are reported within each family separately, because a single numerical cross-family ranking would conflate genuine model differences with artifacts of temporal representation, to which survival metrics are known to be sensitive. Within Family B, Random Survival Forest showed the highest point estimates for time-dependent concordance and the lowest Brier scores across all three horizons; within Family A, Poisson Piecewise-Exponential showed the lowest point estimate for integrated Brier score within a tight five-model cluster. No-refit bootstrap resampling qualifies these positions as directional signals, not claims of strict superiority. Ablation and explainability analyses converged, across all models, on a shared finding: the dominant predictive signal was not primarily demographic or structural, but temporal and behavioral. Calibration corroborated this pattern in the better-discriminating models, except for XGBoost AFT, the sole outlier (analyzed in the Discussion). These results support unified, multi-dimensional benchmarking in Learning Analytics and situate dropout risk as a temporal-behavioral process rather than a function of static background attributes.

Abstract

Explore similar work

A Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics

From Student Risk Prediction to SC2R: Semantics-Constrained Counterfactual Recourse for Educational Decision Support

A Time-Synchronous Prequential Evaluation Framework for Next-Week Student Disengagement Prediction from Moodle LMS Logs