A Time-Synchronous Prequential Evaluation Framework for Next-Week Student Disengagement Prediction from Moodle LMS Logs

David Otoo-Arthur; Terence L. van Zyl

cs.LGOct 12, 2026

A Time-Synchronous Prequential Evaluation Framework for Next-Week Student Disengagement Prediction from Moodle LMS Logs

Authors: David Otoo-Arthur, Terence L. van Zyl

Organizations: University of the Witwatersrand, Johannesburg, South Africa · University of Johannesburg, Johannesburg, South Africa

Abstract

Retrospective evaluation of early-warning models may not reflect information available when predictions are issued and outcomes become observable. We developed a time-synchronous prequential framework for online prediction of next-week student disengagement, operationalised as inactivity onset, from Moodle logs. Ordered by calendar time, models were updated only with available outcomes and remained unchanged for all predictions at each prediction cutoff; development, calibration and evaluation were chronologically separate. We compared stochastic gradient descent logistic regression (SGD-LR), online sequential extreme learning machine (OSELM) and echo state network (ESN) across predictive performance, fixed, frozen and rolling threshold policies, calendar-time variation and four behavioural feature sets. Held-out evaluation comprised 49,329 predictions with 37.68% inactivity-onset prevalence. Discrimination was similar (average precision 0.665-0.679; ROC-AUC 0.755-0.774). OSELM had the highest recall, SGD-LR the highest precision and specificity, and ESN the highest Matthews correlation coefficient and balanced accuracy. Lower thresholds increased recall and alert volume, whereas rolling adaptation did not improve MCC over frozen thresholds. Weekly performance and calibrated-probability distributions varied without sustained deterioration. During model selection, cumulative-and-temporal features achieved the highest individual-set AP, with modest gains from combining all feature groups. These findings indicate that streaming early-warning evaluation should consider calibration, threshold-dependent performance, alert burden and calendar-time variation alongside aggregate discrimination.

Figures & tables

Figure 1: Time-synchronous prequential evaluation framework for next-week inactivity-onset prediction.

Feature group	$p$	Measures
Current activity	7	Log-transformed counts of total events, views, submissions, attempts, downloads, posts, and other actions in the current week.
Cumulative and temporal	8	Week-to-week change in log events; positive-change indicator; mean and standard deviation of log events to date; active-week rate; log mean event count to date; and the mean and standard deviation of log events over the most recent four weeks.
Risk history	7	Log active-streak length; log preceding-inactivity length; log maximum inactivity run to date; inactive-week rate; log inactivity-spell count; reactivation after at least two inactive weeks; and log reactivation count.
All features	22	All current-activity, cumulative-and-temporal, and risk-history predictors.

Table 1: Behavioural feature groups used during model development.

Model	Development search	Selected configuration
SGD-LR	Four feature sets; $\eta_{0}\in\{0.01,0.03,0.10\}$ ; $\lambda\in\{10^{-4},10^{-3},10^{-2}\}$ .	All 22 features; $\eta_{0}=0.01$ ; $\lambda=0.01$ ; $\delta=0.001$ .
OSELM	Four feature sets; hidden units $\in\{16,32,64\}$ ; ridge $\in\{0.1,1,10\}$ .	All 22 features; 64 hidden units; ridge $=10$ ; input scale $=1$ .
ESN	Four feature sets; reservoir units $\in\{32,64\}$ ; spectral radius $\in\{0.5,0.9\}$ ; leak rate $\in\{0.3,0.7\}$ ; ridge $\in\{1,10\}$ .	All 22 features; 64 reservoir units; spectral radius $=0.5$ ; leak rate $=0.7$ ; ridge $=10$ ; input scale $=1$ .

Table 2: Development search spaces and selected model configurations.

Model	$\tau$	AP	ROC- AUC	MCC	F1	Precision	Recall	Specificity	Balanced accuracy
SGD-LR	0.45	0.665	0.755	0.396	0.603	0.656	0.557	0.823	0.690
OSELM	0.35	0.670	0.766	0.405	0.643	0.601	0.692	0.722	0.707
ESN	0.40	0.679	0.774	0.423	0.636	0.648	0.625	0.795	0.710

Table 3: Held-out performance at model-specific frozen thresholds (

n=49{,}329

; prevalence

=37.68\%

); ESN had the highest point estimates for AP, ROC-AUC, MCC and balanced accuracy.

Model	Policy	$\tau$	MCC	Recall	Alert rate (%)
SGD-LR	Fixed	0.300	0.346	0.736	51.37
SGD-LR	Fixed	0.500	0.386	0.489	26.90
SGD-LR	Frozen MCC	0.450	0.396	0.557	32.01
SGD-LR	Rolling MCC	0.450	0.387	0.551	31.86
OSELM	Fixed	0.300	0.392	0.738	48.64
OSELM	Fixed	0.500	0.400	0.545	30.75

Table 4: Threshold-policy performance; lower thresholds increased recall and alert rate, while rolling thresholds did not improve aggregate MCC over frozen thresholds. For rolling policies,

\tau

is the observation-weighted mean threshold.

Figure 2: Weekly held-out AP (a) and MCC (b) for the three models; MCC varied more across weeks than AP, with no significant temporal trend in either measure. Breaks indicate non-estimable weeks; hollow markers denote

n<50

Figure 3: Weekly prevalence, mean calibrated probability and probability-distribution drift relative to calibration; drift was episodic and broadly aligned across models, with the largest shifts in July and December 2017. Breaks indicate weeks with

n<50

Model	Feature set	No. features	AP	ROC-AUC	Raw Brier
SGD-LR	Current	7	0.593	0.711	0.239
SGD-LR	Cumulative and temporal	8	0.696	0.759	0.199
SGD-LR	Risk history	7	0.664	0.727	0.217
SGD-LR	All features	22	0.704	0.769	0.192
OSELM	Current	7	0.596	0.725	0.213
OSELM	Cumulative and temporal	8	0.684	0.759	0.205

Table 5: Model-selection performance across four feature sets; the combined 22-feature set achieved the highest AP and ROC-AUC and lowest raw Brier score for all models.

Explore similar work

When Can We Trust Early Warnings? Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs
May 25, 2026Ngoc Luyen Le, Marie-Hélène Abel, Bertrand LaforgeData Leakage AI in Education
Gamaizer, 93340 Le Raincy, France · Université de technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne Cedex, France · Sorbonne Université, CNRS UMR 7585, LPMHE (Laboratoire de Physique Nucléaire et des Hautes Énergies), 75252 Paris cedex 05, France
From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning
May 12, 2026Eric S. Qiu, Danielle R. Thomas, Boyuan Guo +2Intelligent Tutoring Systems Time Series Forecasting
Cornell University · Ithaca, NY, USA · Carnegie Mellon University +1
Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning
Aug 25, 2026Alperen Kantarci, Visvanathan Ramesh, Gemma RoigOnline Learning Educational Data Mining
Institute of Computer Science Goethe University Frankfurt Frankfurt am Main, Germany · Institute of Computer Science Goethe University Frankfurt The Hessian Center for Artificial Intelligence Frankfurt am Main, Germany

May 25, 2026J/K move · Enter open · S save

When Can We Trust Early Warnings? Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs

Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge

Gamaizer, 93340 Le Raincy, France · Université de technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne Cedex, France · Sorbonne Université, CNRS UMR 7585, LPMHE (Laboratoire de Physique Nucléaire et des Hautes Énergies), 75252 Paris cedex 05, France

Data Leakage AI in Education Educational Data Mining Temporal Data Leakage

Open paper AskPDF arXiv

Early-warning models built from Learning Management System (LMS) logs aim to predict end-of-course outcomes early enough to enable timely learner support. However, reported "early" performance is often inflated by temporal leakage. This occurs when the pipeline uses information that would not yet be available at the time of prediction. We formalize cutoff-based early outcome prediction under a temporal availability constraint and introduce LEAP (Leakage-Excluded Early-Availability Protocol), which enforces cutoff-first truncation prior to joins and aggregation and audits feature provenance to prevent post-cutoff evidence from entering the benchmark. We instantiate LEAP on the public Open University Learning Analytics Dataset (OULAD) as a multi-step protocol for leakage-controlled evaluation across weekly cutoffs. Using several standard learning methods, we evaluate performance using ROC-AUC, PR-AUC, Brier score, and [email protected]. Results show improving performance as the observation window expands, with a marked gain around week~3; Random Forest performs best at the earliest cutoffs, while Gradient Boosting dominates thereafter. Leakage ablations further show that temporal violations, especially through assessment information, can inflate apparent "early" performance.

Abstract

Explore similar work

When Can We Trust Early Warnings? Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs

From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning