Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

Jakob Schwerter; Loreen Sabel; Judith Bose; Matthew L. Bernacki; Di Xu; Marko Schmellenkamp; Thomas Zeume; Philipp Doebler

cs.CYApr 14, 2026

Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

Authors: Jakob Schwerter, Loreen Sabel, Judith Bose, Matthew L. Bernacki, Di Xu, Marko Schmellenkamp, Thomas Zeume, Philipp Doebler

Organizations: Hector Research Institute of Education Sciences and Psychology, University of Tübingen · TU Dortmund University · University of North-Carolina, Chapel Hill · University of California, Irvine ·

Abstract

STEM dropout rates remain high at universities, particularly in computer science programs with theory-intensive courses. Digital learning environments now capture rich behavioral data that could help identify struggling students early, yet the generalizability of data-driven prediction models across courses and institutions remains uncertain. Guided by self-regulated learning (SRL) theory, this study analyzed multimodal digital-trace data from three undergraduate theoretical computer science courses (N1 = 137, N2 = 104, N3 = 148) at two universities. Weekly SRL-aligned digital-trace indicators were modeled using Elastic Net, Random Forest, and XGBoost to evaluate predictive performance over time and across settings, and model calibration both within and across courses. Early prediction of at-risk students was feasible, with SRL-related behaviors such as time management, effort regulation, and sustained engagement emerging as key predictors. While Random Forest achieved the highest in-sample accuracy, Elastic Net generalized more robustly across contexts. Out-of-sample accuracy and calibration declined between institutions with different base rates, underscoring the contextual nature of predictive analytics in higher education. These findings suggest that digital learning traces enable early identification of at-risk students within courses, but generalizing predictive models beyond their original context requires caution, particularly if the at-risk rates differ between contexts.

Figures & tables

Panel of Figure 1 — Figure 1: Graphical overview of course overlaps

Course structure
Aspect	TCS-A	Logic-A	Logic-B
Lecture	2 x 90 min per week	1 x 90 min per week	1 x 90 min per week
Exercise sessions	90 min every week	90 min every other week	90 min every other week
Tutorials	no	no	90 min every week
Slides available	yes	yes	yes
Forum available	yes	yes	yes

Table 1: Overview over the differences of the three studied courses.

Variable	Abbreviation	Explanation
Task Engagement and Coverage
Number of solution deliveries	eng1	Counts of solution submissions for digital practice tasks (relative to the cohort median).
Time on page in minutes over all pages	eng5	Total time spent on pages in the digital practice system, used as an indicator of behavioral engagement with course materials.
Proportion of pages with submission	par2	Fraction of full pages for which a student submits a solution (coverage of assigned practice).
Time Management (Temporal Regulation and Pacing)
Time to deadline in days	eng2	Time between students’ first interaction with a task and its deadline, indexing early versus late initiation.

Table 2: Overview of variables and explanations

Panel of Figure 2 — Figure 2: Exam distributions

Panel of Figure 3 — Figure 3: Data aggregation strategies.

Figure 4: Weekly submission rates.

Figure 5: Weekly model quality metrics for Elastic Net, XGBoost, and Random Forest — Course Logic-A as reference

Figure 6: Variable importance across weeks and courses – Elastic Net all courses

Panel of Figure 7 — Figure 7: Calibration plots (Week 3 & 6, Elastic Net)

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Abbreviation	Description
P	Paper-based practice
I	Incentivized digital practice
NI	Non-incentivized digital practice
Mean	Mean over all weeks
SD	Standard deviation over all weeks
redu1	Tasks of before the deadline

Appendix

Table 3: Complete list of generated variables prior to omission for multicollinearity

Figure 8: Weekly model quality metrics for Elastic Net, XGBoost, and Random Forest — Course TCS-A as reference, progressive feature aggregation, source-threshold transfer

Figure 9: Weekly model quality metrics for Elastic Net, Random Forest, and XGBoost—Course Logic-B as reference, progressive feature aggregation, source-threshold transfer

Figure 10: Variable importance/selection across weeks and courses – Elastic Net Course Logic-A

Figure 11: Variable importance/selection across weeks and courses – Random Forest all courses

Figure 12: Variable importance/selection across weeks and courses – XGBoost all courses

Figure 13: Calibration plots (Week 3–6, Random Forest)

Figure 14: Calibration plots (Week 3–6, XGBoost)

Self-Regulated Learning

See all 8 papers

Jointly Predicting Courses and Grades Using a Transformer-Based Model
Aug 13, 2026Paul SavalaMulti-Task Learning Educational Data Mining
Mathematics, St. Edward’s University, 3001 S Congress Ave, Austin, 78704, TX, USA. · S Congress Ave
A Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics
Apr 10, 2026Rafael da Silva, Jeff Eicher, Gregory LongoEducational Data Mining Survival Analysis
Applied Data Science Program, Eastern University, St. Davids, PA, USA.
Analysis and Prediction of At-Risk Students Using Machine Learning Algorithms
May 24, 2026Soheila Gheisari, Hamid SalarianAI in Education Educational Data Mining
Department of Information Technology, Sydney International School of Technology and Commerce, Sydney, NSW 2000, Australia

Aug 13, 2026J/K move · Enter open · S save

Jointly Predicting Courses and Grades Using a Transformer-Based Model

Paul Savala

Mathematics, St. Edward’s University, 3001 S Congress Ave, Austin, 78704, TX, USA. · S Congress Ave

Multi-Task Learning Educational Data Mining

Open paper AskPDF arXiv

Existing predictive models in learning analytics often treat student academic history as a simple sequence, overlooking the concurrent nature of courses taken within a semester. This simplification can lead to inaccurate performance predictions, particularly for students with heavy or challenging course loads. This paper introduces a TRansformer for Academic Course-grade Estimation (TRACE) that addresses this limitation by jointly predicting both the set of courses a student will take and their corresponding grades for an upcoming semester. Our approach encodes courses on a per-semester basis to capture the effects of course concurrency and utilizes a novel loss function combining course-set prediction with grade prediction. We demonstrate that predicting courses taken in addition to the grades in those courses leads to significant improvements in prediction quality. Trained on ten years of institutional data, our joint prediction model reduces mean absolute error by nearly 50% compared to an identical architecture that predicts grades alone. The model also outperforms traditional LSTM-based sequential models, as well as graph neural network-based approaches, and offers natural ways to incorporate student attribute data. This work demonstrates the utility of modern neural architectures for creating interpretable models that can be adapted to new institutions via retraining and recalibration, as well as the importance of key techniques, such as predicting courses taken during training. We discuss how this model could be incorporated into early detection systems at institutions of higher education.

Abstract

Self-Regulated Learning

Explore similar work

Jointly Predicting Courses and Grades Using a Transformer-Based Model

A Unified Survival Benchmark for Temporal Dropout Risk Prediction in Learning Analytics

Analysis and Prediction of At-Risk Students Using Machine Learning Algorithms