Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning

Alperen Kantarci; Visvanathan Ramesh; Gemma Roig

cs.CVAug 25, 2026

Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning

Authors: Alperen Kantarci, Visvanathan Ramesh, Gemma Roig

Organizations: Institute of Computer Science Goethe University Frankfurt Frankfurt am Main, Germany · Institute of Computer Science Goethe University Frankfurt The Hessian Center for Artificial Intelligence Frankfurt am Main, Germany

Abstract

The prediction of student engagement from the online tutoring videos is difficult because engagement is a multidimensional construct comprising distinct behavioral, emotional, and cognitive states. A reliable prediction requires bringing together different types of behavioral signals as well as expressive cues. Through our analysis of the CASED dataset, it is clear that engagement prediction gets even harder due to the high inter-person variability as well as the subjectivity of the engagement annotation. To tackle these challenges, we develop a multimodal framework that integrates the implicit spatiotemporal features extracted from pretrained video, audio, and image encoders along with structured behavioral modalities like head pose, gaze, facial action units, emotion, and wavelet-based audio features. We integrate these modalities via a Perceiver IO latent bottleneck. Moreover, student and instructor personalities are modeled as variational posteriors over learnable embeddings to enable partial pooling across participants. We employ evidential regression and spectral-normalized Gaussian process classification heads for uncertainty-aware prediction to further improve robustness and calibration. Benchmark on the CASED challenge test set shows that all participating methods converge near random-chance performance, revealing the difficulty of the dataset. In this highly ambiguous regime, our framework achieves competitive performance while uniquely offering well-calibrated uncertainty metrics, demonstrating that reliable risk-quantification is an essential prerequisite for deploying engagement models in real-world educational tools.

Figures & tables

Figure 1. Overview of the proposed multimodal engagement framework. Multi-source inputs (student/instructor video, slides, audio, and behavioral cues) are encoded via specialized models and fused using a Perceiver IO block to form the core feature representation (d=768) This is combined with a personality-driven Bayesian Context embedding via a gating mechanism. The final concatenated representation (d=32) feeds into an Evidential Regression head and a GP Classification head to simultaneously predict engagement levels and explicit epistemic uncertainty.

Participant	F1 (Macro)	F1 (Weighted)	Precision	Recall	MCC
saurabhh	0.52	0.60	0.52	0.52	0.04
mohitvu	0.52	0.61	0.52	0.52	0.04
adim66	0.51	0.58	0.51	0.52	0.03
Ours	0.51	0.60	0.51	0.51	0.02
priscalab	0.51	0.60	0.51	0.51	0.01
caymann	0.50	0.58	0.50	0.50	0.01

Table 1. Classification performance of participating methods on the CASED challenge test set.

Participant	RMSE	MSE	MAE	$R^{2}$	Pearson	CCC
saurabhh	0.68	0.46	0.57	-0.01	-0.00	-0.00
mohitvu	0.78	0.61	0.63	-0.33	-0.00	-0.00
Ours	0.68	0.46	0.57	-0.00	-0.02	-0.00
priscalab	0.68	0.47	0.57	-0.02	0.01	0.00
caymann	0.68	0.47	0.57	-0.01	0.03	0.01
KvochurHegel	0.68	0.47	0.57	-0.01	-0.03	-0.00

Table 2. Regression performances on the CASED challenge test set.

(a) Modality ablation — Full model
Model	Modalities	CCC $\uparrow$	RMSE $\downarrow$	MAE $\downarrow$	F1-macro $\uparrow$	MCC $\uparrow$
Full (S)	Student	-0.016 $\pm$ 0.027	1.048 $\pm$ 0.060	0.848 $\pm$ 0.046	0.486 $\pm$ 0.015	-0.002 $\pm$ 0.029
Full (S+I)	Student + Instructor	-0.021 $\pm$ 0.016	1.020 $\pm$ 0.119	0.832 $\pm$ 0.111	0.484 $\pm$ 0.022	-0.011 $\pm$ 0.016
Full (S+I+C)	Student + Instructor + Content	0.018 $\pm$ 0.029	0.943 $\pm$ 0.038	0.755 $\pm$ 0.040	0.498 $\pm$ 0.019	0.002 $\pm$ 0.034
(b) Component ablation — Full modality set (S+I+C)
Base Transformer	S+I+C	0.015 $\pm$ 0.028	0.821 $\pm$ 0.022	0.671 $\pm$ 0.026	0.497 $\pm$ 0.013	0.012 $\pm$ 0.019

Table 3. Ablation study. (a) Modality contribution using the full model (BayesianContext + NIG + SNGP). (b) Component contribution on the three-modality input (S=student video, I=instructor video, C=content). Metrics are averaged over 5 student-grouped cross-validation folds.

Pearson $r$ (engagement vs aleatoric): 0.016; vs epistemic: 0.025.
Engagement Level	N	Mean Pred	Aleatoric $\downarrow$	Epistemic $\downarrow$	Total
1.0–1.8	41	3.45	42.0879	3323.3606	3365.4482
1.9–2.6	657	3.37	33.1598	2623.2993	2656.4590
2.7–3.4	1447	3.36	34.5978	2726.2368	2760.8345
3.5–4.2	2398	3.38	34.7966	2802.9478	2837.7444
4.3–5.0	435	3.42	36.4595	3008.4961	3044.9551

Table 4. NIG uncertainty decomposition by engagement level. Samples are grouped into five equal-width bins on the 1–5 continuous engagement scale.

Entity	Name	KL from Prior $\downarrow$	$\\|\mu\\|_{2}$	N Clips
All students (mean)	—	14.3253	0.0273	116
All instructors (mean)	—	13.1287	0.0167	1659
Top-3 students by KL (most personalized)
Student	laurencedu	15.7200	0.0257	26
Student	pierreyoussef	15.4660	0.0795	50
Student	richard	15.2584	0.0311	55

Table 5. Bayesian context layer posterior statistics. The KL divergence from the population prior and the posterior mean norm are shown.

Explore similar work

EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
May 2, 2026Zikang Leng, Edan Eyal, Yingtian Shi +3Wearable Sensing User Modeling
College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA · School of Computing Technologies, RMIT University, Melbourne, Australia
Dual-Model Prediction of Affective Engagement and Vocal Attractiveness from Speaker Expressiveness in Video Learning
Mar 19, 2026Hung-Yue Suen, Kuo-En Hung, Fan-Hsun TsengAffective Computing Human Behavior Prediction
Technology Application and Human Resource Development, National Taiwan Normal University, Taipei, Taiwan · Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 701, Taiwan
Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models
Apr 6, 2026Dominik Glandorf, Fares Fawzi, Tanja KäserVideo Understanding Multimodal Model Interpretability
EPFL, Switzerland

May 2, 2026J/K move · Enter open · S save

EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

Zikang Leng, Edan Eyal, Yingtian Shi, Jiaman He+2

College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA · School of Computing Technologies, RMIT University, Melbourne, Australia

Wearable Sensing User Modeling Online Learning Multimodal Sensor Fusion

Open paper AskPDF arXiv

Engagement, which links to attentional, emotional, and cognitive dimensions, plays an important role in learning. In online and video-based learning environments, learners often need to regulate their own interactions with instructional materials. Measuring and reflecting on engagement can therefore support both learners and adaptive learning systems. In this study, we use wearable and camera-based sensing devices to collect physiological and motion signals, including PPG, ECG, EDA, EEG, IMU, heart rate, temperature, and eye-tracking data, to estimate learner engagement. We conducted a user study with 16 participants in a video-based learning scenario, where participants completed learning tasks and provided repeated in-situ engagement-probe ratings on a 1-5 scale, reporting how difficult it was to pay attention during the preceding minute. We establish a benchmark for engagement estimation, compare different sensing modalities, and further analyze the feasibility and effectiveness of multimodal modeling for characterizing learner engagement. Across participant-based cross-validation, the modality-aware reference model achieves an MAE of 0.80, 83.18% within-1 accuracy, 70.96% binary accuracy, and 62.90% binary Macro-F1, outperforming sensor-free, statistical, deep temporal, foundation-model, and LLM-based baselines. Our results suggest that fine-grained engagement estimation is feasible but inherently noisy, and that the preferred sensing configuration depends on the target metric and practical sensing burden. We release the EduGage dataset as the primary contribution of this work, including synchronized multimodal sensor signals, probe-aligned engagement-probe ratings, video metadata, quizzes, and study materials, to support reproducible research on fine-grained sensor-based engagement modeling in self-guided learning.

Abstract

Explore similar work

EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

Dual-Model Prediction of Affective Engagement and Vocal Attractiveness from Speaker Expressiveness in Video Learning

Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models