Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gating

Saniah Kayenat Chowdhury; Muhammad E. H. Chowdhury

cs.CVApr 11, 2026

Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gating

Authors: Saniah Kayenat Chowdhury, Muhammad E. H. Chowdhury

Organizations: Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Electrical Engineering, Qatar University, Doha, 2713, Qatar

Abstract

Student engagement is crucial for improving learning outcomes in group activities. Highly engaged students perform better both individually and contribute to overall group success. However, most existing automated engagement recognition methods are designed for online classrooms or estimate engagement at the individual level. Addressing this gap, we propose DualEngage, a novel two-stream framework for group-level engagement recognition from in-classroom videos. It models engagement as a joint function of both individual and group-level behaviors. The primary stream models person-level motion dynamics by detecting and tracking students, extracting dense optical flow with the Recurrent All-Pairs Field Transforms network, encoding temporal motion patterns using a transformer encoder, and finally aggregating per-student representations through attention pooling into a unified representation. The secondary stream captures scene-level spatiotemporal information from the full video clip, leveraging a pretrained three-dimensional Residual Network. The two-stream representations are combined via softmax-gated fusion, which dynamically weights each stream's contribution based on the joint context of both features. DualEngage learns a joint representation of individual actions with overarching group dynamics. We evaluate the proposed approach using fivefold cross-validation on the Classroom Group Engagement Dataset developed by Ocean University of China, achieving an average classification accuracy of 0.9621+/-0.0161 with a macro-averaged F1 of 0.9530+/-0.0204. To understand the contribution of each branch, we further conduct an ablation study comparing single-stream variants against the two-stream model. This work is among the first in classroom engagement recognition to adopt a dual-stream design that explicitly leverages motion cues as an estimator.

Figures & tables

Figure 1: Overview of the proposed DualEngage. In the primary stream, optical flow is computed using RAFT; per-student motion tubes are cropped using tracked identities and modeled with a transformer encoder to yield a unified motion embedding before being aggregated via attention pooling. In the secondary stream, a 3D ResNet extracts scene-level features from the full spatiotemporal clip. The two feature sets are combined using gated fusion and passed to an MLP classification head to predict overall engagement levels during group activities.

Engagement Level	Number of Clips	Frame Rates (fps)	Duration (s)
High	2,648	22-30	5-15
Medium	1,012	15-30	5-15
Low	3,040	2-30	2-15

Table 1: Dataset Summary (Before Pre-processing)

Engagement Level	Number of Clips	Frame Rates (fps)	Duration (s)
High	2,070	30	10
Medium	1,012	30	10
Low	2,834	30	10

Table 2: Dataset summary (After Pre-processing)

Figure 2: Frame-by-frame student detection and tracking in classroom video. Top (round-table layout): a student enters at frame 208, and is assigned a new ID. Bottom (chessboard layout): two students are intermittently missed being detected due to congestion and occlusion. Colored boxes denote track IDs across frames.

Figure 3: Optical flow map between two consecutive frames.

Figure 4: Overview of the motion extraction process from classroom video to student-specific sequences. The raw video is first converted into an optical flow map, followed by the formation of per-student flow sequences. Temporal dynamics are learned using a transformer encoder, which produces the stacked feature representation.

Parameter	Value
Input clip length	$T{=}300$
Frame Size (scene-level stream)	$224{\times}224$
Frame Size (person-level stream)	$128{\times}128$
Batch size / workers	$16$ / $4$
Optimizer / LR	Adam / $1\!\times\!10^{-4}$
Loss	Cross-entropy with inverse-frequency class weights

Table 3: Training Setup

Fold	Accuracy	Macro Precision	Macro Recall	Macro F1 Score
Fold 1	0.9704	0.9591	0.9762	0.9670
Fold 2	0.9488	0.9240	0.9420	0.9318
Fold 3	0.98	0.9838	0.9682	0.9752
Fold 4	0.9696	0.9640	0.9565	0.96
Fold 5	0.9416	0.9266	0.9378	0.9311
Mean $\pm$ Std	$0.9621\pm 0.0161$	$0.9515\pm 0.0257$	$0.9561\pm 0.0165$	$0.9530\pm 0.0204$

Table 4: Five-fold results on group engagement level recognition. ( Bold = best per column.)

Figure 5: Confusion matrices representing the classification performance of the DualEngage architecture across five-fold cross-validation for group engagement recognition.

Figure 6: Representative frames sampled across classroom clips in a chessboard layout. (a) High engagement example: students maintain attention toward the instructor/board, showing consistent orientation and participation. (b) Low engagement example: several students exhibit reduced attention, such as head-down posture or looking away, indicating weaker collective involvement.

Fold	Accuracy	Macro Precision	Macro Recall	Macro F1 Score
Primary Stream Only	0.5321	0.5257	0.4920	0.5083
Secondary Stream Only	0.6214	0.6075	0.5894	0.5983
DualEngage w/o Temporal Encoder	0.5167	0.5257	0.5167	0.5043
DualEngage LSTM	0.6017	0.7157	0.6017	0.5747
DualEngage 1DCNN	0.4950	0.5066	0.4950	0.4949
DualEngage + R(2+1)D backbone	0.9012	0.9145	0.8824	0.8982

Table 5: Ablation study of DualEngage comparing the full model with single-stream baselines, temporal modeling removal, and alternative backbone choices.)

Explore similar work

Aug 25, 2026

Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning

The prediction of student engagement from the online tutoring videos is difficult because engagement is a multidimensional construct comprising distinct behavioral, emotional, and cognitive states. A reliable prediction requires bringing together different types of behavioral signals as well as expressive cues. Through our analysis of the CASED dataset, it is clear that engagement prediction gets even harder due to the high inter-person variability as well as the subjectivity of the engagement annotation. To tackle these challenges, we develop a multimodal framework that integrates the implicit spatiotemporal features extracted from pretrained video, audio, and image encoders along with structured behavioral modalities like head pose, gaze, facial action units, emotion, and wavelet-based audio features. We integrate these modalities via a Perceiver IO latent bottleneck. Moreover, student and instructor personalities are modeled as variational posteriors over learnable embeddings to enable partial pooling across participants. We employ evidential regression and spectral-normalized Gaussian process classification heads for uncertainty-aware prediction to further improve robustness and calibration. Benchmark on the CASED challenge test set shows that all participating methods converge near random-chance performance, revealing the difficulty of the dataset. In this highly ambiguous regime, our framework achieves competitive performance while uniquely offering well-calibrated uncertainty metrics, demonstrating that reliable risk-quantification is an essential prerequisite for deploying engagement models in real-world educational tools.

Alperen Kantarci, Visvanathan Ramesh, Gemma Roig

Institute of Computer Science Goethe University Frankfurt Frankfurt am Main, Germany · Institute of Computer Science Goethe University Frankfurt The Hessian Center for Artificial Intelligence Frankfurt am Main, Germany

Online Learning Educational Data Mining Uncertainty-Aware Multimodal Fusion

Ask PDF

May 2, 2026

EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

Engagement, which links to attentional, emotional, and cognitive dimensions, plays an important role in learning. In online and video-based learning environments, learners often need to regulate their own interactions with instructional materials. Measuring and reflecting on engagement can therefore support both learners and adaptive learning systems. In this study, we use wearable and camera-based sensing devices to collect physiological and motion signals, including PPG, ECG, EDA, EEG, IMU, heart rate, temperature, and eye-tracking data, to estimate learner engagement. We conducted a user study with 16 participants in a video-based learning scenario, where participants completed learning tasks and provided repeated in-situ engagement-probe ratings on a 1-5 scale, reporting how difficult it was to pay attention during the preceding minute. We establish a benchmark for engagement estimation, compare different sensing modalities, and further analyze the feasibility and effectiveness of multimodal modeling for characterizing learner engagement. Across participant-based cross-validation, the modality-aware reference model achieves an MAE of 0.80, 83.18% within-1 accuracy, 70.96% binary accuracy, and 62.90% binary Macro-F1, outperforming sensor-free, statistical, deep temporal, foundation-model, and LLM-based baselines. Our results suggest that fine-grained engagement estimation is feasible but inherently noisy, and that the preferred sensing configuration depends on the target metric and practical sensing burden. We release the EduGage dataset as the primary contribution of this work, including synchronized multimodal sensor signals, probe-aligned engagement-probe ratings, video metadata, quizzes, and study materials, to support reproducible research on fine-grained sensor-based engagement modeling in self-guided learning.

Zikang Leng, Edan Eyal, Yingtian Shi +3

College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA · School of Computing Technologies, RMIT University, Melbourne, Australia

Wearable Sensing User Modeling Online Learning Multimodal Sensor Fusion

Ask PDF

Apr 6, 2026

Group-DINOmics: Incorporating People Dynamics into DINO for Self-supervised Group Activity Feature Learning

This paper proposes Group Activity Feature (GAF) learning without group activity annotations. Unlike prior work, which uses low-level static local features to learn GAFs, we propose leveraging dynamics-aware and group-aware pretext tasks, along with local and global features provided by DINO, for group-dynamics-aware GAF learning. To adapt DINO and GAF learning to local dynamics and global group features, our pretext tasks use person flow estimation and group-relevant object location estimation, respectively. Person flow estimation is used to represent the local motion of each person, which is an important cue for understanding group activities. In contrast, group-relevant object location estimation encourages GAFs to learn scene context (e.g., spatial relations of people and objects) as global features. Comprehensive experiments on public datasets demonstrate the state-of-the-art performance of our method in group activity retrieval and recognition. Our ablation studies verify the effectiveness of each component in our method. Code: https://github.com/tezuka0001/Group-DINOmics.

Ryuki Tezuka, Chihiro Nakatani, Norimichi Ukita

Toyota Technological Institute

Representation Learning Human Activity Recognition Self-Supervised Visual Representation Learning

Ask PDF