cs.HCMar 20, 2026

Behavioral Engagement in VR-Based Sign Language Learning: Visual Attention as a Predictor of Performance and Temporal Dynamics

Authors: Davide Traini, José Manuel Alcalde-Llergo, Mariana Buenestado-Fernández, Domenico Ursino, Enrique Yeguas-Bolívar

Organizations: Department of Information Engineering, Università delle Marche, Ancona, 60121, Italy · Department of Computer Science and AI, Universidad de Córdoba, Córdoba, 14071, Spain · Department of Education, Universidad de Córdoba, Córdoba, 14071, Spain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This study analyzes behavioral engagement in SONAR, a virtual reality application designed for sign language training and validation. We focus on three automatically derived engagement indicators (Visual Attention (VA), Video Replay Frequency (VRF), and Post-Playback Viewing Time (PPVT)) and examine their relationship with learning performance. Participants completed a self-paced Training phase, followed by a Validation quiz assessing retention. We employed Pearson correlation analysis to examine the relationships between engagement indicators and quiz performance, followed by binomial Generalized Linear Model (GLM) regression to assess their joint predictive contributions. Additionally, we conducted temporal analysis by aggregating moment-to-moment VA traces across all learners to characterize engagement dynamics during the learning session. Results show that VA exhibits a strong positive correlation with quiz performance,followed by PPVT, whereas VRF shows no meaningful association. A binomial GLM confirms that VA and PPVT are significant predictors of learning success, jointly explaining a substantial proportion of performance variance. Going beyond outcome-oriented analysis, we characterize temporal engagement patterns by aggregating moment-to-moment VA traces across all learners. The temporal profile reveals distinct attention peaks aligned with informationally dense segments of both training and validation videos, as well as phase-specific engagement dynamics, including initial acclimatization, oscillatory attention cycles during learning, and pronounced attentional peaks during assessment. Together, these findings highlight the central role of sustained and strategically allocated visual attention in VR-based sign language learning and demonstrate the value of behavioral trace data for understanding and predicting learner engagement in immersive environments.

Figures & tables

Hearing ParticipantEssie (Deaf Avatar)
1. Can you read my lips?Yes.
2. Hi!Hi! How are you?
3. Is this seat available?Yes, please, take it.
4. What’s your name?My name is Essie.
5. Do you have a name sign?, 
What’s your name sign?My sign is (ISENSE sign).
Table 1: Interactions between the hearing participant and the deaf avatar during the VR experience.
Figure 1: \hl Flow diagram of the SONAR experience.
Figure 2: Distribution of correct answers in the Validation phase.
Figure 3: Relationships between quiz performance and the three behavioral engagement metrics: ( a ) Post-Playback Viewing Time (PPVT); ( b ) Visual Attention (VA); ( c ) Video Replay Frequency (VRF).
PredictorCoefficientSEzzpp -ValueOR
Intercept1.81940.08621.104<< 0.001—
VAz\text{VA}_{z}0.75580.0948.040<< 0.0012.129
PPVTz\text{PPVT}_{z}0.43590.0884.931<< 0.0011.546
VRFz\text{VRF}_{z}−- 0.03390.076−- 0.4480.6540.967
Table 2: Binomial GLM for the proportion of correct answers. Predictors are standardized ( zz -scores). Coefficients are on the log-odds scale with Standard Errors (SE), zz -statistics, pp -values, and Odds Ratios (OR).
Figure 4: Predicted proportion of correct answers from the binomial GLM as a function of each standardized engagement metric. The red curve shows the model-predicted probability of a correct response when varying one predictor while holding the others at their mean values: ( a ) Post-Playback Viewing Time ( PPVTz\text{PPVT}_{z} ). Effect of PPVTz\text{PPVT}_{z} on predicted quiz performance; ( b ) Visual Attention ( VAz\text{VA}_{z} ). Effect of VAz\text{VA}_{z} on predicted quiz performance; ( c ) Video Replay Frequency ( VRFz\text{VRF}_{z} ). Effect of VRFz\text{VRF}_{z} on predicted quiz performance.
Figure 5: Temporal profile of aggregated VA across all participants. The blue curve represents mean VA normalized across video frame progression, with the light blue band indicating the 95% confidence interval. Red stars mark the 24 highest peaks, corresponding to moments of particularly intense visual focus during video playback.
More in this topic

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 61 papers

Explore similar work

CardsList
May 2, 2026

EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

Engagement, which links to attentional, emotional, and cognitive dimensions, plays an important role in learning. In online and video-based learning environments, learners often need to regulate their own interactions with instructional materials. Measuring and reflecting on engagement can therefore support both learners and adaptive learning systems. In this study, we use wearable and camera-based sensing devices to collect physiological and motion signals, including PPG, ECG, EDA, EEG, IMU, heart rate, temperature, and eye-tracking data, to estimate learner engagement. We conducted a user study with 16 participants in a video-based learning scenario, where participants completed learning tasks and provided repeated in-situ engagement-probe ratings on a 1-5 scale, reporting how difficult it was to pay attention during the preceding minute. We establish a benchmark for engagement estimation, compare different sensing modalities, and further analyze the feasibility and effectiveness of multimodal modeling for characterizing learner engagement. Across participant-based cross-validation, the modality-aware reference model achieves an MAE of 0.80, 83.18% within-1 accuracy, 70.96% binary accuracy, and 62.90% binary Macro-F1, outperforming sensor-free, statistical, deep temporal, foundation-model, and LLM-based baselines. Our results suggest that fine-grained engagement estimation is feasible but inherently noisy, and that the preferred sensing configuration depends on the target metric and practical sensing burden. We release the EduGage dataset as the primary contribution of this work, including synchronized multimodal sensor signals, probe-aligned engagement-probe ratings, video metadata, quizzes, and study materials, to support reproducible research on fine-grained sensor-based engagement modeling in self-guided learning.

Zikang Leng, Edan Eyal, Yingtian Shi +3

College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA · School of Computing Technologies, RMIT University, Melbourne, Australia

Wearable SensingUser ModelingOnline LearningMultimodal Sensor Fusion
AskPDF
Jul 31, 2026

Attention-Steered Vision-Language Models for Sign Language Translation

Vision-language models (VLMs) have emerged as a powerful framework for multimodal video understanding. However, they remain limited in the sign language translation task, where we identify a key failure mode of existing VLMbased translators: poor spatial-temporal visual grounding. In particular, we find that standard next-token cross-entropy does not directly provide signal for where and when the model should attend, causing models to overlook sign-relevant regions and frames. To address this challenge, we propose AttnSign, a VLM-based spatial-temporal attention steering framework for sign language translation. AttnSign first introduces spatial attention supervision for sign-relevant regions, such as face and hands, in each frame; then develops an RL-based motion-cadence steering method that encourages the model to explore and focus on sign-level keyframes. Experimental results on How2Sign and OpenASL benchmarks show that our proposed AttnSign consistently outperforms existing methods.

Meibo Hu, Guohao Sun, Annemarie D. Ross +2

Rochester Institute of Technology · National Technical Institute for the Deaf / Rochester Institute of Technology · University of Virginia

Sign Language TranslationVision-Language Models
AskPDF
Jun 8, 2026

Toward Signing Activity Projection in Sign Language Interaction

Social robots must interact robustly not only with users assumed by speech-centered systems but also with diverse users whose communication relies on different modalities, e.g., sign language. One important capability gap is predictive turn-taking with signing users. Although Voice Activity Projection (VAP) has been successfully used to model future voice activity in spoken interaction, it remains unclear whether the framework transfers to sign language interaction. This paper presents an initial transfer study of adapting a VAP architecture to dyadic sign language interaction. Using interaction recordings from the Public DGS Corpus, we derive binary signing activity streams from lexical sign annotations and formulate proxy tasks for turn-taking prediction. The model uses pose-derived hand, eye-region, and mouth-region features extracted for each signer. The results show that SHIFT/HOLD prediction is promising, especially with hand cues, while SHIFT-prediction remains difficult. These findings provide initial evidence for both the promise and the current limitations of transferring predictive turn-taking models from spoken interaction to sign language interaction. Predictive modeling of sign language interaction still requires sign-language-specific event definitions that go beyond speech-derived categories.

Takao Obi, Wang Yusong, Koji Inoue +1

Institute of Science Tokyo, Yokohama 230-0045, Japan · Kyoto University, Yoshida-honmachi, Sakyoku, Kyoto, Japan

Human-AI InteractionTurn-Taking Prediction
AskPDF