Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

Dominik Glandorf; Fares Fawzi; Tanja Käser

cs.AIApr 6, 2026

Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

Authors: Dominik Glandorf, Fares Fawzi, Tanja Käser

Organizations: EPFL, Switzerland

Abstract

Learners' use of video controls in educational videos provides implicit signals of cognitive processing and instructional design quality, yet the lack of scalable and explainable predictive models limits instructors' ability to anticipate such behavior before deployment. We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior as proxies for cognitive load from video content alone. Our approach leverages multimodal large language models (MLLMs) to compute embeddings of short video segments and trains a neural classifier to identify temporally fine-grained interaction peaks. Drawing from multimedia learning theory on instructional design for optimal cognitive load, we code features of the video segments using GPT-5 and employ them as a basis for interpreting model predictions via concept activation vectors. We evaluate our pipeline on 77 million video control events from 66 online courses. Our findings demonstrate that classifiers based on MLLM embeddings reliably predict interaction peaks, generalize to unseen academic fields, and encode interpretable, theory-relevant instructional concepts. Overall, our results show the feasibility of cost-efficient, interpretable pre-screening of educational video design and open new opportunities to empirically examine multimedia learning theory at scale.

Figures & tables

Figure 1: Pipeline for predicting learners’ interactions with online learning videos and explaining predictions from multimedia learning theory.

Field	#Courses	#Course runs	#Learners	Avg. #Videos	Avg. Length [min]
Mathematics	18	42	49,311	62.7	12.1
Neuroscience	15	28	78,067	43.7	13.5
Electrical Engineering	10	27	43,841	41.9	17.9
Physics	10	18	60,924	37.8	17.0
Management	9	15	19,451	44.3	10.9
Others	23	44	105,671	41.6	12.0

Table 1: Statistics of online learning data in our study. "Others" include Architecture, Biology, Computer Science, Education, Environmental Science, and Microengineering.

Figure 2: Three modalities of video segments around $t$ are encoded by pre-trained transformers. A neural classifier predicts if ${\text{Signal}}_{v}(t)$ is among the top K% at timepoint $t$ .

Modality	Feature	Description	Scale	Lit.
$F_{t}$	Formula	Math notation beyond single symbols is visible.	{0,1}	[ 11 ]
$F_{t}$	Instructor	Instructor’s head is visible in the video.	{0,1}	[ 9 ]
$F_{t}$	Screen	The user interface of a computer screen is embedded inside the image (e.g., a code editor).	{0,1}	[ 11 ]
$F_{t}$	Structured Information Visualization	A diagram, graph, schematic drawing, or table is on the slide (no formulas, no GUI).	{0,1}	[ 11 ]
$F_{t}$	Text Object	A printed or handwritten sentence, derivation, or bullet point is visible (no labels, titles, footers, text in screenshots or code editors, references).	{0,1}	[ 11 ]
$F_{t}$	Visual Complexity	The amount, complexity, and diversity of textual and graphical content objects visible. Ignore instructors and typical slide elements like logos or titles as elements.	1-5	[ 11 ]

Table 2: Rubric for coding CTML features of the video lecture content.

Signal $\rightarrow$	${\text{Watched}}_{v}(t)$		${\text{PausedAt}}_{v}(t)$		${\text{RewoundTo}}_{v}(t)$		${\text{SkippedFrom}}_{v}(t)$
Embeddings $\downarrow$	Top 10%	Top 5%	Top 10%	Top 5%	Top 10%	Top 5%	Top 10%	Top 5%
$e_{T}$	0.58±0.02	0.58±0.03	0.62±0.02	0.62±0.02	0.59±0.02	0.61±0.02	0.58±0.02	0.57±0.02
$e_{S}$	0.50±0.01	0.52±0.02	0.53±0.01	0.54±0.01	0.56±0.03	0.56±0.02	0.57±0.03	0.57±0.03
$e_{F,t}$	0.59±0.02	0.59±0.03	0.64±0.03	0.68±0.03	0.60±0.03	0.62±0.04	0.57±0.03	0.57±0.02
$e_{F,(t-10,t,t+10)}$	0.63±0.02	0.63±0.04	0.67±0.04	0.70±0.04	0.67±0.03	0.69±0.06	0.62±0.03	0.62±0.05
$e_{V,\{1,32,64\}}$	0.65±0.03	0.67±0.04	0.71±0.03	0.71±0.03	0.71±0.02	0.76±0.03	0.62±0.04	0.62±0.03

Table 3: Predictive performance in terms of AUC (

\pm

std across 10 seeds) of moment-to-moment video interaction prediction from multimodal video content embeddings; Lift@K% (

\pm

std) is additionally reported for the final multimodal model

e_{x}

Figure 3: AUC (

\pm

std, 5 seeds) for prediction of top 5% learner-video interaction moments on fields unseen during training as a measure of generalization of our classifier.

Signal $\rightarrow$	${\text{Watched}}_{v}(t)$		${\text{PausedAt}}_{v}(t)$		${\text{RewoundTo}}_{v}(t)$		${\text{SkippedFrom}}_{v}(t)$
Setting	Top 10%	Top 5%	Top 10%	Top 5%	Top 10%	Top 5%	Top 10%	Top 5%
Reference $\bar{e}_{x}$ + Shared Weights in $h_{1}$	0.67±0.02	0.69±0.03	0.74±0.02	0.74±0.02	0.72±0.02	0.76±0.03	0.64±0.02	0.66±0.03
Reference $\bar{e}_{x}$ + No sharing in $h_{1}$	0.66±0.02	0.68±0.03	0.72±0.02	0.74±0.02	0.71±0.02	0.77±0.03	0.61±0.03	0.65±0.03
No Reference $\bar{e}_{x}$	0.66±0.02	0.68±0.03	0.70±0.03	0.71±0.03	0.71±0.02	0.73±0.03	0.64±0.03	0.63±0.04
1. Qwen-VL Layer $e_{TV,1}$	0.63±0.04	0.61±0.03	0.64±0.02	0.65±0.02	0.65±0.03	0.65±0.03	0.58±0.02	0.60±0.03
32. Qwen-VL Layer $e_{TV,32}$	0.67±0.02	0.69±0.02	0.72±0.02	0.73±0.02	0.72±0.03	0.76±0.03	0.63±0.03	0.64±0.03

Table 4: AUC (

\pm

std, 10 seeds) for different neural architectures (top), three different layers of Qwen-VL (middle), and a smaller vision language model (bottom).

Figure 4: Learner-video interactions at 6,000 video moments (50% from top 5% ranks) by CTML features coded by GPT-5 (inter-rater agreement in the right panel). For example, moments without a formula have an average

{\text{PausedAt}}_{v}(t)

rank of 67% within their video, whereas moments with a formula have a higher average rank of 71%.

	CTML Features				Multimodal Embeddings $e_{x}$
Sample Size $\rightarrow$	500	1,000	2,000	5,000	500	1,000	2,000	5,000	19,000
${\text{PausedAt}}_{v}(t)$	0.66 ± 0.02	0.69 ± 0.01	0.68 ± 0.01	0.68 ± 0.0	0.62 ± 0.02	0.67 ± 0.01	0.70 ± 0.0	0.72 ± 0.0	0.77 ± 0.01
${\text{RewoundTo}}_{v}(t)$	0.73 ± 0.01	0.72 ± 0.0	0.73 ± 0.0	0.73 ± 0.0	0.68 ± 0.02	0.71 ± 0.01	0.73 ± 0.0	0.74 ± 0.0	0.78 ± 0.01

Table 5: Classifier performance (AUC

\pm

5-fold CV std) when using CTML features versus embeddings to predict learners’ interactions across training dataset sizes.

Figure 5: TCAV values of CTML concepts and activations in our classifier $H$ . Significant (*) values above 0.5 mean that the classifier was positively sensitive to the concept present in the activation space.

Explore similar work

VideoLatent: Video-Language Learning via Latent Self-Forcing
Jun 22, 2026Zi-Yuan Hu, Zicong Tang, Shijia Huang +3Multimodal Large Language Models Latent Visual Reasoning
The Chinese University of Hong Kong · Weitu AI
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
Jun 5, 2026Jiahao Meng, Yue Tan, Qi Xu +12Streaming Video Understanding Video-Language Models
School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University +7
Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
Jun 4, 2026Tianxiang Jiang, Linquan Wu, Sheng Xia +5Large Vision-Language Models Latent Visual Reasoning
University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4

Jun 22, 2026J/K move · Enter open · S save

VideoLatent: Video-Language Learning via Latent Self-Forcing

Zi-Yuan Hu, Zicong Tang, Shijia Huang, Yanyang Li+2

The Chinese University of Hong Kong · Weitu AI

Multimodal Large Language Models Latent Visual Reasoning Video-Language Models Video Reasoning VLM Reasoning

Open paper AskPDF arXiv

Recent advancements in chain-of-thought (CoT) reasoning have shown promise in enhancing video understanding and reasoning capabilities of multimodal large language models (MLLMs). However, existing CoT-based MLLMs require labor-intensive CoT annotations and incur substantial training and inference overhead. While visual latent reasoning has emerged as a more efficient alternative, existing methods primarily focus on image tasks and heavily rely on additional supervision signals for visual latent generation (e.g., CoT traces, auxiliary images, or fine-grained annotations), limiting their scalability and transferability to video tasks. To bridge this gap, we introduce VideoLatent, a novel MLLM equipped with a latent injection module tailored for video understanding and reasoning. Specifically, VideoLatent learns to perform visual latent reasoning using a new latent self-forcing training paradigm, which comprises latent alignment and latent diversity objectives, and relies solely on standard video-question-answer triplets. Extensive experiments across 14 benchmarks demonstrate that our model consistently outperforms existing standard and latent MLLMs on general video understanding and complex video reasoning. Compared with Video-R1, our VideoLatent achieves superior computational efficiency, reducing training/inference overhead by

\sim

\times

\sim

\times

. Moreover, experiments demonstrate that our method has strong generalizability to different MLLM backbones and different model scales.

Abstract

Explore similar work

VideoLatent: Video-Language Learning via Latent Self-Forcing

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction