Can Large Language Models Capture Video Game Engagement?
Organizations: Institute of Digital Games, University of Malta Msida, Malta
Abstract
Can out-of-the-box pretrained Large Language Models (LLMs) detect human affect successfully when observing a video? To address this question, for the first time, we evaluate comprehensively the capacity of popular LLMs for successfully predicting continuous affect annotations of videos when prompted by a sequence of text and video frames in a multimodal fashion. In this paper, we test LLMs' ability to correctly label changes of in-game engagement in 80 minutes of annotated videogame footage from 20 first-person shooter games of the GameVibe corpus. We run over 4,800 experiments to investigate the impact of LLM architecture, model size, input modality, prompting strategy, and ground truth processing method on engagement prediction. Our findings suggest that while LLMs rightfully claim human-like performance across multiple domains and able to outperform traditional machine learning baselines, they generally fall behind continuous experience annotations provided by humans. We examine some of the underlying causes for a fluctuating performance across games, highlight the cases where LLMs exceed expectations, and draw a roadmap for the further exploration of automated emotion labelling via LLMs.
Figures & tables
| Properties | GameVibe | GameVibe-LLM |
|---|---|---|
| Annotators | 20 | 20 |
| Number of videos | 120 videos | 80 videos |
| Video database size | 120 minutes | 80 minutes |
| Number of games | 30 games | 20 games |
| Gameplay video duration | 1 minute each | 1 minute each |
| Annotation type | Interval signal | Discrete ordinal |
| Lower F1 | Higher F1 | t-stat |
|---|---|---|
| InternVL2.5 (0.44) | GPT-4o (0.57) | 4.13 |
| Naive Baseline (0.37) | Phi 3.5 (0.44) | 3.53 |
| Gemma 3 (0.50) | 6.13 | |
| LLaVA-OV (0.50) | 7.36 | |
| Qwen2.5-VL (0.52) | 5.9 | |
| GPT-4o (0.57) | 6.23 |
| Type | Lower F1 | Higher F1 | t-stat |
|---|---|---|---|
| Advanced | Naive Baseline (0.37) | Phi 3.5 (0.48) | 2.46 |
| Desc. | Gemma 3 (0.5) | 5.57 | |
| InternVL2.5 (0.5) | 6.18 | ||
| LLaVA-OV (0.49) | 5.57 | ||
| Qwen2.5-VL (0.5) | 6.12 | ||
| GPT-4o (0.53) | 6.72 |
| Advanced Desc. | Basic Desc. | |
|---|---|---|
| Easiest | Doom (0.58) | Wolfram (0.68) |
| Wolfram (0.57) | Fortnite (0.57) | |
| Blitz Brigade (0.54) | Corridor 7 (0.54) | |
| Hardest | CS:GO - Office (0.43) | PUBG (0.40) |
| Borderlands 3 (0.45) | CS:GO - Dust2 (0.41) | |
| CS:GO - Dust2 (0.45) | Insurgency (0.45) |
| Lower F1 | Higher F1 | t-stat |
|---|---|---|
| InternVL2.5 (0.43) | Gemma 3 (0.55) | 4.72 |
| Qwen2.5-VL (0.56) | 4.79 | |
| GPT-4o (0.60) | 7.84 | |
| LLaVA-OV (0.46) | Gemma 3 (0.55) | 3.31 |
| GPT-4o (0.60) | 5.18 | |
| Phi 3.5 (0.48) | GPT-4o (0.60) | 4.74 |
Affective Computing
Affective computing develops systems that recognize, model, and respond to human emotions and affective states from behavior and physiological signals.
See all 149 papers