cs.CVMar 19, 2026
SaveDo Vision Language Models Understand Human Engagement in Games?
Organizations: Arizona State University
Abstract
Inferring human engagement from gameplay video is important for game design and player-experience research, yet it remains unclear whether vision--language models (VLMs) can infer such latent psychological states from visual cues alone. Using the GameVibe Few-Shot dataset across nine first-person shooter games, we evaluate three VLMs under six prompting strategies, including zero-shot prediction, theory-guided prompts grounded in Flow, GameFlow, Self-Determination Theory, and MDA, and retrieval-augmented prompting. We consider both pointwise engagement prediction and pairwise prediction of engagement change between consecutive windows. Results show that zero-shot VLM predictions are generally weak and often fail to outperform simple per-game majority-class baselines. Memory- or retrieval-augmented prompting improves pointwise prediction in some settings, whereas pairwise prediction remains consistently difficult across strategies. Theory-guided prompting alone does not reliably help and can instead reinforce surface-level shortcuts. These findings suggest a perception--understanding gap in current VLMs: although they can recognize visible gameplay cues, they still struggle to robustly infer human engagement across games.
Figures & tables
| Study | Visual Input | VLM Evaluation | Game Domain | Engagement Task | Theory Probes | Cross-Game Transfer | Failure Analysis |
| Melhart et al. (2025) | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
| Bhattacharyya and Wang (2025) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Lu et al. (2024) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Pinitas et al. (2025) | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
| Zhang et al. (2025) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Paglieri et al. (2025) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
Table 1: Comparison of our work with the most related prior studies across key evaluation dimensions.
Figure 1: Overview of our experimental pipeline. The GameVibe Few-Shot (GVFS) dataset provides 9 target games, each containing 61 seconds of gameplay sampled at 16 frames per second (59 evaluation windows after reaction-lag shifting). We apply six prompting strategies of increasing complexity (S1–S6). Three VLMs—InternVL3.5-8B-Instruct, Qwen3-VL-8B-Instruct, and GPT-4o—predict binary engagement labels (High/Low). Predictions are evaluated against human annotations using accuracy, (Eq. 1 ), temporal coherence, and failure-mode analysis.
| Game | Windows | High (%) | Low (%) |
| Borderlands 3 | 59 | 81.4 | 18.6 |
| CS:GO Office | 59 | 33.9 | 66.1 |
| Blitz Brigade | 59 | 64.4 | 35.6 |
| Corridor 7 | 59 | 39.0 | 61.0 |
| Battlefield 42 | 59 | 39.0 | 61.0 |
| Apex Legends | 59 | 55.9 | 44.1 |
Table 2: Label distribution across the 9 evaluation games and the source domain. Each game contributes 61 seconds of gameplay (59 evaluation windows after reaction-lag shifting, 944 frames at 16 fps). All games exhibit varying degrees of class imbalance.
| ID | Strategy | Theory | Retrieval |
| S1 | Zero-shot | ||
| S2 | TG zero-shot | ✓ | |
| S3 | Few-shot (VLM) | VLM embedding | |
| S4 | Few-shot (CLIP) | CLIP embedding | |
| S5 | TG few-shot (VLM) | ✓ | VLM embedding |
| S6 | TG few-shot (CLIP) | ✓ | CLIP embedding |
Table 3: Summary of the six prompting strategies.
| InternVL | Qwen | GPT-4o | |||||||||||||
| Game | M | S1 | S2 | S3 | S4 | S5 | S6 | S1 | S2 | S3 | S4 | S5 | S6 | S1 | S2 |
| Borderlands 3 | 81.4 | 79.7 | 81.4 | 83.4 | 79.7 | 53.5 | 81.4 | 83.1 | 81.4 | 75.4 | 49.2 | 84.2 | 54.2 | 83.1 | 81.4 |
| CS:GO Office | 66.1 | 61.0 | 42.4 | 58.5 | 55.9 | 71.8 | 33.9 | 57.6 | 44.1 | 71.1 | 86.4 | 70.8 | 78.0 | 52.5 | 50.8 |
| Blitz Brigade | 64.4 | 66.1 | 64.4 | 68.7 | 69.5 | 62.7 | 64.4 | 62.7 | 64.4 | 78.9 | 66.1 | 78.2 | 66.1 | 62.7 | 64.4 |
| Corridor 7 | 61.0 | 49.2 | 37.3 | 83.5 | 64.4 | 72.9 | 39.0 | 66.1 | 39.0 | 72.5 | 55.9 | 73.9 | 52.5 | 57.6 | 57.6 |
| Battlefield 42 | 61.0 | 64.4 | 52.5 | 46.8 | 39.0 | 64.8 | 39.0 | 62.7 | 57.6 | 80.3 | 61.0 | 68.3 | 72.9 | 64.4 | 45.8 |
Table 4: Per-game pointwise engagement prediction accuracy (%). M = per-game majority-class baseline. Boldface indicates the best result per game within each model. S3/S5 are evaluated on the 18-window held-out split, all other strategies on all 59 windows; 95% CIs in Appendix E .
| InternVL | Qwen | GPT-4o | |||||||||||||
| Game | M | S1 | S2 | S3 | S4 | S5 | S6 | S1 | S2 | S3 | S4 | S5 | S6 | S1 | S2 |
| Borderlands 3 | 50.1 | 54.6 | 51.5 | 53.0 | 50.0 | 50.0 | 25.0 | 57.6 | 65.2 | 53.0 | 62.1 | 87.5 | 87.5 | 54.6 | 56.1 |
| CS:GO Office | 58.7 | 42.7 | 49.3 | 42.7 | 52.0 | 23.1 | 30.8 | 37.3 | 45.3 | 46.7 | 42.7 | 69.2 | 76.9 | 41.3 | 45.3 |
| Blitz Brigade | 57.6 | 56.1 | 47.0 | 65.1 | 57.6 | 71.4 | 64.3 | 68.2 | 62.1 | 51.5 | 51.5 | 57.1 | 42.9 | 65.2 | 60.6 |
| Corridor 7 | 64.9 | 51.4 | 40.5 | 51.4 | 67.6 | 70.0 | 70.0 | 73.0 | 51.4 | 35.1 | 64.9 | 50.0 | 50.0 | 59.5 | 56.8 |
| Battlefield 42 | 60.7 | 57.4 | 55.7 | 55.7 | 49.2 | 47.4 | 47.4 | 62.3 | 64.0 | 54.1 | 67.2 | 52.6 | 52.6 | 54.1 | 57.4 |
Table 5: Per-game pairwise engagement prediction accuracy (%). M = per-game majority-class baseline. Boldface indicates the best result per game within each model. Per-game sample sizes range from 37 to 78 pairs; 95% CIs in Appendix E .
| Game | Model | FN | FP | Total Err. |
| CSGO18 | InternVL | 1 | 32 | 33 |
| Qwen3-VL | 4 | 37 | 41 | |
| CS:GO Office | InternVL | 6 | 17 | 23 |
| Qwen3-VL | 10 | 15 | 25 |
Table 6: Zero-shot (S1) error distribution on the two CS:GO variants.
| Metric | Dataset | VLM Pred. | Human Labels |
| Flip Rate | CSGO18 | 0.310 | 0.017 |
| CS:GO Office | 0.345 | 0.190 | |
| Autocorr. (lag-1) | CSGO18 | 0.275 | 0.950 |
| CS:GO Office | 0.168 | 0.576 |
Table 7: Temporal coherence metrics, averaged over strategies S1–S6 and the two open-source VLMs. VLM predictions are dramatically less coherent than human labels across consecutive 1-second windows.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Failure Type | CSGO18 | CS:GO Office | Total | % |
| A: Static Scene Misclass. | 3 | 4 | 7 | 9.0 |
| B: Relatedness Shortcut | 19 | 5 | 24 | 30.8 |
| C: Sensation Shortcut | 0 | 10 | 10 | 12.8 |
| D: Challenge Misassess. | 16 | 14 | 30 | 38.5 |
| E: Other/Ambiguous | 6 | 1 | 7 | 9.0 |
| Total Errors | 44 | 34 | 78 |
Table 8: Distribution of failure types. Challenge Misassessment and Relatedness Shortcuts account for over two-thirds of all errors.
| Target Game | Avg Sim. | SD | Acc. (%) | |
| CS:GO Office | 0.657 | 0.030 | 63.9 | 0.14 |
| CSGO18 | 0.628 | 0.033 | 37.7 | 0.01 |
| Similarity: | , Cohen’s | |||
| Accuracy: | ||||
Table 9: CLIP retrieval similarity and accuracy for two CS:GO variants.
Figure 2: CLIP retrieval similarity analysis for two CS:GO variants. Top : Similarity distributions and box plots. Bottom : Per-frame similarity vs. prediction correctness.
Figure 3: Effect of demonstration label distribution on few-shot prediction accuracy. When the original memory contains no positive examples, the model over-predicts the negative class. Introducing positive examples into the demonstration memory consistently improves accuracy across multiple games.
| Bord. 3 | Office | Blitz | Corr. 7 | Battl. 42 | Apex | CSGO19 | CSGO18 | CS 1.6 | Average | ||
| InternVL | S1 | [69.5, 89.8] | [49.2, 72.9] | [54.2, 78.0] | [37.3, 62.7] | [52.5, 76.3] | [32.2, 59.3] | [59.3, 83.1] | [32.2, 57.6] | [18.6, 42.4] | 56.9 [47.3, 66.3] |
| S2 | [71.2, 91.5] | [30.5, 55.9] | [52.5, 76.3] | [25.4, 49.2] | [40.7, 66.1] | [40.7, 66.1] | [47.5, 71.2] | [15.3, 37.3] | [45.8, 71.2] | 52.9 [42.9, 62.9] | |
| S3 | – | – | – | – | – | – | – | – | – | 71.3 [63.0, 78.6] | |
| S4 | [69.5, 89.8] | [42.4, 67.8] | [57.6, 81.4] | [52.5, 76.3] | [27.1, 50.8] | [37.3, 62.7] | [39.0, 62.7] | [37.3, 62.7] | [40.7, 66.1] | 56.9 [49.7, 64.8] | |
| S5 | – | – | – | – | – | – | – | – | – | 68.3 [62.0, 74.3] | |
| S6 | [71.2, 91.5] | [22.0, 45.8] | [52.5, 76.3] | [27.1, 52.5] | [27.1, 52.5] | [33.9, 61.0] | [52.5, 76.3] | [13.6, 35.6] | [57.6, 81.4] | 51.4 [39.9, 63.3] |
Table 10: Bootstrap 95% confidence intervals for the pointwise accuracies of Table 4 . Per-game entries resample the 59 per-window correctness outcomes; the Average column reports the cross-game mean with a game-level bootstrap interval (resampling the nine per-game accuracies). For S3/S5, which are evaluated on smaller held-out subsets ( 30% of windows), only the game-level interval is reported (see text).
| Bord. 3 | Office | Blitz | Corr. 7 | Battl. 42 | Apex | CSGO19 | CSGO18 | CS 1.6 | Average | ||
| InternVL | S1 | [42.4, 66.7] | [32.0, 54.7] | [43.9, 68.2] | [35.1, 67.6] | [44.3, 68.9] | [55.9, 79.4] | [30.4, 58.7] | [40.6, 63.8] | [41.0, 64.1] | 53.1 [48.6, 57.8] |
| S2 | [39.4, 63.6] | [37.3, 60.0] | [34.8, 59.1] | [24.3, 56.8] | [42.6, 68.9] | [45.6, 69.1] | [37.0, 67.4] | [39.1, 62.3] | [32.1, 55.1] | 49.8 [46.4, 53.0] | |
| S3 | [40.9, 65.2] | [32.0, 54.7] | [53.0, 75.8] | [35.1, 67.6] | [42.6, 67.2] | [44.1, 67.6] | [34.8, 65.2] | [34.8, 58.0] | [37.2, 60.3] | 51.8 [48.3, 56.3] | |
| S4 | [37.9, 62.1] | [40.0, 62.7] | [45.5, 69.7] | [51.4, 81.1] | [37.7, 62.3] | [50.0, 73.5] | [28.3, 56.5] | [42.0, 65.2] | [34.6, 56.4] | 53.1 [48.1, 58.3] | |
| S5 | – | – | – | – | – | – | – | – | – | 55.9 [44.1, 66.2] | |
| S6 | – | – | – | – | – | – | – | – | – | 49.8 [36.6, 61.8] |
Table 11: Bootstrap 95% confidence intervals for the pairwise accuracies of Table 5 . Per-game entries resample the per-pair correctness outcomes with the per-game pair counts given in the text; the Average column reports the cross-game mean with a game-level bootstrap interval. For S5/S6, which are evaluated on smaller strategy-specific subsets of pairs, only the game-level interval is reported.
| InternVL | Qwen | |||||||
| Game | S1 (18-win. subsets) | S2 (18-win. subsets) | S3 | S5 | S1 (18-win. subsets) | S2 (18-win. subsets) | S3 | S5 |
| Borderlands 3 | 79.6 [61.1, 94.4] | 81.3 [66.7, 94.4] | 83.4 | 53.5 | 83.0 [66.7, 94.4] | 81.4 [66.7, 94.4] | 75.4 | 84.2 |
| CS:GO Office | 61.2 [44.4, 77.8] | 42.3 [22.2, 61.1] | 58.5 | 71.8 | 57.7 [38.9, 77.8] | 44.0 [27.8, 61.1] | 71.1 | 70.8 |
| Blitz Brigade | 66.1 [50.0, 83.3] | 64.5 [44.4, 83.3] | 68.7 | 62.7 | 62.7 [44.4, 83.3] | 64.5 [44.4, 83.3] | 78.9 | 78.2 |
| Corridor 7 | 49.4 [27.8, 66.7] | 37.3 [16.7, 55.6] | 83.5 | 72.9 | 66.1 [50.0, 83.3] | 38.8 [22.2, 55.6] | 72.5 | 73.9 |
| Battlefield 42 | 64.4 [44.4, 83.3] | 52.6 [33.3, 72.2] | 46.8 | 64.8 | 62.7 [44.4, 83.3] | 57.7 [38.9, 77.8] | 80.3 | 68.3 |
Table 12: Evaluation-subset effect for the zero-shot strategies. For each game we draw 10,000 random 18-window subsets of the 59 evaluation windows and report the mean and central 95% range of S1/S2 accuracy on subsets of the size used by the S3/S5 held-out evaluation; the S3/S5 columns repeat the accuracies from Table 4 for reference. Subset means coincide with the full-set accuracies, and the wide ranges indicate that point comparisons across evaluation subsets of this size carry substantial sampling variability.
| Ordering | Pred. High | Pred. Low |
| Original [L, H, H, L] | 12 | 8 |
| Last-high [L, L, H, H] | 12 | 8 |
| Alternating H-L | 10 | 10 |
| Alternating L-H | 12 | 8 |
Table 13: Prediction distribution under different 4-shot example orderings (CS:GO Office subset). Distributions are stable across orderings.
Figure 4: Prediction accuracy by VLM confidence level, showing severe miscalibration. High-confidence predictions are the least accurate (38.5%), while low-confidence predictions are the most accurate (64.3%). The dashed line shows random baseline (52.4%).
Figure 5: CS:GO Office, frame 24–25. The VLM predicts Low due to the static vote screen, but the ground truth is High.
Figure 6: CS:GO Office, frame 31–32. Blood splatters and weapon animations trigger a High prediction, but the ground truth is Low.
Figure 7: CSGO18, frame 15–16. Visible teammates and HUD indicators lead to a High prediction, but the ground truth is Low.
Figure 8: CSGO18, frame 12–13. The buy menu is dismissed as “no active gameplay,” but the ground truth is High.