cs.CLMar 25, 2026
SaveGameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents
Organizations: University of Southern California
Abstract
Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective, capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World, a natural decomposition for multi-agent environments. From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.
Figures & tables
Figure 1: Question taxonomy of GameplayQA . Questions are organized along two axes: Entity (Self, Other, World) and Temporal Nature (Action/State for agents, Object/Event for world), yielding six primitive label types. These primitives compose into 15 task categories across three cognitive levels: single-reference perception (L1), temporal reasoning (L2), and cross-video understanding (L3). See Sec. 3.1 and Table 2 for details.
Figure 2: Overview of GameplayQA . Gameplay videos undergo (1) dense multi-track temporal captioning on 6 types of target entities (Sec. 3.2 ), (2) captioning includes negative labels for hallucination-inducing distractors, and (3) QA pairs are generated through a combinatorial template-based algorithm (Sec. 3.3 ). After (4) quality assurance (Sec. 3.4 ), the benchmark enables (5) model evaluation with (6) fine-grained hallucination analysis (Sec. 4.2 ).
Figure 3: Example questions from GameplayQA across different question codes and cognitive levels. Each example shows video frames paired with the corresponding QA pair, illustrating the progression from basic perception (L1) to temporal reasoning (L2) to cross-video understanding (L3). Additional cross-domain examples from car collision and egocentric human activity videos demonstrate the generalizability of the framework.
| Benchmark | Domain | Agent-Centric | Multi-POV | Diagnostic | Annotation | #Q | #Vid |
|---|---|---|---|---|---|---|---|
| MVBench Li et al. (2024a) | General | ✗ | ✗ | ✓ | Auto | 4,000 | 4,000 |
| LongVideoBench Wu et al. (2024) | General | ✗ | ✗ | ✓ | Human | 6,678 | 3,763 |
| MVU-Eval Peng et al. (2025) | General | ✗ | ✓ | ✗ | Human, Auto | 1,824 | 4,959 |
| MovieQA Tapaswi et al. (2016) | Movie | ✗ | ✗ | ✗ | Human | 15k | 408 |
| TVQA Lei et al. (2018) | TV Shows | ✗ | ✗ | ✗ | Human | 152k | 21.8k |
| MarioQA Mun et al. (2017) | Gameplay | ✓ | ✗ | ✗ | Auto | 188k | 13 hrs |
Table 1: Comparison of relevant video understanding benchmarks.
| Context Scope | Task | Description | Example Question Codes | #Q | Dur. |
|---|---|---|---|---|---|
| Single Reference (L1, 469) | Action Recognition | Identify or verify existence of self & others’ actions | SA-IDENT , OA-EXIST , … | 162 | 10.0s |
| State Recognition | Identify or verify existence of self & others’ states | SS-IDENT , OS-EXIST , … | 147 | 10.1s | |
| Object Recognition | Identify or verify existence of world objects in scene | WO-IDENT , WO-EXIST | 70 | 9.3s | |
| Event Recognition | Identify world events occurring in the environment | WE-IDENT | 61 | 8.4s | |
| Static Object Count | Count static objects present in the scene | WO-COUNT | 29 | 21.3s | |
| Temporal (L2, 1383) | Cross-Entity Referring | Link one entity to another (X2Y reasoning) | SA2SS-IDENT , WO2SS-EXIST , … | 423 | 23.0s |
Table 2: Definition of 15 categories of cross-entity referring reasoning questions in GameplayQA , including the number of questions and average video duration.
| Model | All | L1 (Single Ref.) | L2 (Temporal) | L3 (Cross-Video) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ActRec | StaRec | ObjRec | EvtRec | SOC | X-Ent | TsRef | TimLoc | AbsRec | OccCnt | Order | Intent | SyncRef | X-VOrd | POV-ID | ||
| Human | 80.5 | 80.0 | 80.0 | 100.0 | 75.0 | 100.0 | 84.2 | 100.0 | 76.9 | 83.3 | 62.5 | 77.8 | 57.1 | 88.9 | 77.8 | 100.0 |
| Proprietary Models | ||||||||||||||||
| GPT-5 | 67.0 | 79.0 | 70.7 | 70.0 | 68.9 | 48.3 | 71.6 | 70.4 | 45.9 | 86.2 | 62.7 | 78.3 | 54.1 | 72.0 | 60.7 | 54.0 |
| GPT-5 Mini | 62.7 | 70.4 | 67.3 | 68.6 | 72.1 | 34.5 | 67.6 | 66.7 | 47.0 | 79.0 | 33.3 | 72.8 | 50.0 | 72.0 | 43.6 | 58.7 |
| GPT-5 Nano | 49.3 | 61.7 | 60.5 | 70.0 | 72.1 | 37.9 | 57.7 | 65.4 | 33.5 | 64.6 | 17.3 | 35.0 | 41.9 | 49.8 | 29.1 | 42.9 |
Table 3: Model performance across task categories. Gold = best, Silver = second best. L1: ActRec (Action Recognition), StaRec (State Recognition), ObjRec (Object Recognition), EvtRec (Event Recognition), SOC (Static Object Count). L2: X-Ent (Cross-Entity Referring), TsRef (Timestamp Referring), TimLoc (Time Localization), AbsRec (Absence Recognition), OccCnt (Occurrence Count), Order (Ordering), Intent (Intent Identification). L3: SyncRef (Sync-Referring), X-VOrd (Cross-Video Ordering), POV-ID (POV Identification).
| Model | All | SA | SS | OA | OS | WO | WE |
|---|---|---|---|---|---|---|---|
| Proprietary Models | |||||||
| Gemini 2.5 Pro | 71.3 | 75.2 | 73.3 | 65.6 | 72.0 | 71.7 | 74.0 |
| Gemini 3 Flash | 68.2 | 70.5 | 73.0 | 65.6 | 64.8 | 70.5 | 68.4 |
| GPT-5 | 67.0 | 66.0 | 73.5 | 60.8 | 67.0 | 71.6 | 69.1 |
| Gemini 2.5 Flash | 63.7 | 65.1 | 66.6 | 60.8 | 60.1 | 67.6 | 64.9 |
| GPT-5 Mini | 62.7 | 64.4 | 68.6 | 57.6 | 60.4 | 66.8 | 63.8 |
Table 4: Model performance by entity category. Gold = best, Silver = second best. SA: Self-Action, SS: Self-State, OA: Other-Action, OS: Other-State, WO: World-Object, WE: World-Event.
Figure 4: Error rate analysis across four dimensions. Top-left : Cross-video and temporal distractors cause the most errors. Top-right : Fast-paced shooters (CS2, Battlefield) are hardest. Bottom-left : Error increases with video length. Bottom-right : Error scales with number of synchronized videos.
| Condition | All | L1 | L2 | L3 |
|---|---|---|---|---|
| Baseline (Full) | 62.7 | 67.2 | 61.9 | 60.6 |
| No Video | 29.4 | 36.0 | 29.1 | 24.2 |
| Random Frame | 41.7 | 52.9 | 40.9 | 33.7 |
| Shuffled Frames | 54.8 | 63.1 | 52.6 | 53.4 |
Table 5: Ablation study on GPT- 5 -mini with degraded visual inputs. Performance drops indicate the contribution of video content and temporal ordering.
| Model | All | L1 (Single Ref.) | L2 (Temporal) | L3 (Cross-Video) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ActRec | StaRec | ObjRec | EvtRec | X-Ent | TsRef | TimLoc | AbsRec | OccCnt | Order | Intent | SyncRef | X-VOrd | POV-ID | ||
| Gemini 2.5 Pro | 66.2 | 61.1 | 86.7 | 83.3 | 100.0 | 62.0 | 80.0 | 52.6 | 72.2 | 40.0 | 50.0 | 66.7 | 76.7 | 25.0 | 55.6 |
| Gemini 2.5 Flash | 62.0 | 72.2 | 73.3 | 66.7 | 80.0 | 64.0 | 80.0 | 36.8 | 61.1 | 40.0 | 83.3 | 55.6 | 70.0 | 50.0 | 38.9 |
| GPT-5 Mini | 61.0 | 55.6 | 66.7 | 100.0 | 80.0 | 66.0 | 70.0 | 63.2 | 77.8 | 20.0 | 66.7 | 66.7 | 53.3 | 50.0 | 27.8 |
| Qwen3 VL 235B | 59.2 | 50.0 | 73.3 | 100.0 | 100.0 | 66.0 | 80.0 | 36.8 | 72.2 | 20.0 | 50.0 | 66.7 | 50.0 | 25.0 | 44.4 |
Table 6: Cross-domain generalization results on real-world ego-centric videos (autonomous driving + multi-human collaboration, 213 questions). Gold = best, Silver = second best. Column headers follow the same conventions as Table 3 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Illustration of the Self–Other–World framework. A first-person player (Self) perceives a teammate (Other) issuing a warning, set against the surrounding game environment (World). These three perspectives define the entity types used in our question taxonomy.
| Dimension | Category | Description | Example Question |
| Number of Videos | Single Video | Inputs a single video | What action did the player perform? |
| Multi-Video | Inputs multiple videos | When POV1 player was reloading, which action did POV2 player perform? | |
| Context Target | Summative | (Single-Video Only) Requires aggregation over a temporal segment. | Which of the following best summarizes the player’s actions during the video? |
| Timestamp Referring | Refers to a specific moment or interval | At [02:45 - 02:52], what action did the player perform? | |
| Target Entity Referring | Refers to the temporal moment of the 6 entity types. | When the player was reloading, what action did the teammate perform? | |
| Cross-Video Referring | (Multi-Video Only) Explicitly references a video index. | Which POV player reload their weapon? |
Table 7: GameplayQA question taxonomy. Each question is defined by a combination of dimensions, enabling systematic coverage of perception, temporal reasoning, and cross-video reasoning.
| Model Name | Version | Inference Provider | Platform |
|---|---|---|---|
| GPT- 5 | gpt-5 | OpenAI | OpenAI |
| GPT- 5 -mini | gpt-5-mini | OpenAI | OpenAI |
| GPT- 5 -nano | gpt-5-nano | OpenAI | OpenAI |
| Gemini- 2.5 -Pro | gemini-2.5-pro | Google AI Studio | |
| Gemini- 3 -Flash | gemini-3-flash | Google AI Studio | |
| Gemini- 2.5 -Flash | gemini-2.5-flash | Google AI Studio |
Table 8: Inference configurations for tested models. All models are evaluated using the listed inference providers.
| Label Type | Count | Share |
|---|---|---|
| Self-Action (SA) | 658 | 24.3% |
| Self-State (SS) | 729 | 26.9% |
| Other-Action (OA) | 160 | 5.9% |
| Other-State (OS) | 190 | 7.0% |
| World-Event (WE) | 417 | 15.4% |
| World-Object (WO) | 555 | 20.5% |
Table 10: Per-type label distribution ( labels/s over 2,219.41 s of annotated footage).
Figure 6: Top left: Distribution of the first four words of the questions. The arc length indicates the frequency. Top right: Distribution of question codes, categorized by count and task type (using color for cognitive level). Bottom: Word cloud visualizing question terms by entity type.
| Game | Expert | Regular | Casual | Low | None |
|---|---|---|---|---|---|
| (>300h) | (>30h) | (>5h) | (>0h) | ||
| Counter-Strike | 1 | 1 | 2 | 1 | 0 |
| Minecraft | 1 | 1 | 3 | 0 | 0 |
| Apex Legends | 0 | 2 | 0 | 3 | 0 |
| Battlefield 6 | 0 | 2 | 0 | 3 | 0 |
| Arc Raiders | 0 | 1 | 0 | 4 | 0 |
Table 11: Annotator familiarity with each game. Values indicate the number of annotators (out of 5) at each familiarity level. Hours denote minimum play time.
Figure 7: Screenshot of the annotation interface in the single-video setting, shown for the game Valheim . Annotators label actions, states, events, and entities on a synchronized timeline.
Figure 8: Screenshot of the annotation interface in the multi-video setting, shown for the game Arc Raiders . Three synchronized video perspectives are displayed in parallel, with aligned timelines enabling annotators to capture cross-video events and temporal relations. The same explosion in the sky is captured by all three videos.
| Form | Entity | Code | Template |
| IDENT | SA | SA-IDENT | Which of the following actions did the POV player perform during the video? |
| SS | SS-IDENT | Which of the following best describes the POV player’s state in the video? | |
| OA | OA-IDENT | Which of the following actions did {other} perform during the video? | |
| OS | OS-IDENT | Which of the following best describes {other} ’s state in the video? | |
| WO | WO-IDENT | Which of the following objects appeared in the video? | |
| WE | WE-IDENT | Which of the following event occurred in the video? |
Table 12: Level 1 (perception) question templates. Entity types: SA (Self-Action), SS (Self-State), OA (Other-Action), OS (Other-State), WO (World-Object), WE (World-Event).
| Form | Ref → Ans | Code | Template |
|---|---|---|---|
| IDENT | SA → SS | SA2SS-IDENT | When the POV player was performing the action: “ {refCaption} ”, which of the following best describes their state? |
| SA → OA | SA2OA-IDENT | When the POV player was performing the action: “ {refCaption} ”, which of the following actions did {other} perform? | |
| SS → SA | SS2SA-IDENT | When the POV player’s “ {refCaption} ”, which of the following actions did they perform? | |
| OA → SS | OA2SS-IDENT | When {other} was performing the action: “ {refCaption} ”, which of the following best describes the POV player’s state? | |
| WO → SA | WO2SA-IDENT | At the moment when the object “ {refCaption} ” appeared, which of the following actions did the POV player perform? | |
| WE → OA | WE2OA-IDENT | At the moment when the event “ {refCaption} ” occurred, which of the following actions did {other} perform? |
Table 13: Level 2 entity-reference question templates (representative examples; full set covers all 30 Ref → Ans pairs across SA, SS, OA, OS, WO, WE). Code format: {Ref}2{Ans}-{Form} . Each EXIST question is further instantiated with distractor subtypes: True , Lexical , Scene , Temporal , and Role .
| Form | Ans | Code | Template |
|---|---|---|---|
| IDENT | SA | TR2SA-IDENT | During {timestamp} , which of the following actions did the POV player perform? |
| SS | TR2SS-IDENT | During {timestamp} , which of the following best describes the POV player’s state? | |
| OA | TR2OA-IDENT | During {timestamp} , which of the following actions did {other} perform? | |
| OS | TR2OS-IDENT | During {timestamp} , which of the following best describes {other} ’s state? | |
| WO | TR2WO-IDENT | During {timestamp} , which of the following objects appeared? | |
| WE | TR2WE-IDENT | During {timestamp} , which of the following events occurred? |
Table 14: Level 2 timestamp-reference (TR) question templates. The placeholder {timestamp} is replaced by a formatted time range such as [00:01 to 00:12] . Code format: TR2{Ans}-{Form} .
| Type | Ref → Ans | Code | Template |
| Cross-Video Reference (V1- { Ref } 2V2- { Ans } - { Form } ) | |||
| IDENT | SA → SA | V1-SA2V2-SA-IDENT | When POV1 player was performing the action: “ {refCaption} ”, which of the following actions did POV2 player perform at the same time? |
| SA → SS | V1-SA2V2-SS-IDENT | When POV1 player was performing the action: “ {refCaption} ”, which of the following best describes POV2 player’s state at the same time? | |
| OA → SA | V1-OA2V2-SA-IDENT | When {refOther} was performing the action: “ {refCaption} ” in POV1, which of the following actions did POV2 player perform at the same time? | |
| WE → WO | V1-WE2V2-WO-IDENT | At the moment when the event “ {refCaption} ” occurred in POV1, which of the following objects appeared in POV2 at the same time? | |
| EXIST | SA → SA | V1-SA2V2-SA-EXIST | When POV1 player was performing the action: “ {refCaption} ”, did POV2 player perform the action: “ {caption} ” at the same time? |
Table 15: Level 3 (cross-video) question templates. Cross-video reference templates cover all 36 Ref → Ans pairs (6 reference types 6 answer types) in both IDENT and EXIST forms; representative examples are shown. V1 / V2 are replaced by actual video indices at generation time. {refOther} refers to the other player in the reference video. POV-ID answer options are video numbers; ORDER options are formatted as “The POV player in Video is [action]”.