GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

Mingyu Ouyang; Siyuan Hu; Kevin Qinghong Lin; Hwee Tou Ng; Mike Zheng Shou

cs.CVApr 8, 2026

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

Authors: Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, Mike Zheng Shou

Organizations: National University of Singapore · University of Oxford

Abstract

Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.

Figures & tables

Figure 1: GameWorld covers 34 diverse games with 170 tasks for standardized evaluation of game agents.

Figure 2: Overview of the GameWorld benchmark with four modules: (i) MLLMs as game agents , (ii) Browser-based sandbox environment , (iii) Games & tasks library , and (iv) Outcome-based state-verifiable evaluation . This closes a continuous and interactive observation-action-verification loop for systematically evaluating game agents.

Game Agent Interfaces
Computer-Use Agent	Generalist Multimodal Agent
Action Space: Computer-use function calls Native tools of mouse and keyboard events. - mouse_move(x,y) # move pointer - left_click(x,y) # primary click - right_click(x,y) # secondary click - double_click(x,y) # double click - click_hold(x,y) # triple click - drag((x1,y1),(x2,y2)) # drag pointer - scroll_up(n) # scroll up - scroll_down(n) # scroll down - type(text) # text entry - press_key(key) # single key - press_keys(key1,key2) # key combo - wait(duration) # idle	Action Space: Game-specific function calls Semantic functions parsed into low-level controls. - move_forward() # player: forward - move_backward() # player: backward - move_left() # player: strafe left - move_right() # player: strafe right - look_up() # view: camera up - look_down() # view: camera down - look_left() # view: camera turn left - look_right() # view: camera turn right - action_jump() # action: player jump - action_duck() # action: player duck - weapon_fire() # item: fire weapon - no_op() # idle: no operation
Unified Control Space (Atomic Events)
Mouse: mouse_move(x,y) mouse_down(button) mouse_up(button) scroll(amount) Keyboard: key_down(key) key_up(key) Others: wait(duration) idle()

Table 1: Two game agent interfaces and action-space taxonomy. Both interfaces are normalized to a unified control space of atomic human-computer interaction events.

Benchmark	# Games	# Tasks	# Models	Vision- Centric	Config. Init. State	Task- Oriented	Parallel Inst.	Verif. Eval.	Notes
Static Benchmarks
GameQA [ 68 ]	30	158	8	✗	NA	✗	NA	✗	Code2Logic QA.
VideoGameQA [ 62 ]	800+	9	16	✓	NA	✗	NA	✗	Non-interactive QA dataset.
Interactive Benchmarks
MCU [ 84 ]	1	150	4	✓	✓	✓	✗	✗	Minecraft only.
LMGame-Bench [ 33 ]	6	6	13	✗	✗	✓	✗	✓	Text-centric benchmark.

Table 2: Comparison with representative game or computer-use agent benchmarks. #Tasks denotes the number of specific instructions, goals, or questions .

Game Genre	Example Screenshot	Key Mechanics	Games
Arcade (7)	Source: pac-man	Fast-paced, closed-loop control with dynamic multi-entity tracking, reactive evasion, and reward collection.	5-breakout 8-core-ball 15-google-snake 23-pacman 25-rocket-league-2d 33-worlds-hardest-game 34-worlds-hardest-game-2
Platformer (8)	Source: captaincallisto	Spatiotemporal navigation demanding precise physics-based movement, localized planning, and hazard evasion across structured terrains.	2-another-gentlemans-adventure 6-captaincallisto 10-doodle-jump 14-geodash 17-mario-game 22-ovo 24-restless-wing-syndrome 30-vex-3
Puzzle (7)	Source: astray	Discrete state-space exploration focusing on long-horizon strategic planning and logical decision-making.	1-2048 3-astray 16-hextris 19-minesweeper 27-stack 29-tetris 32-wordle
Runner (8)	Source: temple-run-2	Continuous state progression requiring high-frequency reactive control and precise timing for obstacle avoidance.	4-boxel-rebound 7-chrome-dino 9-cubefield 11-edge-surf 13-flappy-bird 21-ns-shaft 26-run-3 28-temple-run-2
Simulation (4)	Source: monkey-mart	Open-ended , multi-objective environments evaluating resource management, multi-character cooperation, or strategic exploration.	12-fireboy-and-watergirl 18-minecraft-clone-glm 20-monkey-mart 31-wolf3d

Table 3: Game genre of the GameWorld benchmark. Each row includes one representative screenshot, the dominant interaction mechanics, and the game IDs used in our evaluation.

ID	Game	Description	Gameplay Screenshot
1-2048	2048 [ 13 ]	Sliding-tile puzzle where the player merges matching tiles to build larger values under limited board space.
2-another-gentlemans-adventure	Another Gentleman’s Adventure [ 15 ]	Platform adventure centered on movement, jumping, coin collection, and enemy avoidance.
3-astray	Astray [ 66 ]	Maze-navigation puzzle in which the player must steer through a labyrinth to find the exit.
4-boxel-rebound	Boxel Rebound [ 19 ]	Precision auto-runner where the player times jumps to survive hazards and reach the end of each level.
5-breakout	Breakout [ 6 ]	Classic brick-breaking arcade game where the player controls a paddle to keep the ball in play and clear bricks.
6-captaincallisto	Captain Callisto [ 22 ]	Platform adventure with traversal, jumping, and jetpack-assisted movement toward the exit.

Table 4: Game inventory for the 34-game GameWorld benchmark.

ID	Game	Description	Gameplay Screenshot
13-flappy-bird	Flappy Bird [ 48 ]	One-button flying game that tests precise timing while weaving through pipes.
14-geodash	GeoDash [ 35 ]	Geometry-Dash-style auto-runner where success depends on tightly timed jumps over spikes and gaps.
15-google-snake	Google Snake [ 28 ]	Classic Snake variant where the agent grows by eating food while avoiding walls and self-collisions.
16-hextris	Hextris [ 31 ]	Hexagon-based matching puzzle where the agent rotates and places colored blocks to prevent overflow.
17-mario-game	Mario Game [ 24 ]	Super-Mario-style platformer with enemy avoidance, jumping, and long-horizon navigation to the flagpole.
18-minecraft-clone-glm	Minecraft Clone [ 85 ]	First-person sandbox game focused on movement, camera control, resource gathering, and direct world interaction.

Table 4 (continued): Game inventory for the 34-game GameWorld benchmark.

ID	Game	Description	Gameplay Screenshot
25-rocket-league-2d	Rocket League 2D [ 44 ]	Side-view car-soccer game requiring positioning, jumping, and ball control to score goals.
26-run-3	Run 3 [ 14 ]	Tunnel runner that combines lateral movement and jumps to cross gaps in a rotating corridor.
27-stack	Stack [ 39 ]	Timing puzzle in which moving blocks must be dropped with precise alignment to keep the tower stable.
28-temple-run-2	Temple Run 2 [ 36 ]	Endless runner requiring turn, jump, and slide decisions under high-speed reactive pressure.
29-tetris	Tetris [ 52 ]	Falling-block puzzle focused on line clearing, spatial planning, and managing long-term board structure.
30-vex-3	Vex 3 [ 2 ]	Precision platformer built around checkpoints, trap avoidance, and accurate movement through hazard-heavy levels.

Table 4 (continued): Game inventory for the 34-game GameWorld benchmark.

Model	Computer-Use Agent	Generalist Agent	Model Description
Proprietary
Claude-Sonnet-4.6 [ 5 ]	✓	✓	Anthropic multimodal model supporting computer-use.
Gemini-2.5-Computer-Use [ 29 ]	✓		Computer-use model built on Gemini 2.5 Pro.
Gemini-3-Flash-Preview [ 26 ]		✓	Google fast multimodal foundation model.
GLM-4.6V [ 32 ]		✓	Z.ai VLM with tool use.
GPT-5.2 [ 50 ]		✓	OpenAI multimodal function model with reasoning.

Table 5: Model profiles in GameWorld . Each model is evaluated as a Computer-Use Agent, a Generalist multimodal agent, or both.

Model	Arcade		Platformer		Puzzle		Runner		Simulation		Overall
Model	$\mathcal{SR}$	$\mathcal{PG}$	$\mathcal{SR}$	$\mathcal{PG}$	$\mathcal{SR}$	$\mathcal{PG}$	$\mathcal{SR}$	$\mathcal{PG}$	$\mathcal{SR}$	$\mathcal{PG}$	$\mathcal{SR}$	$\mathcal{PG}$	Rank
Human
Novice Player	45.7	55.5	60.0	65.6	51.4	63.1	60.0	72.0	60.0	62.0	55.3	64.1	–
Expert Player	65.7	73.9	85.0	88.0	68.6	77.1	82.5	87.8	85.0	86.0	77.1	82.6	–
Computer-Use Agents
Claude-Sonnet-4.6	8.6	27.2	22.5	36.5	20.0	43.8	30.0	55.6	10.0	16.8	19.4	38.3	2

Table 6: Main results on GameWorld across 34 games and 170 tasks. We report genre-level and overall

\mathcal{SR}

(Success Rate, %) and

\mathcal{PG}

(Progress, %) for 18 models (10 generalist multimodal agents and 8 computer-use agents). The final rank is determined by overall

\mathcal{PG}

Figure 3: Per-game progress heatmap across the GameWorld benchmark. Rows correspond to 18 evaluated game agents with model: (a) Claude-Sonnet-4.6, (b) Gemini-2.5-Computer-Use, (c) OpenAI-Computer-Use, (d) Qwen3-VL-Plus, (e) Seed-1.8, (f) Qwen3-VL-235B-A22B, (g) Qwen3-VL-30B-A3B, (h) UI-TARS-1.5-7B, (i) Claude-Sonnet-4.6, (j) Gemini-3-Flash-Preview, (k) GLM-4.6V, (l) GPT-5.2, (m) Grok-4.1-Fast-Reasoning, (n) Kimi-K2.5, (o) Qwen3-VL-Plus, (p) Seed-1.8, (q) Qwen3-VL-235B-A22B, and (r) Qwen3-VL-30B-A3B. (a)-(h) are Computer-Use Agents and (i)-(r) are Generalist Multimodal Agents. Colors represent average task progress for each game from high (green) to medium (yellow) to low (red) .

Model	Agent Interface	Repeats	Overall $\mathcal{SR}$	Overall $\mathcal{PG}$
Qwen3-VL-30B-A3B	Computer-Use Agent	10	12.7 $\pm$ 1.2	30.9 $\pm$ 1.1
Qwen3-VL-30B-A3B	Generalist Agent	10	12.5 $\pm$ 1.3	30.7 $\pm$ 1.1
Qwen3-VL-235B-A22B	Computer-Use Agent	10	13.8 $\pm$ 0.7	30.4 $\pm$ 0.7
Qwen3-VL-235B-A22B	Generalist Agent	10	13.6 $\pm$ 1.4	30.1 $\pm$ 0.5

Table 7: Repeat-averaged overall

\mathcal{SR}

and

\mathcal{PG}

for the four Qwen model–interface pairs used in the repeated-evaluation study. We report mean

\pm

standard deviation computed from the ten full-benchmark repeat averages.

Figure 4: Per-game average progress across the 34 benchmark games for the four Qwen model–interface pairs used in the repeated-evaluation study. Each panel corresponds to one model–interface pair, and each horizontal bar shows the mean progress over the same ten full-benchmark reruns. Error bars denote one run-level standard deviation.

Figure 5: Capability-aligned five-level curriculum profiles across agent interfaces and models. Left: Generalist agents. Right: Computer-Use agents. Each radar axis corresponds to one curriculum level and values are average task progress of all the games in the level.

Table 15

Table 16

Figure 6: Case-study visualization for game-agent interface comparison. This Mario page stacks a matched CUA trajectory above a Generalist trajectory so that their divergence can be attributed to the action interface rather than the backbone.

Figure 7: Case-study visualization for long-horizon simulation and resource collection. This Minecraft Clone trajectory shows locally plausible interactions that advance the progress, but still fail to fully meet the task target.

Figure 8: Case-study visualization for real-time reaction and timing control. This Flappy Bird window shows how a visually small timing error can still be mechanically decisive under tight control constraints.

Status	Runtime description
loading	Assets or engine bootstrap are not yet ready for model control.
menu	The game is initialized but still in a pre-play state such as title screen, level select, or pause menu.
ready	The game is initialized and ready to begin, but may still require one trusted start action.
playing	The gameplay loop is active and safe for model control.
paused	The game is temporarily paused; this status is used by the sandbox pause mechanism during model inference.
terminal	The current in-game episode has ended.

Table 12: Status values consumed by the readiness gate and evaluator.

Model	Input Tokens / Step	Output Tokens / Step	Total Cost (USD)
Claude-Sonnet-4.6 (Computer-Use)	3344.9	131.7	172.46
Claude-Sonnet-4.6 (Generalist)	4865.8	86.3	244.03
Gemini-2.5-Computer-Use	2086.6	15.4	41.06
Gemini-3-Flash-Preview	4005.6	40.5	29.13
GLM-4.6V	4820.2	253.2	24.79
GPT-5.2	3924.6	38.9	110.68

Table 13: Estimated benchmark cost of evaluating all 170 tasks for each model. Input / output token counts are averaged per step.

Explore similar work

Mar 25, 2026

GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents

Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective, capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World, a natural decomposition for multi-agent environments. From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.

Yunzhe Wang, Runhui Xu, Kexin Zheng +4

University of Southern California

AI Agent Benchmarks Egocentric Video QA Video QA Egocentric Video Understanding Game-Playing Agents

Ask PDF

Feb 11, 2026

GameDevBench: Evaluating Agentic Capabilities Through Game Development

Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.

Wayne Chi, Yixiong Fang, Arnav Yayavaram +8

Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA

Software Engineering Agents Agent Evaluation AI Agent Benchmarks

Ask PDF

Aug 13, 2026

PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce PlayWorld, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are available at https://github.com/kxding/PlayWorld.

Kaixin Ding, Xi Chen, Minghong Cai +9

The University of Hong Kong · Kling Team, Kuaishou Technology · The Chinese University of Hong Kong +1

Long-Horizon Agent Evaluation World Model Evaluation Interactive World Models Video World Models

Ask PDF