cs.CVApr 8, 2026
SaveGameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
Organizations: National University of Singapore · University of Oxford
Abstract
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.
Figures & tables
Figure 1: GameWorld covers 34 diverse games with 170 tasks for standardized evaluation of game agents.
Figure 2: Overview of the GameWorld benchmark with four modules: (i) MLLMs as game agents , (ii) Browser-based sandbox environment , (iii) Games & tasks library , and (iv) Outcome-based state-verifiable evaluation . This closes a continuous and interactive observation-action-verification loop for systematically evaluating game agents.
| Game Agent Interfaces | |
| Computer-Use Agent | Generalist Multimodal Agent |
| Action Space: Computer-use function calls Native tools of mouse and keyboard events. - mouse_move(x,y) # move pointer - left_click(x,y) # primary click - right_click(x,y) # secondary click - double_click(x,y) # double click - click_hold(x,y) # triple click - drag((x1,y1),(x2,y2)) # drag pointer - scroll_up(n) # scroll up - scroll_down(n) # scroll down - type(text) # text entry - press_key(key) # single key - press_keys(key1,key2) # key combo - wait(duration) # idle | Action Space: Game-specific function calls Semantic functions parsed into low-level controls. - move_forward() # player: forward - move_backward() # player: backward - move_left() # player: strafe left - move_right() # player: strafe right - look_up() # view: camera up - look_down() # view: camera down - look_left() # view: camera turn left - look_right() # view: camera turn right - action_jump() # action: player jump - action_duck() # action: player duck - weapon_fire() # item: fire weapon - no_op() # idle: no operation |
| Unified Control Space (Atomic Events) | |
| Mouse: mouse_move(x,y) mouse_down(button) mouse_up(button) scroll(amount) Keyboard: key_down(key) key_up(key) Others: wait(duration) idle() | |
Table 1: Two game agent interfaces and action-space taxonomy. Both interfaces are normalized to a unified control space of atomic human-computer interaction events.
| Benchmark | # Games | # Tasks | # Models | Vision- Centric | Config. Init. State | Task- Oriented | Parallel Inst. | Verif. Eval. | Notes |
| Static Benchmarks | |||||||||
| GameQA [ 68 ] | 30 | 158 | 8 | ✗ | NA | ✗ | NA | ✗ | Code2Logic QA. |
| VideoGameQA [ 62 ] | 800+ | 9 | 16 | ✓ | NA | ✗ | NA | ✗ | Non-interactive QA dataset. |
| Interactive Benchmarks | |||||||||
| MCU [ 84 ] | 1 | 150 | 4 | ✓ | ✓ | ✓ | ✗ | ✗ | Minecraft only. |
| LMGame-Bench [ 33 ] | 6 | 6 | 13 | ✗ | ✗ | ✓ | ✗ | ✓ | Text-centric benchmark. |
Table 2: Comparison with representative game or computer-use agent benchmarks. #Tasks denotes the number of specific instructions, goals, or questions .
| Game Genre | Example Screenshot | Key Mechanics | Games |
| Arcade (7) | Source: pac-man | Fast-paced, closed-loop control with dynamic multi-entity tracking, reactive evasion, and reward collection. | 5-breakout 8-core-ball 15-google-snake 23-pacman 25-rocket-league-2d 33-worlds-hardest-game 34-worlds-hardest-game-2 |
| Platformer (8) | Source: captaincallisto | Spatiotemporal navigation demanding precise physics-based movement, localized planning, and hazard evasion across structured terrains. | 2-another-gentlemans-adventure 6-captaincallisto 10-doodle-jump 14-geodash 17-mario-game 22-ovo 24-restless-wing-syndrome 30-vex-3 |
| Puzzle (7) | Source: astray | Discrete state-space exploration focusing on long-horizon strategic planning and logical decision-making. | 1-2048 3-astray 16-hextris 19-minesweeper 27-stack 29-tetris 32-wordle |
| Runner (8) | Source: temple-run-2 | Continuous state progression requiring high-frequency reactive control and precise timing for obstacle avoidance. | 4-boxel-rebound 7-chrome-dino 9-cubefield 11-edge-surf 13-flappy-bird 21-ns-shaft 26-run-3 28-temple-run-2 |
| Simulation (4) | Source: monkey-mart | Open-ended , multi-objective environments evaluating resource management, multi-character cooperation, or strategic exploration. | 12-fireboy-and-watergirl 18-minecraft-clone-glm 20-monkey-mart 31-wolf3d |
Table 3: Game genre of the GameWorld benchmark. Each row includes one representative screenshot, the dominant interaction mechanics, and the game IDs used in our evaluation.
| ID | Game | Description | Gameplay Screenshot |
| 1-2048 | 2048 [ 13 ] | Sliding-tile puzzle where the player merges matching tiles to build larger values under limited board space. | |
| 2-another-gentlemans-adventure | Another Gentleman’s Adventure [ 15 ] | Platform adventure centered on movement, jumping, coin collection, and enemy avoidance. | |
| 3-astray | Astray [ 66 ] | Maze-navigation puzzle in which the player must steer through a labyrinth to find the exit. | |
| 4-boxel-rebound | Boxel Rebound [ 19 ] | Precision auto-runner where the player times jumps to survive hazards and reach the end of each level. | |
| 5-breakout | Breakout [ 6 ] | Classic brick-breaking arcade game where the player controls a paddle to keep the ball in play and clear bricks. | |
| 6-captaincallisto | Captain Callisto [ 22 ] | Platform adventure with traversal, jumping, and jetpack-assisted movement toward the exit. |
Table 4: Game inventory for the 34-game GameWorld benchmark.
| ID | Game | Description | Gameplay Screenshot |
| 13-flappy-bird | Flappy Bird [ 48 ] | One-button flying game that tests precise timing while weaving through pipes. | |
| 14-geodash | GeoDash [ 35 ] | Geometry-Dash-style auto-runner where success depends on tightly timed jumps over spikes and gaps. | |
| 15-google-snake | Google Snake [ 28 ] | Classic Snake variant where the agent grows by eating food while avoiding walls and self-collisions. | |
| 16-hextris | Hextris [ 31 ] | Hexagon-based matching puzzle where the agent rotates and places colored blocks to prevent overflow. | |
| 17-mario-game | Mario Game [ 24 ] | Super-Mario-style platformer with enemy avoidance, jumping, and long-horizon navigation to the flagpole. | |
| 18-minecraft-clone-glm | Minecraft Clone [ 85 ] | First-person sandbox game focused on movement, camera control, resource gathering, and direct world interaction. |
Table 4 (continued): Game inventory for the 34-game GameWorld benchmark.
| ID | Game | Description | Gameplay Screenshot |
| 25-rocket-league-2d | Rocket League 2D [ 44 ] | Side-view car-soccer game requiring positioning, jumping, and ball control to score goals. | |
| 26-run-3 | Run 3 [ 14 ] | Tunnel runner that combines lateral movement and jumps to cross gaps in a rotating corridor. | |
| 27-stack | Stack [ 39 ] | Timing puzzle in which moving blocks must be dropped with precise alignment to keep the tower stable. | |
| 28-temple-run-2 | Temple Run 2 [ 36 ] | Endless runner requiring turn, jump, and slide decisions under high-speed reactive pressure. | |
| 29-tetris | Tetris [ 52 ] | Falling-block puzzle focused on line clearing, spatial planning, and managing long-term board structure. | |
| 30-vex-3 | Vex 3 [ 2 ] | Precision platformer built around checkpoints, trap avoidance, and accurate movement through hazard-heavy levels. |
Table 4 (continued): Game inventory for the 34-game GameWorld benchmark.
| Model | Computer-Use Agent | Generalist Agent | Model Description |
| Proprietary | |||
| Claude-Sonnet-4.6 [ 5 ] | ✓ | ✓ | Anthropic multimodal model supporting computer-use. |
| Gemini-2.5-Computer-Use [ 29 ] | ✓ | Computer-use model built on Gemini 2.5 Pro. | |
| Gemini-3-Flash-Preview [ 26 ] | ✓ | Google fast multimodal foundation model. | |
| GLM-4.6V [ 32 ] | ✓ | Z.ai VLM with tool use. | |
| GPT-5.2 [ 50 ] | ✓ | OpenAI multimodal function model with reasoning. | |
Table 5: Model profiles in GameWorld . Each model is evaluated as a Computer-Use Agent, a Generalist multimodal agent, or both.
| Model | Arcade | Platformer | Puzzle | Runner | Simulation | Overall | |||||||
| Rank | |||||||||||||
| Human | |||||||||||||
| Novice Player | 45.7 | 55.5 | 60.0 | 65.6 | 51.4 | 63.1 | 60.0 | 72.0 | 60.0 | 62.0 | 55.3 | 64.1 | – |
| Expert Player | 65.7 | 73.9 | 85.0 | 88.0 | 68.6 | 77.1 | 82.5 | 87.8 | 85.0 | 86.0 | 77.1 | 82.6 | – |
| Computer-Use Agents | |||||||||||||
| Claude-Sonnet-4.6 | 8.6 | 27.2 | 22.5 | 36.5 | 20.0 | 43.8 | 30.0 | 55.6 | 10.0 | 16.8 | 19.4 | 38.3 | 2 |
Table 6: Main results on GameWorld across 34 games and 170 tasks. We report genre-level and overall (Success Rate, %) and (Progress, %) for 18 models (10 generalist multimodal agents and 8 computer-use agents). The final rank is determined by overall .
Figure 3: Per-game progress heatmap across the GameWorld benchmark. Rows correspond to 18 evaluated game agents with model: (a) Claude-Sonnet-4.6, (b) Gemini-2.5-Computer-Use, (c) OpenAI-Computer-Use, (d) Qwen3-VL-Plus, (e) Seed-1.8, (f) Qwen3-VL-235B-A22B, (g) Qwen3-VL-30B-A3B, (h) UI-TARS-1.5-7B, (i) Claude-Sonnet-4.6, (j) Gemini-3-Flash-Preview, (k) GLM-4.6V, (l) GPT-5.2, (m) Grok-4.1-Fast-Reasoning, (n) Kimi-K2.5, (o) Qwen3-VL-Plus, (p) Seed-1.8, (q) Qwen3-VL-235B-A22B, and (r) Qwen3-VL-30B-A3B. (a)-(h) are Computer-Use Agents and (i)-(r) are Generalist Multimodal Agents. Colors represent average task progress for each game from high (green) to medium (yellow) to low (red) .
| Model | Agent Interface | Repeats | Overall | Overall |
| Qwen3-VL-30B-A3B | Computer-Use Agent | 10 | 12.7 1.2 | 30.9 1.1 |
| Qwen3-VL-30B-A3B | Generalist Agent | 10 | 12.5 1.3 | 30.7 1.1 |
| Qwen3-VL-235B-A22B | Computer-Use Agent | 10 | 13.8 0.7 | 30.4 0.7 |
| Qwen3-VL-235B-A22B | Generalist Agent | 10 | 13.6 1.4 | 30.1 0.5 |
Table 7: Repeat-averaged overall and for the four Qwen model–interface pairs used in the repeated-evaluation study. We report mean standard deviation computed from the ten full-benchmark repeat averages.
Figure 4: Per-game average progress across the 34 benchmark games for the four Qwen model–interface pairs used in the repeated-evaluation study. Each panel corresponds to one model–interface pair, and each horizontal bar shows the mean progress over the same ten full-benchmark reruns. Error bars denote one run-level standard deviation.
Figure 5: Capability-aligned five-level curriculum profiles across agent interfaces and models. Left: Generalist agents. Right: Computer-Use agents. Each radar axis corresponds to one curriculum level and values are average task progress of all the games in the level.
Figure 6: Case-study visualization for game-agent interface comparison. This Mario page stacks a matched CUA trajectory above a Generalist trajectory so that their divergence can be attributed to the action interface rather than the backbone.
Figure 7: Case-study visualization for long-horizon simulation and resource collection. This Minecraft Clone trajectory shows locally plausible interactions that advance the progress, but still fail to fully meet the task target.
Figure 8: Case-study visualization for real-time reaction and timing control. This Flappy Bird window shows how a visually small timing error can still be mechanically decisive under tight control constraints.
| Status | Runtime description |
| loading | Assets or engine bootstrap are not yet ready for model control. |
| menu | The game is initialized but still in a pre-play state such as title screen, level select, or pause menu. |
| ready | The game is initialized and ready to begin, but may still require one trusted start action. |
| playing | The gameplay loop is active and safe for model control. |
| paused | The game is temporarily paused; this status is used by the sandbox pause mechanism during model inference. |
| terminal | The current in-game episode has ended. |
Table 12: Status values consumed by the readiness gate and evaluator.
| Model | Input Tokens / Step | Output Tokens / Step | Total Cost (USD) |
| Claude-Sonnet-4.6 (Computer-Use) | 3344.9 | 131.7 | 172.46 |
| Claude-Sonnet-4.6 (Generalist) | 4865.8 | 86.3 | 244.03 |
| Gemini-2.5-Computer-Use | 2086.6 | 15.4 | 41.06 |
| Gemini-3-Flash-Preview | 4005.6 | 40.5 | 29.13 |
| GLM-4.6V | 4820.2 | 253.2 | 24.79 |
| GPT-5.2 | 3924.6 | 38.9 | 110.68 |
Table 13: Estimated benchmark cost of evaluating all 170 tasks for each model. Input / output token counts are averaged per step.