GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
Authors: Boyu Qiao, Zixin Tang, Xiaoshuai Hao, Wenbo Li
Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Zhongguancun Laboratory · Xiaomi EV · Alibaba DAMO Academy
Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research has extensively evaluated the ability of coding agents to generate programs from textual specifications. However, under black-box conditions where neither source code nor documentation is available, it remains underexplored whether an agent can induce the rules solely through visual observation and active interaction and reproduce the target system as a verifiable executable system. To this end, we present GameReplica, a closed-loop evaluation framework for end-to-end black-box game replication that covers the full perception, exploration, induction, reproduction, and verification pipeline. GameReplica comprises 125 tasks spanning 25 games across 5 core mechanism families, with each game instantiated at five difficulty levels. The tasks require an agent to access the target game only through screenshots and an action interface, induce the key visual elements and gameplay rules from pixel feedback and interaction outcomes, and generate a self-contained, runnable game replica that can be automatically verified by an external program. Experiments show that current coding agents still face substantial challenges in end-to-end black-box replication: the best-performing model (Claude Opus 4.8) achieves an overall score of 71.6%, while the remaining models score only 4.0%--42.9%. Further analysis reveals a consistent pattern across all models: visual-fidelity scores are substantially higher than implementation- and rule-consistency scores, indicating that agents replicate visual appearance more readily than game mechanics. The difficulty levels further amplify the performance gap: from L1 to L5, the overall score of weaker agents drops sharply, whereas that of the best-performing agent declines only slightly.
Figures & tables
Figure 1: Motivation for GameReplica. (a) Program-synthesis and software-engineering benchmarks ask agents to implement a supplied specification. (b) Vision-to-code benchmarks provide visual evidence or interaction trajectories. (c) Game benchmarks evaluate actions in a fixed environment. (d) GameReplica instead requires an agent to actively recover the latent specification of a black-box game, reconstruct it as an executable program, and verify the resulting replica.
Figure 2: Overview of GameReplica. Left: Complexity-Controlled Multi-Difficulty Task Construction expands each base game into five levels (L1–L5) with a semantically aligned spec, rubric, and source code. Middle: Pixel-Only Black-Box Game Replication requires the agent to actively explore the running game through a screenshot–action interface and produce a self-contained executable replica together with an induced specification. Right: Reference-Grounded Replication Evaluation compares the replica against curated references along rule consistency, implementation consistency, and visual fidelity.
Figure 3: Environment interaction across models (agent runtime fixed to Claude Code). (a) Mean per-instance HTTP calls broken down over five endpoint types (action / reset / screenshot / keyboard / other), separating the observation phase (left axis) from the self-check phase (right axis). (b) Total HTTP calls per instance (process cost) plotted against Overall replication score.
Figure 4: Tool-call counts per instance. Each box plot shows the per-instance distribution of tool calls; the white diamond marks the mean μ , annotated above the box. (a) varying the model with the agent runtime fixed; (b) varying the agent runtime with the model fixed.
Figure 5: Environment interaction across agent runtimes (model fixed to Claude Opus 4.8). (a) Mean per-instance HTTP calls over the five endpoint types, separating the observation phase (left axis) from the self-check phase (right axis). (b) Total HTTP calls per instance (process cost) plotted against Overall replication score.
Figure 6: Outcome-level failure decomposition of game replication. Each bar shows a producer’s share of the 125 instances across the five tiers (F1 No output, F2 Unrecognizable, F3 Rule violation, F4 Visual mismatch, F5 Faithful). The dashed line marks the producible boundary.
Figure 7: Process-level failure attribution of game replication. Blue, magenta, and green color families correspond to the Exploration, Implementation, and Self-check phases respectively; lighter and darker shades distinguish sub-modes within each phase. The dashed line on each bar marks the boundary between the Exploration phase and the subsequent phases.
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.
Kun Chen, Haorong Hong, Peizhong Gao +7
Hunyuan Team, Tencent · MAIS, Institute of Automation, Chinese Academy of Sciences · Lightspeed Studios, Tencent +3
Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game can end in a valid state even after violating its rules during the run. Current game-development benchmarks replay fixed examples, score videos, or ask another model to judge the result. However, no existing benchmark checks game rules throughout execution across varied evaluator-selected scenarios while ensuring exactly reproducible verdicts. We introduce GameLogicBench, a benchmark of 72 gameplay-logic tasks in Godot projects. An automated evaluator checks each game's rules at every simulation tick. Across 403 hand-designed scenarios, seeded parameter variations produce 1,451 test cases. To ensure that the evaluator measures behavior rather than implementation choice, it must accept different correct implementations for each task while rejecting mutants, implementations with one required capability removed. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. Across 20 combinations of language models and scaffolds, the best observed run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as task scope expands from isolated mechanics, through interacting systems, to repository-scale features. Agents inspect code more often and make more tool calls on repository-scale tasks than on isolated-mechanic tasks. Most unsuccessful submissions are runnable, but implement some required game behavior incorrectly. We compared versions of our benchmark evaluator built with and without validation using mutants. Without this validation, incorrect agent submissions passed. A separate analysis finds agents copying code from public repositories when network access is open. Reliable evaluation thus depends both on what the tests reject and on what external code agents can access.