From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

Zixuan Dong; Yanghai Wang; Hanyan Bian; Jiaxi Tao; Junze Li; Jiadong Wang; Xinyu Che; Lin Liu

cs.AIOct 19, 2026

From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

Authors: Zixuan Dong, Yanghai Wang, Hanyan Bian, Jiaxi Tao, Junze Li, Jiadong Wang, Xinyu Che, Lin Liu,

Organizations: National University of Defense Technology · Nanjing University · The University of Tokyo · Xi’an Jiaotong University ·

Abstract

A game ships when a player can launch it, finish it, and return to a saved world, none of which a successful build guarantees, yet game-creation agents are mostly tested on one engine or on one-prompt browser games. We present CEGame-Bench, the first long-horizon benchmark for complete-game creation across five production engines: Defold, Godot, Roblox Studio, Unity, and Unreal Engine. Agents build each game from an empty project and a design document in runs averaging up to 295 steps and 65M tokens; the document doubles as a contract, and a hidden evaluator accepts a game only by real-input play. Under four scaffolds, agents earn most integration points but usually under half the gameplay points, and their games reach the ending in only 16% of runs, while references built by engineers with coding agents score near the maximum, which places this artifact-to-play gap in the agents. No two engines rank configurations alike (Kendall's tau 0.14-0.62), and the same game usually scores higher on Unity and Unreal Engine, which check more before play. We release Game Completion Boost, an agent skill for requirement-driven playtesting and repair that raises the execution-based score of three models by 14.1-24.3 points.

Figures & tables

Figure 1: The artifact-to-play gap. CEGame-Bench plays each delivered game through complete player journeys on five production engines.

Panel of Table 1 — Table 1: Comparison with prior game-creation work. Replay: the evaluator plays the artifact with input; Hidden eval.: unseen checks decide the score; Reopen: persistence is checked in a fresh process; Engine swap: a task is re-run on another engine.

Figure 2: Construction and composition of CEGame-Bench . (a) Tasks are specified as contracts, built, evaluated independently, and calibrated by play. (b) Coverage: engines, difficulty tiers, spatial forms, and genres.

Figure 3: GCB workflow: build the player path, play and inspect, then repair, replay, and restore if needed.

Configuration		Defold								Godot
Scaffold	Model	D1	D2	D3	D4	D5	D6	$\mathbf{F}$	$\mathbf{O}$	D1	D2	D3	D4	D5	D6	$\mathbf{F}$	$\mathbf{O}$
	GPT-6 Astra	10.0	20.3	11.3	14.8	6.2	9.5	62.7	72.2	9.4	15.8	8.2	12.1	4.4	5.1	49.8	54.8
	GPT-5.6 Sol	10.0	19.0	9.8	8.7	5.0	5.8	52.5	58.3	9.6	3.8	1.9	4.9	1.3	3.1	21.5	24.6
	GPT-5.6 Terra	7.2	7.4	8.0	8.0	2.6	2.7	33.2	35.9	9.2	1.6	1.5	3.1	1.1	3.4	16.4	19.8
Codex CLI	DeepSeek-V4-Pro	9.5	7.2	3.7	3.7	1.8	1.8	25.8	27.6	8.1	10.2	5.4	7.2	3.3	2.7	34.2	36.9
DeepSeek Harness	DeepSeek-V4-Pro	10.0	13.2	8.0	7.8	2.6	3.7	41.6	45.3	9.8	9.6	4.6	8.1	3.3	3.6	35.4	39.0

Table 2: Scores of seven scaffold–model configurations on each engine track and across all five engines. D1–D6 are integration, completeness, mechanics, lifecycle, feedback, and presentation, with D6 capped when the player journey fails;

F

sums D1–D5 (out of 90) and

O

adds D6 (out of 100). Bold marks the best

F

and

O

per panel; entries are rounded independently and may differ from their sums by 0.1.

			Script-centric engine ( $S$ )
	Engine	Metric	Defold	Godot	Roblox
Heavyweight ( $H$ )	Unity	$\bar{F}_{H}$	$40.6_{\scriptscriptstyle\pm\,8.5}$	$38.0_{\scriptscriptstyle\pm\,8.9}$	$35.0_{\scriptscriptstyle\pm\,12.0}$
		$\bar{F}_{S}$	$29.0_{\scriptscriptstyle\pm\,11.1}$	$27.6_{\scriptscriptstyle\pm\,6.4}$	$39.3_{\scriptscriptstyle\pm\,9.7}$
		$\Delta F$	$\mathbf{+11.6}_{\scriptscriptstyle\pm\,4.3}$	$\mathbf{+10.4}_{\scriptscriptstyle\pm\,3.1}$	$\mathbf{-4.3}_{\scriptscriptstyle\pm\,5.4}$
	Unreal Engine	$\bar{F}_{H}$	$49.0_{\scriptscriptstyle\pm\,10.6}$	$45.0_{\scriptscriptstyle\pm\,12.1}$	$34.0_{\scriptscriptstyle\pm\,10.4}$
		$\bar{F}_{S}$	$27.8_{\scriptscriptstyle\pm\,14.3}$	$23.5_{\scriptscriptstyle\pm\,13.2}$	$25.5_{\scriptscriptstyle\pm\,16.2}$

Table 3: The same games on two engines: mean

F

on the heavyweight (

\bar{F}_{H}

, rows) and script-centric (

\bar{F}_{S}

, columns) engine and

\Delta F=\bar{F}_{H}-\bar{F}_{S}

, with standard deviations over games as subscripts. Each pair pools its ten swapped games, five native to each engine.

Figure 4: Construction cost against

F

for a uniform sample of game–configuration pairs: mean workflow steps (a, c) and mean tokens on a log scale (b, d), colored by engine (a, b) or model (c, d; DeepSeek-V4-Pro pools its two scaffolds).

Configuration	$\bm{F}$	$\bm{\Delta F}$	$\bm{O}$	$\bm{\Delta O}$
DeepSeek-V4-Pro	50.6	—	54.8	—
DeepSeek-V4-Pro + GCB	71.6	+21.0	76.3	+21.5
GPT-5.6 Sol	27.0	—	30.9	—
GPT-5.6 Sol + GCB	41.1	+14.1	45.6	+14.7
GPT-5.6 Terra	9.7	—	10.5	—
GPT-5.6 Terra + GCB	34.0	+24.3	36.3	+25.8

Table 4: GCB under Codex CLI on the 15 games, three per engine track, ordered by baseline

O

\Delta F

and

\Delta O

are gains over the paired baseline.

Engine	Engine build	Language	Scene representation	Checks before play	Evaluator adapter
Script-centric engines
Defold	1.13.0	Lua	Data-driven game objects and collections.	Resource and reference validation; Lua syntax checks during build.	Engine Lua hook.
Godot	4.7.1.stable	GDScript	Node trees serialized as text scenes.	GDScript parsing and scene-loading diagnostics.	Headless script.
Roblox Studio	0.742.0.7421053	Luau	Studio instance trees stored in local .rbxl files.	Luau analysis and editor diagnostics.	Studio plugin and local play-test.
Heavyweight engines
Unity	6.0 LTS 6000.0.80f1	C#	YAML scenes and prefabs with an asset-import pipeline.	C# compilation, serialized-reference checks, and Unity tests.	PlayMode harness.

Table 5: Engine builds, project representations, pre-play diagnostics, and evaluator adapters for the five tracks.

#	GDD section	Information provided to the agent	Symbol	Dimensions
1	Pitch and core loop	Game premise, session structure, and target duration.	—	D2
2	Engine and toolchain	Engine build, dependencies, and build/run commands.	—	D1
3	Controls	Input devices, bindings, actions, and press/hold/release behavior.	$\mathcal{A}$	D3, D5
4	Entities and attributes	Required actors, props, UI elements, and numerical attributes.	$\mathcal{N}$	D2, D3
5	Scene inventory	Entry scene, levels, scene contents, and connections.	$\mathcal{S}_{a}$	D1, D2
6	Mechanics and rules	Action resolution, costs, resources, and collision behavior.	$\mathcal{S}_{r}$	D3

Table 6: The shared GDD structure and its mapping to contract symbols and evaluation dimensions.

GDD §	Requirement	Sym.	Dim.	$\bm{w}$	Route	Leaf assertion
5	The game opens on the title menu; Start enters the Harbor level.	$\mathcal{S}_{a}$	D1	2	Static, input	The entry scene is the title menu; after Start , level.id is harbor .
3	Press Shift to dash in the facing direction.	$\mathcal{A}$	D3	3	Input + read	After Shift, player.state is dash and player.pos moves along player.facing .
6	A dash spends one charge.	$\mathcal{S}_{r}$	D3	3	Input + read	player.charges decreases by exactly one per dash.
7	A new dash is rejected for 0.4 s after the last one.	$\mathcal{T}$	D3	2	Input + read	A second dash at 0.2 s is rejected; one at 0.5 s is accepted.
8	With no charges left, dash does nothing.	$\mathcal{B}$	D3	2	Input + read	At player.charges $=0$ , Shift leaves position and charges unchanged.
10	Charges never exceed the maximum.	$\mathcal{I}$	D3	2	Input + read	During a 10 s recharge, player.charges $\leq$ player.max_charges at every sample.

Table 7: Example requirement-to-check mapping. Sym., Dim., and

w

denote the contract symbol, score dimension, and leaf weight (1 minor, 2 major, 3 critical).

Stage	Evidence	Feeds
Static	import and compile, entry scene, serialized references, manifests	D1, D2
Local dynamic	real input, $\Phi$ snapshots, collision, AI, and resource assertions, negative controls	D2, D3, D5
Transition	scene change, resource transfer, death and restart, pause and resume, terminal freeze	D3, D4
Player journey	entry-to-terminal play with public controls, closed-loop input, step budget	$J$
Fresh lifecycle	dirty close, new process, clean entry, restored references	D4, $\Lambda$
Visual	reference-state captures, paired reference images, written rubric, artifact hashes	D6

Table 8: Evaluator stages, their evidence sources, and the dimensions or outcome flags they feed.

ID	$W_{d}$	Leaves	Name	Primary evidence
D1	10	5.1	Integration	import, compile, entry scene, serialized references, manifests
D2	30	14.0	Completeness	required entities, levels, modes, UI, authored content
D3	20	11.3	Mechanics	actions, rules, resources, collision, AI, negative branches
D4	20	8.8	Lifecycle	transitions, terminal state, restart, save, fresh reopen
D5	10	4.3	Feedback	controls, HUD, observable response to every player action
D6	10	3.7	Presentation	asset assembly, layout, layering, style, and finish against reference captures

Table 9: The six outcome dimensions, shared by all five tracks.

W_{d}

is the dimension weight and Leaves the mean number of leaf checks per task (47.2 in total). D1–D5 are scored by execution and D6 by a vision–language model.

State	Score effect	Meaning
pass	full weight	obligation observed under valid preconditions
fail_observed	zero, in denominator	submission violates the obligation under valid observation
product_blocked	zero, in denominator	submission itself prevents the required state or action
setup_failed	excluded	fixture preparation failed, also through the alternate route
observer_failed	excluded	evidence acquisition failed or disagreed across routes
inconclusive	excluded	evidence insufficient; never inferred from absence

Table 10: Leaf states and their effect on the score. The first three are judgments of the submission; the last three leave the denominator.

Engine	Sc.	Ent.	States	Steps	Leaves
Defold	2.1	11.8	7.4	31.0	40.8
Godot	2.3	13.1	8.1	35.0	45.1
Roblox Studio	2.6	14.9	8.6	41.0	48.0
Unity	2.8	15.6	8.9	41.0	49.5
Unreal Engine	3.2	16.4	9.3	45.0	52.6
All	2.6	14.4	8.5	38.6	47.2

Table 11: Mean task structure. Sc./Ent.: scenes/entities; States: state nodes; Steps: reference-play inputs; Leaves: leaf checks.

Family	Examples	Stressor
T1 UI and rule puzzle	Four Shot, Arrow Escape	authored content, event rules, input edges, reset
T2 Action	Driftfolk Chronicle, Bomb Frog	collision, timing, camera, feedback
T3 3D spatial action	Dungeon Duel, Noir Roombreak	navigation, combat state, multi-room transitions
T4 Systems	Relay Foundry, Ape Tower Defense	resources, AI, dependency graphs, persistence
T5 Narrative	There Are Signs	branch tables, authored presentation, visual assembly

Table 12: Task families and the stressor each family is designed to expose.

Stage	Required action
Specify	Map public requirements, prerequisites, and resources.
Build	Complete the required path; launch early.
Inspect	Link real input to state and image observations.
Repair	Archive a repair anchor; invalidate affected checks.
Replay	Recheck target and protected checks; retain or restore.
Finalize	Play the full journey; reopen when required.

Table 13: Execution stages of GCB.

Scaffold	Version	Model	ID	Eff.
		Astra	gpt-6-astra	max
		Sol	gpt-5.6-sol	max
		Terra	gpt-5.6-terra	max
Codex CLI	0.159.2	V4-Pro	deepseek-v4-pro	max
DS Harness	0.2.0-rc.2	V4-Pro	deepseek-v4-pro	max
Claude Code	2.1.285	Opus 5	claude-opus-5	max

Table 14: Scaffold versions, model identifiers, and reasoning-effort settings. DS Harness is DeepSeek Harness.

Engine	D1 (%)	Play (%)	Gap (pp)	Mean $\bm{F}$	Low (%)	Spread
Defold	88.6	37.2	51.4	38.6	71.4	56.9
Godot	88.7	23.4	65.3	27.6	85.7	35.3
Roblox Studio	91.3	35.7	55.6	37.7	85.7	53.6
Unity	97.0	56.9	40.1	55.2	42.9	29.0
Unreal Engine	94.6	30.4	64.1	33.7	85.7	51.3

Table 15: Summary derived from Table 2 . D1 and Play normalize integration and gameplay scores; Gap is their difference. Low is the share of configurations below 50% gameplay, and Spread is the range of

F

	Godot	Roblox	Unity	Unreal
Defold	0.52	0.52	0.62	0.43
Godot		0.62	0.33	0.33
Roblox Studio			0.52	0.14
Unity				0.43

Table 16: Kendall’s

\tau

between engine-specific configuration rankings, computed using

F

from Table 2 .

Table 21

Configuration		Workflow steps (%)
Scaffold	Model	Edit	Build	Play	View	Other
	GPT-6 Astra	26.3	17.4	7.7	5.6	43.0
	GPT-5.6 Sol	17.1	19.9	5.3	4.8	52.9
	GPT-5.6 Terra	31.3	22.4	5.0	4.2	37.1
Codex CLI	DeepSeek-V4-Pro	28.3	8.9	8.7	4.6	49.5
DeepSeek Harness	DeepSeek-V4-Pro	34.1	6.3	3.1	4.5	52.0

Table 19: Action shares in the construction traces of native runs (%). Other covers reading files and documents, planning, and remaining actions.

Panel of Figure 5 — Figure 5: Three delivered games from the bottom, middle, and top of the score range, as captured by the evaluator. (a) Captain2D Platformer on Unreal Engine ( $O=15.2$ ): the player character, enemies, and springs stand below the room. (b) RollBall on Unreal Engine ( $O=57.2$ ): the HUD reports a trap hit while the cropped view leaves the ball at the lower edge. (c) Relay Foundry on Roblox Studio ( $O=95.7$ ): the rules hold, but system buttons and the HUD crowd the world signs.

Automated Game Testing

See all 10 papers

Jun 16, 2026

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22

1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5

AI Agent Benchmarks Agentic Code Generation Code Generation Evaluation

Ask PDF

Sep 19, 2026

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.

Shutong Wu, Kevin Calderone, Andy Tsen

University of Rochester · RamenVR

Coding Agents AI Agent Evaluation Code Generation Benchmarks Benchmark Design

Ask PDF

Oct 12, 2026

Game-Creation-Agents: Autonomous Game Production

Game-Creation-Agents (GCA) turns a natural-language request into a runnable Phaser 4 browser game. The pipeline first builds a multimodal game design document that couples mechanics and scene contracts to grounded visual references. It then projects that document into a runtime scaffold, machine-readable contracts, and deterministic tests. Eight specialized agents next carry out production: scenes are implemented serially, assets are produced in parallel, and one reduced manifest joins the two branches into a single build. Build-linked real-UI checks and a code-blind critic then verify that build and decide whether the game may ship. Each blocker returns to the agent that owns the affected artifact for a bounded repair, and the game is released once all the gates pass. We evaluate finished games separately, through their rendered interfaces alone, over prompt-matched tasks run under a common protocol. GCA achieves stronger requirement satisfaction, core mechanics, content depth, and functional visuals than a comparable open pipeline. In a side-by-side human study, raters also score GCA's games higher on every dimension and prefer them overall.

Tencent Multimodal RL Team, Xiangxin Zhou, Jiaqi Tang +15

Multimodal Reinforcement Learning Team, Tencent

AI Coding Agents Software Engineering Agents

Ask PDF

Abstract

Automated Game Testing

Explore similar work

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Game-Creation-Agents: Autonomous Game Production