From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

Zixuan Dong; Yanghai Wang; Hanyan Bian; Jiaxi Tao; Junze Li; Jiadong Wang; Xinyu Che; Lin Liu

cs.AIOct 19, 2026

From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

Authors: Zixuan Dong, Yanghai Wang, Hanyan Bian, Jiaxi Tao, Junze Li, Jiadong Wang, Xinyu Che, Lin Liu,

Organizations: National University of Defense Technology · Nanjing University · The University of Tokyo · Xi’an Jiaotong University ·

Abstract

A game ships when a player can launch it, finish it, and return to a saved world, none of which a successful build guarantees, yet game-creation agents are mostly tested on one engine or on one-prompt browser games. We present CEGame-Bench, the first long-horizon benchmark for complete-game creation across five production engines: Defold, Godot, Roblox Studio, Unity, and Unreal Engine. Agents build each game from an empty project and a design document in runs averaging up to 295 steps and 65M tokens; the document doubles as a contract, and a hidden evaluator accepts a game only by real-input play. Under four scaffolds, agents earn most integration points but usually under half the gameplay points, and their games reach the ending in only 16% of runs, while references built by engineers with coding agents score near the maximum, which places this artifact-to-play gap in the agents. No two engines rank configurations alike (Kendall's tau 0.14-0.62), and the same game usually scores higher on Unity and Unreal Engine, which check more before play. We release Game Completion Boost, an agent skill for requirement-driven playtesting and repair that raises the execution-based score of three models by 14.1-24.3 points.

Figures & tables

Figure 1: The artifact-to-play gap. CEGame-Bench plays each delivered game through complete player journeys on five production engines.

Panel of Table 1 — Table 1: Comparison with prior game-creation work. Replay: the evaluator plays the artifact with input; Hidden eval.: unseen checks decide the score; Reopen: persistence is checked in a fresh process; Engine swap: a task is re-run on another engine.

Figure 2: Construction and composition of CEGame-Bench . (a) Tasks are specified as contracts, built, evaluated independently, and calibrated by play. (b) Coverage: engines, difficulty tiers, spatial forms, and genres.

Figure 3: GCB workflow: build the player path, play and inspect, then repair, replay, and restore if needed.

Configuration		Defold								Godot
Scaffold	Model	D1	D2	D3	D4	D5	D6	$\mathbf{F}$	$\mathbf{O}$	D1	D2	D3	D4	D5	D6	$\mathbf{F}$	$\mathbf{O}$
	GPT-6 Astra	10.0	20.3	11.3	14.8	6.2	9.5	62.7	72.2	9.4	15.8	8.2	12.1	4.4	5.1	49.8	54.8
	GPT-5.6 Sol	10.0	19.0	9.8	8.7	5.0	5.8	52.5	58.3	9.6	3.8	1.9	4.9	1.3	3.1	21.5	24.6
	GPT-5.6 Terra	7.2	7.4	8.0	8.0	2.6	2.7	33.2	35.9	9.2	1.6	1.5	3.1	1.1	3.4	16.4	19.8
Codex CLI	DeepSeek-V4-Pro	9.5	7.2	3.7	3.7	1.8	1.8	25.8	27.6	8.1	10.2	5.4	7.2	3.3	2.7	34.2	36.9
DeepSeek Harness	DeepSeek-V4-Pro	10.0	13.2	8.0	7.8	2.6	3.7	41.6	45.3	9.8	9.6	4.6	8.1	3.3	3.6	35.4	39.0

Table 2: Scores of seven scaffold–model configurations on each engine track and across all five engines. D1–D6 are integration, completeness, mechanics, lifecycle, feedback, and presentation, with D6 capped when the player journey fails;

F

sums D1–D5 (out of 90) and

O

adds D6 (out of 100). Bold marks the best

F

and

O

per panel; entries are rounded independently and may differ from their sums by 0.1.

			Script-centric engine ( $S$ )
	Engine	Metric	Defold	Godot	Roblox
Heavyweight ( $H$ )	Unity	$\bar{F}_{H}$	$40.6_{\scriptscriptstyle\pm\,8.5}$	$38.0_{\scriptscriptstyle\pm\,8.9}$	$35.0_{\scriptscriptstyle\pm\,12.0}$
		$\bar{F}_{S}$	$29.0_{\scriptscriptstyle\pm\,11.1}$	$27.6_{\scriptscriptstyle\pm\,6.4}$	$39.3_{\scriptscriptstyle\pm\,9.7}$
		$\Delta F$	$\mathbf{+11.6}_{\scriptscriptstyle\pm\,4.3}$	$\mathbf{+10.4}_{\scriptscriptstyle\pm\,3.1}$	$\mathbf{-4.3}_{\scriptscriptstyle\pm\,5.4}$
	Unreal Engine	$\bar{F}_{H}$	$49.0_{\scriptscriptstyle\pm\,10.6}$	$45.0_{\scriptscriptstyle\pm\,12.1}$	$34.0_{\scriptscriptstyle\pm\,10.4}$
		$\bar{F}_{S}$	$27.8_{\scriptscriptstyle\pm\,14.3}$	$23.5_{\scriptscriptstyle\pm\,13.2}$	$25.5_{\scriptscriptstyle\pm\,16.2}$

Table 3: The same games on two engines: mean

F

on the heavyweight (

\bar{F}_{H}

, rows) and script-centric (

\bar{F}_{S}

, columns) engine and

\Delta F=\bar{F}_{H}-\bar{F}_{S}

, with standard deviations over games as subscripts. Each pair pools its ten swapped games, five native to each engine.

Figure 4: Construction cost against

F

for a uniform sample of game–configuration pairs: mean workflow steps (a, c) and mean tokens on a log scale (b, d), colored by engine (a, b) or model (c, d; DeepSeek-V4-Pro pools its two scaffolds).

Configuration	$\bm{F}$	$\bm{\Delta F}$	$\bm{O}$	$\bm{\Delta O}$
DeepSeek-V4-Pro	50.6	—	54.8	—
DeepSeek-V4-Pro + GCB	71.6	+21.0	76.3	+21.5
GPT-5.6 Sol	27.0	—	30.9	—
GPT-5.6 Sol + GCB	41.1	+14.1	45.6	+14.7
GPT-5.6 Terra	9.7	—	10.5	—
GPT-5.6 Terra + GCB	34.0	+24.3	36.3	+25.8

Table 4: GCB under Codex CLI on the 15 games, three per engine track, ordered by baseline

O

\Delta F

and

\Delta O

are gains over the paired baseline.

Engine	Engine build	Language	Scene representation	Checks before play	Evaluator adapter
Script-centric engines
Defold	1.13.0	Lua	Data-driven game objects and collections.	Resource and reference validation; Lua syntax checks during build.	Engine Lua hook.
Godot	4.7.1.stable	GDScript	Node trees serialized as text scenes.	GDScript parsing and scene-loading diagnostics.	Headless script.
Roblox Studio	0.742.0.7421053	Luau	Studio instance trees stored in local .rbxl files.	Luau analysis and editor diagnostics.	Studio plugin and local play-test.
Heavyweight engines
Unity	6.0 LTS 6000.0.80f1	C#	YAML scenes and prefabs with an asset-import pipeline.	C# compilation, serialized-reference checks, and Unity tests.	PlayMode harness.

Table 5: Engine builds, project representations, pre-play diagnostics, and evaluator adapters for the five tracks.

#	GDD section	Information provided to the agent	Symbol	Dimensions
1	Pitch and core loop	Game premise, session structure, and target duration.	—	D2
2	Engine and toolchain	Engine build, dependencies, and build/run commands.	—	D1
3	Controls	Input devices, bindings, actions, and press/hold/release behavior.	$\mathcal{A}$	D3, D5
4	Entities and attributes	Required actors, props, UI elements, and numerical attributes.	$\mathcal{N}$	D2, D3
5	Scene inventory	Entry scene, levels, scene contents, and connections.	$\mathcal{S}_{a}$	D1, D2
6	Mechanics and rules	Action resolution, costs, resources, and collision behavior.	$\mathcal{S}_{r}$	D3

Table 6: The shared GDD structure and its mapping to contract symbols and evaluation dimensions.

GDD §	Requirement	Sym.	Dim.	$\bm{w}$	Route	Leaf assertion
5	The game opens on the title menu; Start enters the Harbor level.	$\mathcal{S}_{a}$	D1	2	Static, input	The entry scene is the title menu; after Start , level.id is harbor .
3	Press Shift to dash in the facing direction.	$\mathcal{A}$	D3	3	Input + read	After Shift, player.state is dash and player.pos moves along player.facing .
6	A dash spends one charge.	$\mathcal{S}_{r}$	D3	3	Input + read	player.charges decreases by exactly one per dash.
7	A new dash is rejected for 0.4 s after the last one.	$\mathcal{T}$	D3	2	Input + read	A second dash at 0.2 s is rejected; one at 0.5 s is accepted.
8	With no charges left, dash does nothing.	$\mathcal{B}$	D3	2	Input + read	At player.charges $=0$ , Shift leaves position and charges unchanged.
10	Charges never exceed the maximum.	$\mathcal{I}$	D3	2	Input + read	During a 10 s recharge, player.charges $\leq$ player.max_charges at every sample.

Table 7: Example requirement-to-check mapping. Sym., Dim., and

w

denote the contract symbol, score dimension, and leaf weight (1 minor, 2 major, 3 critical).

Stage	Evidence	Feeds
Static	import and compile, entry scene, serialized references, manifests	D1, D2
Local dynamic	real input, $\Phi$ snapshots, collision, AI, and resource assertions, negative controls	D2, D3, D5
Transition	scene change, resource transfer, death and restart, pause and resume, terminal freeze	D3, D4
Player journey	entry-to-terminal play with public controls, closed-loop input, step budget	$J$
Fresh lifecycle	dirty close, new process, clean entry, restored references	D4, $\Lambda$
Visual	reference-state captures, paired reference images, written rubric, artifact hashes	D6

Table 8: Evaluator stages, their evidence sources, and the dimensions or outcome flags they feed.

ID	$W_{d}$	Leaves	Name	Primary evidence
D1	10	5.1	Integration	import, compile, entry scene, serialized references, manifests
D2	30	14.0	Completeness	required entities, levels, modes, UI, authored content
D3	20	11.3	Mechanics	actions, rules, resources, collision, AI, negative branches
D4	20	8.8	Lifecycle	transitions, terminal state, restart, save, fresh reopen
D5	10	4.3	Feedback	controls, HUD, observable response to every player action
D6	10	3.7	Presentation	asset assembly, layout, layering, style, and finish against reference captures

Table 9: The six outcome dimensions, shared by all five tracks.

W_{d}

is the dimension weight and Leaves the mean number of leaf checks per task (47.2 in total). D1–D5 are scored by execution and D6 by a vision–language model.

State	Score effect	Meaning
pass	full weight	obligation observed under valid preconditions
fail_observed	zero, in denominator	submission violates the obligation under valid observation
product_blocked	zero, in denominator	submission itself prevents the required state or action
setup_failed	excluded	fixture preparation failed, also through the alternate route
observer_failed	excluded	evidence acquisition failed or disagreed across routes
inconclusive	excluded	evidence insufficient; never inferred from absence

Table 10: Leaf states and their effect on the score. The first three are judgments of the submission; the last three leave the denominator.

Engine	Sc.	Ent.	States	Steps	Leaves
Defold	2.1	11.8	7.4	31.0	40.8
Godot	2.3	13.1	8.1	35.0	45.1
Roblox Studio	2.6	14.9	8.6	41.0	48.0
Unity	2.8	15.6	8.9	41.0	49.5
Unreal Engine	3.2	16.4	9.3	45.0	52.6
All	2.6	14.4	8.5	38.6	47.2

Table 11: Mean task structure. Sc./Ent.: scenes/entities; States: state nodes; Steps: reference-play inputs; Leaves: leaf checks.

Family	Examples	Stressor
T1 UI and rule puzzle	Four Shot, Arrow Escape	authored content, event rules, input edges, reset
T2 Action	Driftfolk Chronicle, Bomb Frog	collision, timing, camera, feedback
T3 3D spatial action	Dungeon Duel, Noir Roombreak	navigation, combat state, multi-room transitions
T4 Systems	Relay Foundry, Ape Tower Defense	resources, AI, dependency graphs, persistence
T5 Narrative	There Are Signs	branch tables, authored presentation, visual assembly

Table 12: Task families and the stressor each family is designed to expose.

Stage	Required action
Specify	Map public requirements, prerequisites, and resources.
Build	Complete the required path; launch early.
Inspect	Link real input to state and image observations.
Repair	Archive a repair anchor; invalidate affected checks.
Replay	Recheck target and protected checks; retain or restore.
Finalize	Play the full journey; reopen when required.

Table 13: Execution stages of GCB.

Scaffold	Version	Model	ID	Eff.
		Astra	gpt-6-astra	max
		Sol	gpt-5.6-sol	max
		Terra	gpt-5.6-terra	max
Codex CLI	0.159.2	V4-Pro	deepseek-v4-pro	max
DS Harness	0.2.0-rc.2	V4-Pro	deepseek-v4-pro	max
Claude Code	2.1.285	Opus 5	claude-opus-5	max

Table 14: Scaffold versions, model identifiers, and reasoning-effort settings. DS Harness is DeepSeek Harness.

Engine	D1 (%)	Play (%)	Gap (pp)	Mean $\bm{F}$	Low (%)	Spread
Defold	88.6	37.2	51.4	38.6	71.4	56.9
Godot	88.7	23.4	65.3	27.6	85.7	35.3
Roblox Studio	91.3	35.7	55.6	37.7	85.7	53.6
Unity	97.0	56.9	40.1	55.2	42.9	29.0
Unreal Engine	94.6	30.4	64.1	33.7	85.7	51.3

Table 15: Summary derived from Table 2 . D1 and Play normalize integration and gameplay scores; Gap is their difference. Low is the share of configurations below 50% gameplay, and Spread is the range of

F

	Godot	Roblox	Unity	Unreal
Defold	0.52	0.52	0.62	0.43
Godot		0.62	0.33	0.33
Roblox Studio			0.52	0.14
Unity				0.43

Table 16: Kendall’s

\tau

between engine-specific configuration rankings, computed using

F

from Table 2 .

Table 21

Configuration		Workflow steps (%)
Scaffold	Model	Edit	Build	Play	View	Other
	GPT-6 Astra	26.3	17.4	7.7	5.6	43.0
	GPT-5.6 Sol	17.1	19.9	5.3	4.8	52.9
	GPT-5.6 Terra	31.3	22.4	5.0	4.2	37.1
Codex CLI	DeepSeek-V4-Pro	28.3	8.9	8.7	4.6	49.5
DeepSeek Harness	DeepSeek-V4-Pro	34.1	6.3	3.1	4.5	52.0

Table 19: Action shares in the construction traces of native runs (%). Other covers reading files and documents, planning, and remaining actions.

Panel of Figure 5 — Figure 5: Three delivered games from the bottom, middle, and top of the score range, as captured by the evaluator. (a) Captain2D Platformer on Unreal Engine ( $O=15.2$ ): the player character, enemies, and springs stand below the room. (b) RollBall on Unreal Engine ( $O=57.2$ ): the HUD reports a trap hit while the cropped view leaves the ball at the lower edge. (c) Relay Foundry on Roblox Studio ( $O=95.7$ ): the rules hold, but system buttons and the HUD crowd the world signs.

Automated Game Testing

See all 10 papers

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
Jun 16, 2026Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22AI Agent Benchmarks Agentic Code Generation
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Sep 19, 2026Shutong Wu, Kevin Calderone, Andy TsenCoding Agents AI Agent Evaluation
University of Rochester · RamenVR
Game-Creation-Agents: Autonomous Game Production
Oct 12, 2026Tencent Multimodal RL Team, Xiangxin Zhou, Jiaqi Tang +15AI Coding Agents Software Engineering Agents
Multimodal Reinforcement Learning Team, Tencent

Jun 16, 2026J/K move · Enter open · S save

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu+21

1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB+5

AI Agent Benchmarks Agentic Code Generation Code Generation Evaluation

Open paper AskPDF arXiv

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Abstract

Automated Game Testing

Explore similar work

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Game-Creation-Agents: Autonomous Game Production