Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?

Yixue Cai; Yuzhe Zhao; Hanxiang Chao; Qingsen Ma; Ziheng Xiong; Jinhu Qi; Irwin King

cs.AIOct 3, 2026

Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?

Authors: Yixue Cai, Yuzhe Zhao, Hanxiang Chao, Qingsen Ma, Ziheng Xiong, Jinhu Qi, Irwin King

Organizations: The Chinese University of Hong Kong Hong Kong, China · Nankai University Tianjin, China · Wuhan University Wuhan, China

Abstract

Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language. To evaluate large language models' ability to realize detailed specifications as complete interactive programs, we introduce Spec2Game, a benchmark that requires models to generate complete Pygame projects from detailed natural-language game specifications. Spec2Game comprises 15 game families and 150 task instances, with one canonical task and nine controlled rule variants per family, spanning three levels of implementation complexity. Using source-code, runtime, and visual evidence, we evaluate generated projects along four dimensions---Executability, Specification Realization, Code Quality, and User-Facing Quality. Across 14 LLMs and 3,330 generated projects, we find that high executability does not imply faithful specification realization. Component-level analysis further shows that models perform substantially better on Game Element Modeling than on Rule and Mechanism Modeling or Goal and Termination Modeling, indicating that faithfully implementing game rules and termination logic remains a major challenge.

Figures & tables

Figure 1. Overview of Spec2Game. (a) The benchmark comprises 15 game families, each with one canonical task and nine controlled variants, forming Core-15 (15 canonical tasks) and Full-150 (150 task instances). (b) A fixed staged generation protocol produces complete Pygame projects, followed by isolated execution and evaluation evidence collection. (c) Generated projects are evaluated along four complementary dimensions (D1–D4). Code snippets, traces, and screenshots are schematic.

Game Type	Easy	Medium	Hard
Real-Time Reaction	Snake Dodge Blocks Whack-a-Mole	–	–
Shooting and Combat Action	Asteroids	Space Invaders	Top-down Arena Shooter
Spatial Exploration and Platform Adventure	Maze Runner	Simple Platformer	Mini Zelda-like Room Adventure
Logic and Puzzle	–	Tetris Match-3	Pipe Connection Puzzle
Strategy and Resource Management	–	Simple Shop Management	Carrot Defense-like Farming-lite

Table 1. The 15 game families by type and implementation complexity.

Dimension	Components
D1 Executability	Syntax Validity; Dependency Readiness; Window Creation; Event Handling; Short-Horizon Runtime Stability; Controlled Termination
D2 Specification Realization	Game Element Modeling; Interaction Modeling; Rule and Mechanism Modeling; Dynamic Evolution; Goal and Termination Modeling
D3 Code Quality	Maintainability; Reliability; Security; Efficiency; Conformance
D4 User-Facing Quality	Visual Presentation; Dynamic Feedback Fidelity; Visual Actionability

Table 2. Evaluation dimensions and components in Spec2Game.

Candidate Model	Access Route	Full-150	Core-15
GPT-5.6 Sol	OpenRouter	✓	✓
Grok 4.3	OpenRouter	✓	✓
GLM-5.2	Zhipu official	✓	✓
Kimi K2.6	Moonshot official	✓	✓
Qwen3.8 Max-0902	Alibaba official	✓	✓
DeepSeek V4 Pro-0813	DeepSeek official	✓	✓

Table 3. Candidate models, access routes, and evaluation tracks.

Figure 2. Core-15 comparison of Overall performance and dimension-wise scores. Overall scores in panel (a) are computed first and then aggregated; dimension scores in panel (b) are aggregated separately.

Model	M	D	B	$D-M$	$B-D$
GPT-5.6 Terra	55.18	74.28	75.40	$+19.10$	$+1.11$
Qwen3.7 Plus	45.79	64.22	59.84	$+18.43$	$-4.38$
Grok 4.3	50.79	67.35	70.02	$+16.56$	$+2.66$
Overall	50.59	68.62	68.42	$+18.03$	$-0.20$

Table 4. Mean Core-15 overall scores under three specification-input conditions.

Judge condition	Qwen wins	Terra wins	Ties	$p$ -value	Qwen win rate
Qwen Judge	13	14	0	1.000	48.1%
GPT-5.6 Sol Judge	15	12	0	0.701	55.6%

Table 5. Pairwise rankings of Qwen- and Terra-generated candidates under two text judges.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Sub-dimensions	Captures
Operation Complexity	A1–A2	Input actions, target localization, and operation flow
Object and Rule Complexity	B1–B6	State representation, rule updates, constraint checking, object interactions, progression, and termination conditions
Spatial and Scene Complexity	C1–C2	Spatial structure and dependencies involving layout and reachability
Adversarial Behavior Complexity	D1	Independent adversarial entities, their behavior patterns, and state dependence
Interface and Information Complexity	E1–E2	Organization of interface information and visual state representation

Appendix

Table 6. Overview of the implementation complexity dimensions and sub-dimensions in Spec2Game.

Game	A1	A2	B1	B2	B3	B4	B5	B6	C1	C2	D1	E1	E2	Total
Snake	1	1	2	2	2	2	1	1	2	1	1	1	1	18
Dodge Blocks	1	1	2	1	1	1	1	1	1	1	1	1	1	14
Whack-a-Mole	1	2	2	2	1	1	1	1	2	1	1	1	1	17
Asteroids	2	1	2	2	1	2	1	2	1	1	1	1	1	18
Maze Runner	1	1	2	2	1	1	1	2	2	2	1	1	1	18
Space Invaders	2	1	2	2	1	2	1	2	1	1	2	1	1	19

Appendix

Table 7. Implementation complexity scores for the 15 game families.

Stage	Task-Specific Inputs	Prior Context
state.py	Game Overview; Game Objects & States; Game Flow & Progression. D2.1 checkpoint-derived requirements related to state modeling.	None
game.py	Game Overview; Game Objects & States; Player Interaction; Core Rules & Mechanics; Game Flow & Progression; Goal, Feedback & Termination. D2.2–D2.5 checkpoint-derived requirements related to interaction, rules, dynamic evolution, and goal/termination behavior.	Complete locked state.py
rendering.py	Game Overview; Game Scene & Interface; Goal, Feedback & Termination. Fixed task-independent visual-quality requirements. No checkpoint-derived functional requirements.	Complete locked state.py
main.py	Player Interaction. No checkpoint-derived requirements.	Public API summaries of state.py , game.py , and rendering.py

Appendix

Table 8. Stage-specific task information and prior context used in Spec2Game.

Version	Changes relative to V1
1. Snake
V2	Increase initial snake length from 3 to 5 and change its starting position.
V3	Increase movement speed from 8 to 10 steps/s.
V4	Add fixed obstacles; obstacle collisions end the game, and food cannot spawn on obstacles.
V5	Maintain two food items simultaneously; replace each item individually when consumed.
V6	Increase snake length by 2 per food item while retaining a score increment of 1.

Appendix

Table 9. Rule changes across the 135 Full-150 variants.

Game	Complexity	Included in Final Benchmark
Snake	Easy	Yes
Flappy Bird	Easy	No
2048	Medium	No
Space Invaders	Medium	Yes
Carrot Defense-like	Hard	Yes
Lode Runner-like	Hard	No

Appendix

Table 10. Pilot tasks and their inclusion in the final benchmark.

Check	Pass condition	Recorded evidence
Syntax Validity	All four generated files, state.py , game.py , rendering.py , and main.py , are present and can be successfully parsed by ast.parse .	Project-file validation results and per-file syntax errors.
Dependency Readiness	Syntax Validity has passed; the source code imports Pygame; the evaluation environment can successfully execute import pygame in an isolated subprocess; and the generated files pass the frozen dependency-direction validation.	Pygame availability, import results, dependency-direction validation results, and related project-validation information.
Window Creation	Dependency Readiness has passed; during the runtime stability probe, execution of the entry module triggers pygame.display.set_mode(...) , and the returned object is a pygame.Surface .	Probe return code, stdout/stderr, and window-creation and surface-return status.
Event Handling	Dependency Readiness has passed; either pygame.event.get() or pygame.event.poll() is observed at runtime, or one of these event-fetching calls is detected in the AST.	Runtime event-fetch evidence and AST inspection results.
Short-Horizon Runtime Stability	The entry module does not terminate abnormally during the 5-second runtime stability probe and remains running until the evaluator reaches the predefined probe timeout.	Return code, stdout/stderr, timeout status, and runtime status.
Controlled Termination	Short-Horizon Runtime Stability has passed; the termination probe begins repeatedly injecting pygame.QUIT events 1 second after program launch, and at least one QUIT event is successfully posted. The candidate program must then terminate on its own with return code 0 within the probe’s 5-second timeout.	Evidence of successful QUIT posting, return code, stdout/stderr, and timeout status.

Appendix

Table 11. Operational definitions for D1 Executability.

Penalty	Operational anchor
0	No substantive extra mechanism or mechanism incompatible with the specification is present; implementation details, decorative presentation, and operational menus are not treated as deviations.
1	A single localized and minor extra behavior is present, with limited impact on the requested core system; removing it would not change the main gameplay or task identity.
2	A complete extra system or a clear mechanism-level deviation is present; it materially changes a rule, interaction, dynamic evolution process, or termination logic, but does not redefine the task as a whole.
3	Multiple extra systems are present, or a single major deviation is sufficient to change the task identity, main gameplay, or objective.

Appendix

Table 12. Severity anchors for specification-deviation penalties in D2.

Component	Deterministic	judge	Total
Maintainability	20	10	30
Reliability	15	10	25
Security	12	3	15
Efficiency	10	5	15
Conformance	13	2	15
Total	70	30	100

Appendix

Table 13. Point allocation for D3 Code Quality.

Component	Raw max	Scoring rule
Maintainability	30	This component consists of four parts. Complexity (12 points) uses Radon complexity ranks, mapping A/B/C/D/E/F to $1/.8/.5/.25/0/0$ , and aggregates the resulting factors as $12\times(0.1\,\mathrm{mean}+0.9\,\mathrm{min})$ ; Structural lint (8 points) starts from 8 and deducts 1.5 points for each C901 or PLR0904/0911/0912/0913/0914/0915/0916/0917 finding; Duplicate structure (5 points) deducts 1 point for each additional normalized AST duplicate window consisting of six statements; Testability (5 points) awards 2 points for a main guard, 2 points for a successful clean import, and 1 point when there are at most three executable top-level statements.
Reliability	20	This component starts from 20 points. Each F821/F822/F823/B012 finding deducts 5 points; each B006/B008/B023/BLE001/E722/F632/F811/TRY201/TRY203 finding deducts 2 points; and each other matching B* , BLE* , PLE* , PLW* , or TRY* finding deducts 1 point. AST analysis additionally deducts 1 point for each bare or broad exception handler; if its sole body is pass or continue , a further 2 points are deducted. The cumulative AST deduction is capped at 4 points.
Security	15	This component starts from 15 points. For each Bandit finding, the deduction is the product of a severity weight and a confidence weight, where High/Medium/Low severity correspond to $6/3/1$ and High/Medium/Low confidence correspond to $1/.75/.5$ . If at least one forbidden call is detected, an aggregate deduction of 15 points is applied; if at least one external import other than Pygame or a Python standard-library package is detected, another aggregate deduction of 15 points is applied.
Efficiency	10	This component starts from 10 points. Each Ruff PERF* finding deducts 1 point, with this deduction capped at 4 points. Each hotspot call inside a for / while loop deducts 2 points, with this deduction capped at 6 points. Hotspots include open , Pygame font creation, image/sound loading, pygame.Surface , rotation, and scaling.
Conformance	10	The score is based on a weighted finding density computed from Ruff findings in E* , W* , I* , N* , UP* , SIM* , F401 , and F841 . Findings in E1--E3/W1--W3 receive weight $0.1$ ; E501/I/UP/SIM* receive weight $0.5$ ; and all other included findings receive weight $1$ . Let $\rho=100\times\text{weighted findings}/\max(1,\text{logical LOC})$ . The raw score is then $10\max(0,1-\rho/10)$ .

Appendix

Table 14. Deterministic scoring rules for D3. All deduction-based scores are floored at zero.

Component	Assessment scope
Maintainability	Responsibility boundaries, cohesion/coupling, change locality, abstraction discipline, and state architecture.
Reliability	State consistency, initialization/reset/termination, update ordering, edge cases, mutation safety, and defect risk.
Security	Indirect dangerous behavior, unnecessary external access, and compliance with offline Pygame task constraints.
Efficiency	Repeated work, algorithmic hotspots, collection growth, resource lifetime, caching, and per-frame work.
Conformance	Semantic naming, readable organization, useful comments, magic values, and main control flow.

Appendix

Table 15. Assessment scope of the semantic judge in D3.

Score	Operational anchor
0	The visual presentation is poor or cannot be meaningfully evaluated, for example because the frame is blank, broken, clearly unfinished, or contains severe problems in composition, hierarchy, or style.
1	The visual content is recognizable and broadly coherent, but shows noticeable weaknesses in composition, color, shape and style consistency, polish, or dynamic presentation.
2	The visual presentation shows clear design intent, is coherent and well-finished overall, and contains no substantive defects in composition, palette, shape and style consistency, or dynamic presentation.

Appendix

Table 16. Shared scoring anchors for the five Visual Presentation criteria.

Score	Operational anchor
0	No evaluable supporting evidence is available, for example because the frame is blank or broken, the feedback is missing, the observed behavior contradicts the requirement, or the feedback is unintelligible.
1	Only minimal traces of the target feedback are present, while severe defects dominate.
2	Recognizable supporting evidence is present, but major omissions, ambiguity, inconsistency, or clear timing problems remain.
3	The main requirement is satisfied, but several noticeable weaknesses remain.
4	The feedback is overall complete and convincing, with only minor defects.
5	The feedback is clear, complete, timely, and well designed, with no substantive defects.

Appendix

Table 17. Scoring anchors for Dynamic Feedback Fidelity.

Model	D1	D2	D3	D4	Overall
Claude Sonnet 5	96.30	69.95	84.42	60.01	77.27
Claude Haiku 4.5	97.04	69.74	81.42	56.10	74.91
DeepSeek V4 Pro-0813	96.30	67.90	82.22	55.50	73.60
Grok 4.3	95.56	66.81	76.45	55.10	73.37
Qwen3.8 Max-0902	73.33	72.85	81.30	57.73	71.95
GPT-5.6 Luna	89.63	64.53	86.39	54.04	71.35

Appendix

Table 18. Family-balanced D1–D4 and Overall scores of the 14 candidate models on Core-15.

Figure 3. Executability diagnostics on Core-15. Left: observed pass rates of the six D1 checks. Right: the proportion of candidates passing all six D1 checks and the mean D2 score for each model.

Figure 4. Component-level average scores for D2 and D4 on Core-15.

Figure 5. Component-level average scores for D2 and D4 on Full-150.

Figure 6. Performance on Core-15 by assigned implementation complexity. The left panel shows family-level Overall scores for the five game families within each complexity level together with the corresponding level mean; the right panel shows family-level variation across all 15 game families.

Figure 7. Performance on Full-150 by assigned implementation complexity. The left panel shows family-level Overall scores for the five game families within each complexity level together with the corresponding level mean; the right panel shows family-level variation across all 15 game families.

Figure 8. Overall scores of the six Full-150 models on canonical tasks, controlled variants, and all 150 task instances.

Figure 9. Family-level Overall score changes of controlled variants relative to canonical tasks in Full-150. Each cell reports the difference between the mean Overall score of the nine variants and the Overall score of the corresponding canonical task for a given model–family pair.

Figure 10. Overall score variability across three independent generations on Core-15. The left panel compares, for each model, the standard deviation of the three generation-index means with the mean within-task standard deviation across the 15 canonical tasks. The right panel uses GPT-5.6 Sol as an example and shows the Overall scores of its three generations for each canonical task.

File	Specification and prior context	Additional inputs in B
state.py	Overview; objects and states; flow and progression.	D2.1 state-modeling requirements.
game.py	Overview; objects; interaction; rules; progression; goals, feedback, and termination; locked state.py .	D2.2–D2.5 behavioral requirements.
rendering.py	Overview; scene and interface; goals, feedback, and termination; locked state.py .	None.
main.py	Player interaction; public API summaries of preceding modules.	None.

Appendix

Table 19. Stage-specific inputs for D and B. B adds the public requirements in the final column to the corresponding D inputs.

	GPT-5.6 Terra			Qwen3.7 Plus			Grok 4.3
Game	M	D	B	M	D	B	M	D	B
Dodge Blocks	44.12	82.30	79.24	53.57	78.77	76.49	59.21	75.00	74.60
Farming-lite	42.78	72.17	71.16	22.46	59.80	49.53	40.78	66.51	55.43
Carrot Defense-like	60.89	68.35	73.58	48.84	49.15	61.87	43.78	45.42	58.93
Snake	59.77	71.20	66.67	59.28	67.86	73.55	60.47	70.81	73.77
Tetris	67.07	79.15	74.51	37.51	72.27	75.11	72.93	75.31	72.97

Appendix

Table 20. Complete Core-15 ablation results (0–100) from the final evaluation rerun. The table contains 135 model–game–condition scores. Overall scores use the continuous D1 gate in Equation 13 .

	GPT-5.6 Terra		Qwen3.7 Plus		Grok 4.3
Game	$D-M$	$B-D$	$D-M$	$B-D$	$D-M$	$B-D$
Dodge Blocks	$+38.18$	$-3.06$	$+25.20$	$-2.28$	$+15.78$	$-0.39$
Farming-lite	$+29.39$	$-1.00$	$+37.34$	$-10.26$	$+25.72$	$-11.07$
Carrot Defense-like	$+7.46$	$+5.23$	$+0.31$	$+12.72$	$+1.64$	$+13.51$
Snake	$+11.43$	$-4.52$	$+8.58$	$+5.69$	$+10.34$	$+2.96$
Tetris	$+12.08$	$-4.63$	$+34.76$	$+2.85$	$+2.38$	$-2.34$

Appendix

Table 21. Paired differences in Overall score for the same model and game. Differences are computed before rounding.

Model	M	D	B	$D-M$	$B-D$
GPT-5.6 Terra	53.10	75.29	75.55	$+22.19$	$+0.27$
Qwen3.7 Plus	45.94	65.45	68.05	$+19.51$	$+2.60$
Grok 4.3	51.75	68.41	70.34	$+16.65$	$+1.93$

Appendix

Table 22. Mean Overall scores on the common 11-game subset, excluding Carrot Defense-like, Pipe Connection Puzzle, Match-3, and Mini Zelda-like Room Adventure consistently across models and conditions. Scores use the same continuous D1 gate as the full Core-15 results.

#	Game	Experiment ID
1	Snake	snake
2	Dodge Blocks	dodge_blocks
3	Asteroids	asteroids
4	Tetris	tetris
5	Simple Platformer	simple_platformer
6	Match-3	match3

Appendix

Table 23. Nine games included in Ablation Study II.

Configuration	Qwen text judge condition	GPT-5.6 Sol text judge condition
Text judge	Qwen3-235B-A22B-2507	GPT-5.6 Sol
Model ID	qwen.qwen3-235b-a22b-2507	openai/gpt-5.6-sol
Access route	AWS Bedrock Mantle	OpenRouter
Scored components	D2 functionality, D2 specification deviation, D3 code quality	D2 functionality, D2 specification deviation, D3 code quality
Temperature	0.0	0.0
Reasoning	disabled ( enable_thinking=false )	disabled ( reasoning.effort=none )

Appendix

Table 24. Configurations of the two text judge conditions in Ablation Study II.

Judge condition	Qwen wins	Terra wins	Ties	Binomial $p$	Qwen win rate
Qwen3-235B-A22B-2507	13	14	0	1.000	48.1%
GPT-5.6 Sol	15	12	0	0.701	55.6%

Appendix

Table 25. Paired ranking results under the two text judge conditions.

Comparison	$N$	$\rho$	$\tau$	$r$	MAE
D2.1 Element modeling	438	0.049	0.048	0.266	0.12
D2.2 Interaction modeling	384	0.466	0.418	0.596	0.37
D2.3 Rules and mechanisms	314	0.551	0.457	0.539	0.50
D2.4 Dynamic evolution	391	0.521	0.481	0.697	0.25
D2.5 Goal and Termination Modeling	277	0.805	0.699	0.799	0.33
D2 overall	243	0.779	0.655	0.791	0.13

Appendix

Table 26. Candidate-level questionnaire–LLM alignment on primary assignments.

N

denotes paired candidates for each comparison. D2 errors use a 0–2 scale, and D4 errors a 0–100 scale. D4.3 compares questionnaire accuracy on one question with the official two-question automatic score. Overall comparisons use the reported candidate-level aggregates and are not averages of the correlations above.

Assessment category	Pairable / total	Available $\alpha$	Complete $\alpha$
D2 functional checkpoints	60/75	0.687	0.352
D4.1 overall visual quality	10/15	0.049	0.049
D4.2 dynamic feedback	20/30	0.753	0.349
D4.3 action choice	9/15	0.544	0.544
Question validity (auxiliary)	9/15	0.049	0.049

Appendix

Table 27. Descriptive agreement among shared-set responses. Counts refer to assessment items rather than distinct projects. Complete-case counts are 37, 10, 7, 9, and 9, respectively. These estimates do not measure questionnaire–LLM alignment and do not establish annotation independence.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
Jun 16, 2026Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22AI Agent Benchmarks Agentic Code Generation
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection
May 8, 2026Chaobo Jia, Ruipeng Wan, Ting Sun +5Automated Software Testing Formal Verification
1CUHK · 2HUST · 3Lionrock AI Lab +3
Distilling Game Code World Model Generation into Lightweight Large Language Models
May 23, 2026Tyrone Serapio, Arjun Prakash, Haoyang Xu +2LLM-Based Program Synthesis
Brown University Providence, United States

Jun 16, 2026J/K move · Enter open · S save

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu+21

1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB+5

AI Agent Benchmarks Agentic Code Generation Code Generation Evaluation

Open paper AskPDF arXiv

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Abstract

Code Generation

Explore similar work

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection

Distilling Game Code World Model Generation into Lightweight Large Language Models