Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language. To evaluate large language models' ability to realize detailed specifications as complete interactive programs, we introduce Spec2Game, a benchmark that requires models to generate complete Pygame projects from detailed natural-language game specifications. Spec2Game comprises 15 game families and 150 task instances, with one canonical task and nine controlled rule variants per family, spanning three levels of implementation complexity. Using source-code, runtime, and visual evidence, we evaluate generated projects along four dimensions---Executability, Specification Realization, Code Quality, and User-Facing Quality. Across 14 LLMs and 3,330 generated projects, we find that high executability does not imply faithful specification realization. Component-level analysis further shows that models perform substantially better on Game Element Modeling than on Rule and Mechanism Modeling or Goal and Termination Modeling, indicating that faithfully implementing game rules and termination logic remains a major challenge.
Figures & tables
Figure 1. Overview of Spec2Game. (a) The benchmark comprises 15 game families, each with one canonical task and nine controlled variants, forming Core-15 (15 canonical tasks) and Full-150 (150 task instances). (b) A fixed staged generation protocol produces complete Pygame projects, followed by isolated execution and evaluation evidence collection. (c) Generated projects are evaluated along four complementary dimensions (D1–D4). Code snippets, traces, and screenshots are schematic.
Game Type
Easy
Medium
Hard
Real-Time Reaction
Snake Dodge Blocks Whack-a-Mole
–
–
Shooting and Combat Action
Asteroids
Space Invaders
Top-down Arena Shooter
Spatial Exploration and Platform Adventure
Maze Runner
Simple Platformer
Mini Zelda-like Room Adventure
Logic and Puzzle
–
Tetris Match-3
Pipe Connection Puzzle
Strategy and Resource Management
–
Simple Shop Management
Carrot Defense-like Farming-lite
Table 1. The 15 game families by type and implementation complexity.
Table 2. Evaluation dimensions and components in Spec2Game.
Candidate Model
Access Route
Full-150
Core-15
GPT-5.6 Sol
OpenRouter
✓
✓
Grok 4.3
OpenRouter
✓
✓
GLM-5.2
Zhipu official
✓
✓
Kimi K2.6
Moonshot official
✓
✓
Qwen3.8 Max-0902
Alibaba official
✓
✓
DeepSeek V4 Pro-0813
DeepSeek official
✓
✓
Table 3. Candidate models, access routes, and evaluation tracks.
Figure 2. Core-15 comparison of Overall performance and dimension-wise scores. Overall scores in panel (a) are computed first and then aggregated; dimension scores in panel (b) are aggregated separately.
Model
M
D
B
D−M
B−D
GPT-5.6 Terra
55.18
74.28
75.40
+19.10
+1.11
Qwen3.7 Plus
45.79
64.22
59.84
+18.43
−4.38
Grok 4.3
50.79
67.35
70.02
+16.56
+2.66
Overall
50.59
68.62
68.42
+18.03
−0.20
Table 4. Mean Core-15 overall scores under three specification-input conditions.
Judge condition
Qwen wins
Terra wins
Ties
p -value
Qwen win rate
Qwen Judge
13
14
0
1.000
48.1%
GPT-5.6 Sol Judge
15
12
0
0.701
55.6%
Table 5. Pairwise rankings of Qwen- and Terra-generated candidates under two text judges.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Sub-dimensions
Captures
Operation Complexity
A1–A2
Input actions, target localization, and operation flow
Object and Rule Complexity
B1–B6
State representation, rule updates, constraint checking, object interactions, progression, and termination conditions
Spatial and Scene Complexity
C1–C2
Spatial structure and dependencies involving layout and reachability
Adversarial Behavior Complexity
D1
Independent adversarial entities, their behavior patterns, and state dependence
Interface and Information Complexity
E1–E2
Organization of interface information and visual state representation
Appendix
Table 6. Overview of the implementation complexity dimensions and sub-dimensions in Spec2Game.
Game
A1
A2
B1
B2
B3
B4
B5
B6
C1
C2
D1
E1
E2
Total
Snake
1
1
2
2
2
2
1
1
2
1
1
1
1
18
Dodge Blocks
1
1
2
1
1
1
1
1
1
1
1
1
1
14
Whack-a-Mole
1
2
2
2
1
1
1
1
2
1
1
1
1
17
Asteroids
2
1
2
2
1
2
1
2
1
1
1
1
1
18
Maze Runner
1
1
2
2
1
1
1
2
2
2
1
1
1
18
Space Invaders
2
1
2
2
1
2
1
2
1
1
2
1
1
19
Appendix
Table 7. Implementation complexity scores for the 15 game families.
Stage
Task-Specific Inputs
Prior Context
state.py
Game Overview; Game Objects & States; Game Flow & Progression. D2.1 checkpoint-derived requirements related to state modeling.
None
game.py
Game Overview; Game Objects & States; Player Interaction; Core Rules & Mechanics; Game Flow & Progression; Goal, Feedback & Termination. D2.2–D2.5 checkpoint-derived requirements related to interaction, rules, dynamic evolution, and goal/termination behavior.
Complete locked state.py
rendering.py
Game Overview; Game Scene & Interface; Goal, Feedback & Termination. Fixed task-independent visual-quality requirements. No checkpoint-derived functional requirements.
Complete locked state.py
main.py
Player Interaction. No checkpoint-derived requirements.
Public API summaries of state.py , game.py , and rendering.py
Appendix
Table 8. Stage-specific task information and prior context used in Spec2Game.
Version
Changes relative to V1
1. Snake
V2
Increase initial snake length from 3 to 5 and change its starting position.
V3
Increase movement speed from 8 to 10 steps/s.
V4
Add fixed obstacles; obstacle collisions end the game, and food cannot spawn on obstacles.
V5
Maintain two food items simultaneously; replace each item individually when consumed.
V6
Increase snake length by 2 per food item while retaining a score increment of 1.
Appendix
Table 9. Rule changes across the 135 Full-150 variants.
Game
Complexity
Included in Final Benchmark
Snake
Easy
Yes
Flappy Bird
Easy
No
2048
Medium
No
Space Invaders
Medium
Yes
Carrot Defense-like
Hard
Yes
Lode Runner-like
Hard
No
Appendix
Table 10. Pilot tasks and their inclusion in the final benchmark.
Check
Pass condition
Recorded evidence
Syntax Validity
All four generated files, state.py , game.py , rendering.py , and main.py , are present and can be successfully parsed by ast.parse .
Project-file validation results and per-file syntax errors.
Dependency Readiness
Syntax Validity has passed; the source code imports Pygame; the evaluation environment can successfully execute import pygame in an isolated subprocess; and the generated files pass the frozen dependency-direction validation.
Pygame availability, import results, dependency-direction validation results, and related project-validation information.
Window Creation
Dependency Readiness has passed; during the runtime stability probe, execution of the entry module triggers pygame.display.set_mode(...) , and the returned object is a pygame.Surface .
Probe return code, stdout/stderr, and window-creation and surface-return status.
Event Handling
Dependency Readiness has passed; either pygame.event.get() or pygame.event.poll() is observed at runtime, or one of these event-fetching calls is detected in the AST.
Runtime event-fetch evidence and AST inspection results.
Short-Horizon Runtime Stability
The entry module does not terminate abnormally during the 5-second runtime stability probe and remains running until the evaluator reaches the predefined probe timeout.
Return code, stdout/stderr, timeout status, and runtime status.
Controlled Termination
Short-Horizon Runtime Stability has passed; the termination probe begins repeatedly injecting pygame.QUIT events 1 second after program launch, and at least one QUIT event is successfully posted. The candidate program must then terminate on its own with return code 0 within the probe’s 5-second timeout.
Evidence of successful QUIT posting, return code, stdout/stderr, and timeout status.
Appendix
Table 11. Operational definitions for D1 Executability.
Penalty
Operational anchor
0
No substantive extra mechanism or mechanism incompatible with the specification is present; implementation details, decorative presentation, and operational menus are not treated as deviations.
1
A single localized and minor extra behavior is present, with limited impact on the requested core system; removing it would not change the main gameplay or task identity.
2
A complete extra system or a clear mechanism-level deviation is present; it materially changes a rule, interaction, dynamic evolution process, or termination logic, but does not redefine the task as a whole.
3
Multiple extra systems are present, or a single major deviation is sufficient to change the task identity, main gameplay, or objective.
Appendix
Table 12. Severity anchors for specification-deviation penalties in D2.
Component
Deterministic
judge
Total
Maintainability
20
10
30
Reliability
15
10
25
Security
12
3
15
Efficiency
10
5
15
Conformance
13
2
15
Total
70
30
100
Appendix
Table 13. Point allocation for D3 Code Quality.
Component
Raw max
Scoring rule
Maintainability
30
This component consists of four parts. Complexity (12 points) uses Radon complexity ranks, mapping A/B/C/D/E/F to 1/.8/.5/.25/0/0 , and aggregates the resulting factors as 12×(0.1mean+0.9min) ; Structural lint (8 points) starts from 8 and deducts 1.5 points for each C901 or PLR0904/0911/0912/0913/0914/0915/0916/0917 finding; Duplicate structure (5 points) deducts 1 point for each additional normalized AST duplicate window consisting of six statements; Testability (5 points) awards 2 points for a main guard, 2 points for a successful clean import, and 1 point when there are at most three executable top-level statements.
Reliability
20
This component starts from 20 points. Each F821/F822/F823/B012 finding deducts 5 points; each B006/B008/B023/BLE001/E722/F632/F811/TRY201/TRY203 finding deducts 2 points; and each other matching B* , BLE* , PLE* , PLW* , or TRY* finding deducts 1 point. AST analysis additionally deducts 1 point for each bare or broad exception handler; if its sole body is pass or continue , a further 2 points are deducted. The cumulative AST deduction is capped at 4 points.
Security
15
This component starts from 15 points. For each Bandit finding, the deduction is the product of a severity weight and a confidence weight, where High/Medium/Low severity correspond to 6/3/1 and High/Medium/Low confidence correspond to 1/.75/.5 . If at least one forbidden call is detected, an aggregate deduction of 15 points is applied; if at least one external import other than Pygame or a Python standard-library package is detected, another aggregate deduction of 15 points is applied.
Efficiency
10
This component starts from 10 points. Each Ruff PERF* finding deducts 1 point, with this deduction capped at 4 points. Each hotspot call inside a for / while loop deducts 2 points, with this deduction capped at 6 points. Hotspots include open , Pygame font creation, image/sound loading, pygame.Surface , rotation, and scaling.
Conformance
10
The score is based on a weighted finding density computed from Ruff findings in E* , W* , I* , N* , UP* , SIM* , F401 , and F841 . Findings in E1--E3/W1--W3 receive weight 0.1 ; E501/I*/UP*/SIM* receive weight 0.5 ; and all other included findings receive weight 1 . Let ρ=100×weighted findings/max(1,logical LOC) . The raw score is then 10max(0,1−ρ/10) .
Appendix
Table 14. Deterministic scoring rules for D3. All deduction-based scores are floored at zero.
Component
Assessment scope
Maintainability
Responsibility boundaries, cohesion/coupling, change locality, abstraction discipline, and state architecture.
Reliability
State consistency, initialization/reset/termination, update ordering, edge cases, mutation safety, and defect risk.
Security
Indirect dangerous behavior, unnecessary external access, and compliance with offline Pygame task constraints.
Semantic naming, readable organization, useful comments, magic values, and main control flow.
Appendix
Table 15. Assessment scope of the semantic judge in D3.
Score
Operational anchor
0
The visual presentation is poor or cannot be meaningfully evaluated, for example because the frame is blank, broken, clearly unfinished, or contains severe problems in composition, hierarchy, or style.
1
The visual content is recognizable and broadly coherent, but shows noticeable weaknesses in composition, color, shape and style consistency, polish, or dynamic presentation.
2
The visual presentation shows clear design intent, is coherent and well-finished overall, and contains no substantive defects in composition, palette, shape and style consistency, or dynamic presentation.
Appendix
Table 16. Shared scoring anchors for the five Visual Presentation criteria.
Score
Operational anchor
0
No evaluable supporting evidence is available, for example because the frame is blank or broken, the feedback is missing, the observed behavior contradicts the requirement, or the feedback is unintelligible.
1
Only minimal traces of the target feedback are present, while severe defects dominate.
2
Recognizable supporting evidence is present, but major omissions, ambiguity, inconsistency, or clear timing problems remain.
3
The main requirement is satisfied, but several noticeable weaknesses remain.
4
The feedback is overall complete and convincing, with only minor defects.
5
The feedback is clear, complete, timely, and well designed, with no substantive defects.
Appendix
Table 17. Scoring anchors for Dynamic Feedback Fidelity.
Model
D1
D2
D3
D4
Overall
Claude Sonnet 5
96.30
69.95
84.42
60.01
77.27
Claude Haiku 4.5
97.04
69.74
81.42
56.10
74.91
DeepSeek V4 Pro-0813
96.30
67.90
82.22
55.50
73.60
Grok 4.3
95.56
66.81
76.45
55.10
73.37
Qwen3.8 Max-0902
73.33
72.85
81.30
57.73
71.95
GPT-5.6 Luna
89.63
64.53
86.39
54.04
71.35
Appendix
Table 18. Family-balanced D1–D4 and Overall scores of the 14 candidate models on Core-15.
Figure 3. Executability diagnostics on Core-15. Left: observed pass rates of the six D1 checks. Right: the proportion of candidates passing all six D1 checks and the mean D2 score for each model.
Figure 4. Component-level average scores for D2 and D4 on Core-15.
Figure 5. Component-level average scores for D2 and D4 on Full-150.
Figure 6. Performance on Core-15 by assigned implementation complexity. The left panel shows family-level Overall scores for the five game families within each complexity level together with the corresponding level mean; the right panel shows family-level variation across all 15 game families.
Figure 7. Performance on Full-150 by assigned implementation complexity. The left panel shows family-level Overall scores for the five game families within each complexity level together with the corresponding level mean; the right panel shows family-level variation across all 15 game families.
Figure 8. Overall scores of the six Full-150 models on canonical tasks, controlled variants, and all 150 task instances.
Figure 9. Family-level Overall score changes of controlled variants relative to canonical tasks in Full-150. Each cell reports the difference between the mean Overall score of the nine variants and the Overall score of the corresponding canonical task for a given model–family pair.
Figure 10. Overall score variability across three independent generations on Core-15. The left panel compares, for each model, the standard deviation of the three generation-index means with the mean within-task standard deviation across the 15 canonical tasks. The right panel uses GPT-5.6 Sol as an example and shows the Overall scores of its three generations for each canonical task.
File
Specification and prior context
Additional inputs in B
state.py
Overview; objects and states; flow and progression.
Overview; scene and interface; goals, feedback, and termination; locked state.py .
None.
main.py
Player interaction; public API summaries of preceding modules.
None.
Appendix
Table 19. Stage-specific inputs for D and B. B adds the public requirements in the final column to the corresponding D inputs.
GPT-5.6 Terra
Qwen3.7 Plus
Grok 4.3
Game
M
D
B
M
D
B
M
D
B
Dodge Blocks
44.12
82.30
79.24
53.57
78.77
76.49
59.21
75.00
74.60
Farming-lite
42.78
72.17
71.16
22.46
59.80
49.53
40.78
66.51
55.43
Carrot Defense-like
60.89
68.35
73.58
48.84
49.15
61.87
43.78
45.42
58.93
Snake
59.77
71.20
66.67
59.28
67.86
73.55
60.47
70.81
73.77
Tetris
67.07
79.15
74.51
37.51
72.27
75.11
72.93
75.31
72.97
Appendix
Table 20. Complete Core-15 ablation results (0–100) from the final evaluation rerun. The table contains 135 model–game–condition scores. Overall scores use the continuous D1 gate in Equation 13 .
GPT-5.6 Terra
Qwen3.7 Plus
Grok 4.3
Game
D−M
B−D
D−M
B−D
D−M
B−D
Dodge Blocks
+38.18
−3.06
+25.20
−2.28
+15.78
−0.39
Farming-lite
+29.39
−1.00
+37.34
−10.26
+25.72
−11.07
Carrot Defense-like
+7.46
+5.23
+0.31
+12.72
+1.64
+13.51
Snake
+11.43
−4.52
+8.58
+5.69
+10.34
+2.96
Tetris
+12.08
−4.63
+34.76
+2.85
+2.38
−2.34
Appendix
Table 21. Paired differences in Overall score for the same model and game. Differences are computed before rounding.
Model
M
D
B
D−M
B−D
GPT-5.6 Terra
53.10
75.29
75.55
+22.19
+0.27
Qwen3.7 Plus
45.94
65.45
68.05
+19.51
+2.60
Grok 4.3
51.75
68.41
70.34
+16.65
+1.93
Appendix
Table 22. Mean Overall scores on the common 11-game subset, excluding Carrot Defense-like, Pipe Connection Puzzle, Match-3, and Mini Zelda-like Room Adventure consistently across models and conditions. Scores use the same continuous D1 gate as the full Core-15 results.
#
Game
Experiment ID
1
Snake
snake
2
Dodge Blocks
dodge_blocks
3
Asteroids
asteroids
4
Tetris
tetris
5
Simple Platformer
simple_platformer
6
Match-3
match3
Appendix
Table 23. Nine games included in Ablation Study II.
Table 24. Configurations of the two text judge conditions in Ablation Study II.
Judge condition
Qwen wins
Terra wins
Ties
Binomial p
Qwen win rate
Qwen3-235B-A22B-2507
13
14
0
1.000
48.1%
GPT-5.6 Sol
15
12
0
0.701
55.6%
Appendix
Table 25. Paired ranking results under the two text judge conditions.
Comparison
N
ρ
τ
r
MAE
D2.1 Element modeling
438
0.049
0.048
0.266
0.12
D2.2 Interaction modeling
384
0.466
0.418
0.596
0.37
D2.3 Rules and mechanisms
314
0.551
0.457
0.539
0.50
D2.4 Dynamic evolution
391
0.521
0.481
0.697
0.25
D2.5 Goal and Termination Modeling
277
0.805
0.699
0.799
0.33
D2 overall
243
0.779
0.655
0.791
0.13
Appendix
Table 26. Candidate-level questionnaire–LLM alignment on primary assignments. N denotes paired candidates for each comparison. D2 errors use a 0–2 scale, and D4 errors a 0–100 scale. D4.3 compares questionnaire accuracy on one question with the official two-question automatic score. Overall comparisons use the reported candidate-level aggregates and are not averages of the correlations above.
Assessment category
Pairable / total
Available α
Complete α
D2 functional checkpoints
60/75
0.687
0.352
D4.1 overall visual quality
10/15
0.049
0.049
D4.2 dynamic feedback
20/30
0.753
0.349
D4.3 action choice
9/15
0.544
0.544
Question validity (auxiliary)
9/15
0.049
0.049
Appendix
Table 27. Descriptive agreement among shared-set responses. Counts refer to assessment items rather than distinct projects. Complete-case counts are 37, 10, 7, 9, and 9, respectively. These estimates do not measure questionnaire–LLM alignment and do not establish annotation independence.
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
LLM-based game generation promises to turn natural-language specifications into executable games, but progress is limited by the lack of reliable automated verification. Unlike conventional code generation, game correctness is defined over long-horizon interaction: a game may appear correct while violating core mechanics such as state updates, interaction rules, and phase transitions. Existing Agent-as-a-Verifier approaches collapse verification into open-ended gameplay, making verdicts reachability-bound, time-consuming, coverage-limited, and sensitive to the agent's gameplay ability. We present GameGen-Verifier, an automated verification paradigm for LLM-generated games that decomposes a specification into verifiable keypoints and grounds them into independent verification units. Each unit patches the game runtime into a concrete target state, executes a bounded interaction, and judges the outcome against the keypoint assertion. We implement GGV-Harness, a scalable agentic harness providing concurrency management, runtime isolation, and fault recovery. On VeriGame, our dataset of 100 games across seven genres, GameGen-Verifier achieves up to 92.2% accuracy against human judgments versus 58.8% for the coverage-enforced Agent-as-a-Verifier baseline, while reducing wall-clock time by up to 16.6x.
Large Language Models (LLMs) have shown great ability in generating executable code from natural language, opening the possibility of automatically constructing environments for AI agents. Recent work on Code World Models (CWMs) demonstrates that LLMs can translate game rules into Python implementations compatible with solvers like Monte Carlo Tree Search. We study this problem in game settings, where generated environments must implement rules, legal actions, state transitions, observations, and rewards. We refer to these game-specific executable models as Game Code World Models (GameCWMs). However, current approaches to generating code world models rely on frontier models and inference-time refinement loops, limiting accessibility and scalability. This work investigates whether GameCWM generation capabilities can be distilled into smaller models through post-training. We introduce: (1) a curated dataset of 30 games spanning perfect and imperfect information games, (2) a verification framework that evaluates generated code against structural and semantic game properties, and (3) a post-training pipeline combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR). We experiment with Qwen2.5-3B-Instruct and find that SFT can increase syntactic correctness, while RLVR can improve execution-level adherence to game rules, thereby improving Qwen's ability to generate valid GameCWMs in both perfect and imperfect information games. Overall, our pipeline makes Qwen2.5-3B-Instruct more capable of generating valid GameCWMs, thereby offering a scalable path toward automatic environment generation from natural language.