Complete game generation requires agents to coordinate rules, interactions, states, and presentation over a long development process. A single aggregate score obscures which capabilities are implemented and which requirements remain unmet. We present OCTAVE, a benchmark for complete HTML5 game generation organized around eight Capability Dimensions, with 180 tasks and 1,800 Evaluation Requirements drawn from real player requests, commercial-mechanics reconstruction, and human expert design. It covers PC Web, Mobile Web, 2D, and 3D. Requirements distinguish Completeness, assessing publicly specified core implementation, from Richness, assessing hidden, deeper system relationships. Static source-code judgments provide requirement-level diagnosis across capabilities and task conditions. Evaluation of 11 model-harness combinations yields 33.2% Completeness and 32.5% Richness; 19.9% of generation records are static gate failures or missing artifacts. Excluding same-model self-evaluation, even the best combination remains below 50% on both public and hidden requirements. Richness is not harder than Completeness under every task condition. Systems perform relatively better on Core Rules and Numerical Design, while AI/NPC and Feedback and Game Feel show more prominent gaps. Requirement difficulty offers coarse stratification but is only weakly negatively correlated with pass rates. A limited human cross-check finds that requirements judged satisfied by humans but failed by the automated evaluator are the main direction of disagreement. OCTAVE extends evaluation beyond aggregate scores to diagnosis localized by requirement, capability, and task condition.
Figures & tables
Benchmark
Capability-controlled construction
Richness evaluation
Mobile Web
Complete game generation
Tasks
Requirements
GameDevBench Chi et al. (2026)
No
No
No
No
333
—
OpenGame-Bench Jiang et al. (2026)
No
No
No
Yes
150
—
V-GameGym Zhang et al. (2025)
No
No
No
Yes
2,219
2,219†
GameGen-Verifier Jia et al. (2026)
No
No
No
Yes
100
—
GameCraft-Bench Luo et al. (2026)
No
Yes
No
Yes
140
2,305
WebGameBench Zhang et al. (2026c)
No
No
No
Yes
111
111†
Table 1: Comparison of existing game-development benchmarks and OCTAVE. Tasks are counted as independent game-project evaluation instances. Requirements are the specific functional requirements or scoring checklist entries accumulated across games, without repeated counting across models or runs. When only an overall evaluation is available and no game-specific functional scoring subitems are provided, each game is counted as one requirement (†); general quality dimensions are not counted repeatedly. GameXpert-Bench Chen et al. (2026) is restricted to its GameGen track. Dashes indicate that the cited papers provide insufficient information to determine the totals. GameCraft-Bench’s total is counted from its released rubrics.
Figure 1: Construction and evaluation pipeline
Reference design concept
Capability Dimension
Operational scope
Mechanics
C1 Core Rules
Game rules, state transitions, core mechanics, and win/loss conditions
Systems
C2 AI/NPC
NPC behavior, decision logic, and autonomous system behavior
Systems
C3 Numerical Design
Attributes, resources, economies, progression, and numerical relationships
Gameplay
C4 Interaction and Input
Player input, interaction mapping, and game controls
Macrostructure
C5 Levels and Scenes
Spatial organization, levels, scenes, and game progression
Player Experience
C6 Animation and Rendering
Rendering, animation, and dynamic visual states
Table 2: Capability Dimensions and their relationship to game-design concepts.
Statistic
Value
Candidate tasks / final tasks
280 / 180
Task sources
150 player requests; 20 commercial-mechanics reconstructions; 10 expert-designed tasks
Game genres
6
Visual forms
2D, 3D
Target platforms
95 PC Web; 85 Mobile Web
Requirements per task
6 Completeness; 4 Richness
Table 3: Overall statistics of OCTAVE. Candidate tasks are the 280 tasks entering final selection: 250 player requests, the 20 retained commercial-mechanics reconstructions, and the 10 expert-designed tasks. A further 11 reconstruction drafts were discarded upstream and are not counted.
Dimension
Category
Tasks
Source
Player request
150
Commercial-mechanics reconstruction
20
Expert design
10
Game genre
Action
30
Adventure & Puzzle
31
RPG
24
Table 4: Category distribution of the final tasks. Each classification dimension is counted separately, and each dimension sums to 180 tasks.
Figure 2: (a) Distribution of primary requirement capabilities; (b) task-level multilabel capability coverage; and (c) distribution of requirement difficulty. The primary-capability distribution uses the first capability label of each Evaluation Requirement; task-level multilabel coverage reflects the occurrence of different capabilities in the task set; and the requirement-difficulty distribution shows the number of requirements in the four equal-count strata defined in Appendix C. Detailed label summaries are provided in Appendix A.4, and the difficulty-stratification rules are provided in Appendix C.
Human / machine
Fail (0)
Pass (1)
Satisfied
81
102
Not satisfied
47
11
Table 5: Requirement-level human–machine confusion matrix. Only records that humans could judge and that were aligned requirement by requirement with the Astra evaluation are included. Judgment records for artifacts that failed the static gate are retained. Each artifact was assessed by a single human annotator.
Model
Harness
Generation records
Gate failure or missing artifact
Completeness
Richness
DeepSeek-V4.1-Flash
CC
180
20.6%
25.9%
27.6%
DeepSeek-V4-Flash-0731
CC
180
26.7%
23.5%
26.9%
Fable 5
CC
180
21.1%
39.0%
36.4%
Fable 5.1
CC
180
17.8%
44.7%
38.2%
GLM-5.2
CC
180
26.7%
24.3%
25.8%
GLM-5.3
CC
180
21.7%
31.1%
31.8%
Table 6: Main results for the model–harness combinations. CC denotes Claude Code and CX denotes Codex CLI; † marks same-model self-evaluation. Each combination is evaluated on 180 tasks, and requirement pass rates use all-record analysis.
Figure 3: Completeness and Richness by task source
Figure 4: Requirement pass-rate point estimates by capability. Coverage and aggregation sensitivity are reported in Appendix G.2.
Figure 5: Pass rates across requirement-difficulty strata. E, M, H, and X denote Easy, Medium, Hard, and Extreme, respectively. The three curves show Completeness, Richness, and the combined pass rate for both requirement types. Difficulty strata follow the equal-count partition defined in Appendix C. These are pooled descriptive curves; subset-specific rates and composition sensitivity are reported in Appendix G.1.
Figure 6: Results by platform
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Paper capability
Coverage in 250 player candidates
Coverage in 150 selected player tasks
C1
250 (100.0%)
150 (100.0%)
C2
215 (86.0%)
133 (88.7%)
C3
224 (89.6%)
130 (86.7%)
C4
115 (46.0%)
74 (49.3%)
C5
154 (61.6%)
91 (60.7%)
C6
111 (44.4%)
74 (49.3%)
Appendix
Table A.1: Player-task capability coverage before and after selection.
Figure A.1: Capability coverage before and after player-task selection
Platform
Player tasks
Reconstruction tasks
Expert-designed tasks
Total
PC Web
75
10
10
95
Mobile Web
75
10
0
85
Total
150
20
10
180
Appendix
Table A.2: Task-source composition by platform.
Section
Content
[Game]
Game genre and core concept.
[Core Mechanics]
Principal systems, states, causal relationships, and gameplay loop.
[Must Implement]
Required behaviors that support the core design.
[Creative Freedom]
Presentation, content, and extensions left to the implementer.
Appendix
Table B.1: Four-part template for player-task specifications.
Layer
Full replacement
Same-source revision
Retained unchanged
Total
Public
196 (43.6%)
73 (16.2%)
181 (40.2%)
450
Hidden
99 (33.0%)
29 (9.7%)
172 (57.3%)
300
Appendix
Table B.2: Revisions to Mobile Web requirements.
Dimension
0
1
2
3
D1 Breadth of system coupling
A single loop and accessory elements
Two hard systems, primarily with one-way relationships
Three hard systems, including bidirectional relationships
At least four densely coupled hard systems or a causal chain spanning three systems
D2 Algorithmic and simulation core
State machines and numerical tables
One standard algorithm
A nontrivial core
Multiple jointly operating cores or a simulation-level core
D3 Verification rigor
Functional presence
Seed consistency or single-point reproducibility
Comparison or conservation
Replay/hashing combined with multiple types of constraints
D4 Dynamic governance and adaptation
No dynamic balancing
A single feedback loop
Multiple feedback loops or persistent stability constraints
Closed-loop adaptation and interlocking multiple loops
D5 Multi-agent and social cognition
None, or scripted NPCs
State machines and simple perception
Local information or opponent modeling
Multi-party games, reputation, or shared beliefs
D6 Scale and performance
No scale requirement
A soft stability constraint
Explicitly tens to hundreds of entities
Thousands of entities or hundreds of nodes, with hard smoothness requirements
Appendix
Table C.1: Design-complexity dimensions and scoring anchors.
model_key
Display name
Agent harness
Version
effort
cc_deepseek_flash_41
DeepSeek-V4.1-Flash (API id deepseek-flash)
Claude Code
2.1.263
None
cc_deepseek_v4_flash
DeepSeek-V4-Flash-0731
Claude Code
2.1.263
None
cc_fable_5
Fable 5 xhigh
Claude Code
2.1.263
xhigh
cc_fable_5_1
Fable 5.1 xhigh
Claude Code
2.1.263
xhigh
cc_glm_5_2
GLM-5.2
Claude Code
2.1.263
None
cc_glm_5_3
GLM-5.3
Claude Code
2.1.263
None
Appendix
Table D.1: Model and agent-harness configurations.
System
Paired items
Human pass
Judge pass
GPT-6-astra
34
94.1%
61.8%
Fable 5.1
33
84.8%
60.6%
GLM-5.3
33
57.6%
33.3%
Opus 5
31
90.3%
58.1%
Fable 5
28
42.9%
32.1%
GPT-5.5
25
80.0%
48.0%
Appendix
Table F.1: Per-system human and judge pass rates on the paired items of the human cross-check. Only systems with at least 20 paired items are listed; the judge column applies the all-record gate rule.
Difficulty
Requirements
Model–requirement records
Public pass rate
Hidden pass rate
Overall pass rate
Easy
450
4,950
38.75%
40.43%
39.54%
Medium
450
4,950
32.69%
31.64%
32.22%
Hard
450
4,950
31.80%
29.18%
30.75%
Extreme
450
4,950
30.70%
25.65%
29.25%
Appendix
Table G.1: Requirement pass rates across difficulty strata.
Subset
Easy
Medium
Hard
Extreme
Player PC
18.34%
20.01%
20.02%
20.48%
Player Mobile
37.80%
34.67%
35.95%
36.69%
Reconstruction
56.91%
49.12%
47.92%
49.73%
Expert
72.11%
63.30%
65.66%
72.73%
Appendix
Table G.2: Overall requirement pass rates within each subset across the global requirement-difficulty strata.
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
Coding agents are increasingly used as application builders, yet many evaluations still focus on source code, repository-level tests, or intermediate traces rather than the delivered application. We introduce WebGameBench, a requirement-to-application benchmark that evaluates whether coding agents can turn a frozen Structured WebGame Specification into a browser-accessible game. Browser-native games provide a compact but behavior-dense testbed: even simple games require coordinated input handling, spatial mapping, rule execution, state transitions, terminal conditions, restart behavior, and visible feedback. In WebGameBench, each generated artifact is built, served, and exposed as a browser-accessible application under a unified deployment protocol. A runtime evaluator then interacts with the delivered game in a real browser and assigns a three-way label: EXCELLENT, USABLE, or UNUSABLE. On a human-reviewed subset, the runtime label is broadly aligned with human gameplay review under the Usable-rate criterion. Across 111 tasks, 12 coding agents, and 14 evaluation configurations, WebGameBench separates current systems: the best configuration reaches a 76.9% usable rate but only a 20.2% excellent rate. This gap shows that crossing the minimum playable-delivery threshold is still far from complete requirement satisfaction. To our knowledge, WebGameBench is the first requirement-to-application benchmark for browser-native game delivery that validates delivered-application runtime labels against independent human gameplay review under the Usable-rate criterion.
Wenyu Zhang, Guoliang You, Tianlun +8
1Baidu · University of Science and Technology of China
Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.