Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.
Figures & tables
Document
File
Content
Generation prompt
target.md
Workspace, delivery protocol, and concise gameplay brief.
Gameplay design requirement
game-spec.md
Player-visible mechanics and the minimum playable loop.
Evaluation interface specification
tdd.md
Scenario setup, action and observation semantics, rejection behavior, and invariants.
Table 1: Task documents and their roles. All three are available to the coding agent; only the generation prompt is passed directly to the harness.
Figure 1: Composition of the 47 tasks. Each panel covers all 47 tasks; legend entries give counts and percentages. RTS and TBS denote real-time and turn-based strategy, respectively. Reference technology identifies the rendering technology used by each reference implementation.
L1: source-level checks
L2: behavioral checks
Tool checks
43
Prerequisite checks (P0)
102
Regular-expression checks
288
Core requirement checks (P1)
534
Anti-pattern checks
5
Extended capability checks (P2)
249
Total
336
Total
885
Table 2: Check counts for the 47 tasks used in all reported experiments. Layer and priority describe different properties of a check.
Model
Harness
Strict task success
L1
L2
Mean
P0
P1
P2
GPT-6-Astra
Codex CLI
26/47 (55.3)
98.0
93.2
99.0
92.7
92.4
Claude-Opus-5
Claude Code
24/47 (51.1)
99.6
90.4
91.2
89.9
91.1
GPT-5.6-Sol
Codex CLI
21/47 (44.7)
98.9
91.3
100.0
89.7
91.9
DeepSeek-V4-Flash
Claude Code
18/47 (38.3)
99.4
89.3
98.0
86.7
90.2
DeepSeek-V4-Pro
Claude Code
15/47 (31.9)
99.6
85.2
85.3
83.9
88.9
Table 3: End-to-end results at maximum reasoning effort. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate (%); all other entries are check pass rates (%). L1 and overall L2 entries are mean per-task check pass rates; L2 P0–P2 entries pool applicable checks within each priority. Bold denotes the highest observed value in each column.
Model
Harness
Artifact size (KB)
Input tokens (k)
Output tokens (k)
Monetary cost (USD)
GPT-6-Astra
Codex CLI
78.82
896.1
55.5
4.4445
Claude-Opus-5
Claude Code
68.52
12,632.6
142.4
11.8500
GPT-5.6-Sol
Codex CLI
64.67
1,382.5
45.7
1.7867
DeepSeek-V4-Flash
Claude Code
85.21
13,038.4
203.6
0.2461
DeepSeek-V4-Pro
Claude Code
90.34
12,944.3
202.9
0.7431
Kimi-K3
Claude Code
57.51
1,263.9
72.3
2.5518
Table 4: Mean artifact size and reported resource use. Artifact sizes cover all 47 tasks per stack. Input tokens combine the reported input, cache-read, and cache-write categories.
Tool setting
Strict task success
L1
L2 P0
L2 P1
L2 P2
No tools (serialized API)
7/47 (14.9)
98.5
83.3
65.3
80.1
File read/write only
6/47 (12.8)
98.8
87.3
66.2
73.4
Files + syntax checking
9/47 (19.1)
98.2
86.3
63.4
69.3
Full tool access
18/47 (38.3)
99.4
98.0
86.7
90.2
Table 5: Tool configurations for DeepSeek-V4-Flash. All 47 tasks complete evaluation in each condition. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate (%); check pass rates are percentages. Syntax checking permits Bash for node --check , but no browser. The configuration with full tool access uses the nominal 120-turn baseline.
Nominal turn budget
Evaluated / planned
Strict task success
L1
L2 P0
L2 P1
L2 P2
All planned
Conditional
30
10/47
7/47 (14.9)
7/10 (70.0)
100.0
100.0
94.0
97.6
60
32/47
13/47 (27.7)
13/32 (40.6)
99.7
88.6
81.6
88.5
120
47/47
18/47 (38.3)
18/47 (38.3)
99.4
98.0
86.7
90.2
Table 6: Nominal turn-budget comparison for DeepSeek-V4-Flash with full tool access. The primary strict task success rate uses all 47 planned tasks as the denominator; conditional strict task success rates and check pass rates describe only the evaluated subset. Counts are followed by percentages in parentheses; check pass rates are percentages.
Effort
Strict task success
L1
L2 P0
L2 P1
L2 P2
Reasoning tokens
Low
7/47 (14.9)
98.3
93.1
71.7
80.2
37,699
High
19/47 (40.4)
99.4
97.1
82.7
88.9
138,232
Max
18/47 (38.3)
99.4
98.0
86.7
90.2
189,210
Table 7: Reasoning-effort comparison for DeepSeek-V4-Flash. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate (%); check pass rates are percentages. Reasoning-token counts are averaged across tasks. Bold denotes the highest observed number of strict task successes or check pass rate.
Harness
Strict task success
L1
L2 P0
L2 P1
L2 P2
Claude Code
18/47 (38.3)
99.4
98.0
86.7
90.2
Codex CLI
18/47 (38.3)
99.2
96.1
87.1
92.2
Table 8: Harness comparison for DeepSeek-V4-Flash at maximum reasoning effort. Strict task success is reported as the number of successful tasks out of 47 and the corresponding rate (%); check pass rates are percentages. Bold denotes the highest observed value in each column.
Artifact / stack
Observed evidence
Role of the checks
Diner Dasher GPT-6-Astra
L1 and L2 P0/P2 pass; L2 P1 mouse and touch drag checks fail because real drags produce no service progress.
Real-input checks expose a core interaction failure despite other passing checks.
Turbo Smash Beast DeepSeek-V4-Flash
Reset or scenario loading stops natural-time simulation; acceleration, coasting, and related real-input checks fail.
Checks combining scenario preparation and real input expose a failure in their interaction.
Table 9: Diagnostic evidence linking evaluation checks to observed failures. The GPT-6-Astra case uses Codex CLI; the DeepSeek-V4-Flash case uses Claude Code with maximum reasoning effort and full tool access.