3D GameDev Arena: Benchmarking Coding Agents That Plan, Build, and Playtest extreme Long-Horizon Browser Games
Organizations: Fudan University · Shanghai Innovation Institute · Xi’an Jiaotong University · Shanghai Jiao Tong University
Abstract
Coding agents now resolve repository issues and generate compact playable games, but whether they can build a large 3D game from scratch remains open. Such games tightly couple mechanics, scenes, assets, and interaction logic, and their quality rests on player enjoyment, which existing benchmarks rarely assess. Thus, we introduce 3D GameDev Arena, a first extreme long-horizon benchmark of 108 from-scratch game-development tasks across 14 genres. Given only a text specification and UI/UX reference images, a fully automated plan--implement--test--repair pipeline builds each task into a self-contained browser game. Runs average 1,000 LLM calls ; the longest uses 14,407 calls, 2.13B tokens, and 608 assets. Evaluation spans (1) objective code-level functional acceptance against the game specification, (2) agent-based QA and playtesting of 3D presentation and playability, and (3) blind pairwise Bradley-Terry human preference in a public arena. The preliminary findings indicate that long-horizon 3D game development remains far from solved, and code-level correctness alone does not dictate player preference. The public arena is hosted online.
Figures & tables
| Benchmark | 3D/Web | Input | Starter code | Run length | Evaluation |
|---|---|---|---|---|---|
| GameCraft [ 28 ] | / | T,A | NR | 128 tool calls; 2 h cap | Replay + VLM rubric |
| GameXpert [ 9 ] | / | T,S | No (Gen); Yes (Fix/Opt) | NR | Runtime tests, agent judge, human ratings |
| GameASG [ 47 ] | / | T | No | 120-turn cap | Static + executable browser checks |
| GameGen-Verifier [ 20 ] | —/ | T | NR | NR | State injection, VLM, assertions |
| WebGameBench [ 46 ] | —/ | T | Template | 20–24 turns; 2 h cap | Browser agent judge |
| SWE-Game [ 10 ] | / | T,A,V,S | Varies by mode | 57–111 tool calls | Evaluator checks, reference comparison, VLM rubric |
| Functional | Agent playtesting | Human preference | ||||
|---|---|---|---|---|---|---|
| Configuration | Score | 3D Presentation | Playability | Coverage | BT Score | Raw Win |
| codex-gpt-6.1-sol | 64.7 | 57.8 | 68.4 | 91.3 | 69.11 | 62.8 |
| codex-gpt-6-astra | 59.4 | 58.5 | 69.5 | 95.8 | 69.24 | 66.2 |
| kimi-k3 | 59.8 | 42.3 | 64.9 | 95.9 | 25.86 | 27.4 |
| zcode-glm-5.3 | 51.2 | 23.5 | 54.9 | 93.2 | 35.78 | 30.6 |
| Configuration | Norm. Time (h) | ( ) | ( ) | Tool Calls | Turns / Sub-tasks | Source Files | Clean LOC | BT Score | Raw Win |
|---|---|---|---|---|---|---|---|---|---|
| codex-gpt-6.1-sol | 3.82 | 162.0 | 0.624 | 1,067 | 1,186 / 18.5 | 35.0 | 2,328 | 69.11 | 62.8% |
| codex-gpt-6-astra | 2.72 | 121.5 | 0.532 | 921 | 1,044 / 17.0 | 39.5 | 2,885 | 69.24 | 66.2% |
| zcode-glm-5.3 | 2.65 | 70.1 | 0.456 | 692 | 645 / 25.0 | 82.0 | 13,824 | 35.78 | 30.6% |
| kimi-k3 | 7.11 | 202.1 | 0.694 | 1,045 | 756 / 7.0 | 51.5 | 7,572 | 25.86 | 27.4% |
| Config. | BT Score | Uncached In | Cache Read | Cache Replay |
|---|---|---|---|---|
| gpt-6.1-sol | 69.11 | 6.82M | 307.41M | |
| gpt-6-astra | 69.24 | 5.15M | 200.46M | |
| kimi-k3 | 25.86 | 4.91M | 241.04M | |
| glm-5.3 | 35.78 | 1.86M | 82.27M |
| Config. | Debug Calls | tsc + vitest | build + test | Smoke | Func. Score | BT / Raw Win |
|---|---|---|---|---|---|---|
| gpt-6.1-sol | 103.0 (184.6) | 8.4% | 83.3% | 8.1% | 64.7% | 69.1 / 62.8% |
| gpt-6-astra | 55.0 (120.9) | 11.2% | 76.8% | 11.4% | 59.4% | 69.2 / 66.2% |
| kimi-k3 | 159.5 (167.9) | 63.5% | 17.7% | 16.9% | 59.8% | 25.9 / 27.4% |
| glm-5.3 | 103.5 (120.1) | 63.4% | 15.2% | 21.1% | 51.2% | 35.8 / 30.6% |
| Strategy | Mean count per game | |||||||
| Config. | Prim | Ret | Gen | Hyb | Dyn | Generated | Retrieved | Primitives |
| gpt-6-astra | 17 | |||||||
| gpt-6.1-sol | — | — | 0 | 8 | ||||
| kimi-k3 | — | — | — | — | 0 | 13 | 60 | |
| glm-5.3 | — | — | — | — | 0 | 0 | 11 | |
| Goal retention (%) Plan adherence (%) Config. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 gpt-6.1-sol 100 100 100 100 100 100 96.2 91.4 gpt-6-astra 100 100 100 100 100 100 100 100 kimi-k3 84.5 94.0 98.8 96.2 100 100 100 98.6 |
| Full run (%) Calls by task size Config. GR GR † Small Medium Large BT gpt-6.1-sol 99.7 99.9 849 1,186 1,645 69.1 gpt-6-astra 100 100 971 1,132 1,404 69.2 kimi-k3 96.1 99.3 601 730 761 25.9 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Specification Section | Formal Purpose | Instantiation A: MOBA Slice ( 1v1 Nexus Duel ) | Instantiation B: Arcade Racing Slice ( Town Highway ) |
|---|---|---|---|
| Role and Objective | Defines the target software artifact, player/AI configuration, and functional task scope. | Construct a playable 3D browser MOBA duel pitting one human-controlled champion against an autonomous AI opponent, featuring six selectable champions and terminating upon Nexus destruction. | Construct a single-player 3D arcade kart-racing adaptation featuring the Town Highway circuit, three selectable drivers, a Cotton PRO kart, and a two-lap race against three AI competitors. |
| Game Overview and Player Experience | Articulates the intended player experience, core loop pacing, mechanical highlights, and visual direction. | Deliver an 8–15-minute tactical lane duel centered on minion last-hitting, skillshot evasion, damage trading, itemization, and wave-assisted structure pushing. | Deliver a 2–3-minute race set in a vibrant cartoon town, emphasizing corner entry, drift initiation, counter-steering, grip recovery, and strategic nitro-booster overtaking. |
| Project Setup and Deliverables | Establishes engine/rendering constraints, build contracts, and self-contained packaging rules. | Deliver complete source code and a root-level build.sh script compiling an entry index.html , with all runtime assets bundled locally offline. | Deliver complete source code and a root-level build.sh compiling index.html with all runtime assets bundled locally. |
| Complete Game Design | Formalizes controls, state variables, numerical parameters, state transitions, and termination rules. | The Nexus holds and remains invulnerable while its defensive lane turret stands. Champion eliminations create wave-pushing windows rather than immediate victory. | Sustained moving drifts charge a gauge; reaching grants one booster (up to two stored slots). Pressing Ctrl consumes one booster only when no boost is currently active. |
| Acceptance Criteria | Translates design rules into deterministic, reproducible checks with observable runtime outcomes. | Attacking the Nexus while its turret is intact must leave Nexus HP unchanged; destroying the turret and subsequently reducing Nexus HP to zero must trigger the victory state while logging all HP deltas and state transitions. | Verify that stationary steering yields zero gauge charge. Execute valid drifts to earn and trigger a booster: stored count must decrement by one and kart velocity must increase, while repeated Ctrl inputs during active boost must neither consume extra stock nor extend boost duration. |
| Level | Control: does the action get a readable response? | Flow: does the player know the state and the next step? |
|---|---|---|
| 1 | The player probably cannot tell whether the action worked, or misreads the result. | The player probably misses the change, or thinks the game is stuck or broken. |
| 2 | Something changes, but the player has to search or guess what it means. | The player notices a change but has to guess the state or the next step. |
| 3 | The result is recognizable, but feedback is weak, far from where the player is looking, or hard to tell apart. | The state is clear, but the player has to explore to find out how to continue. |
| 4 | The result is clear, timely, and where the player is looking; only minor roughness. | State and next step are both clear; only small details are missing. |
| 5 | As 4, and the kind, size, and target of the result are obvious at a glance. | As 4, and the player also knows what to expect, such as a time limit or the cost of failing. |
| Config. | 3D Pres. | Playability | Control | Flow | BT |
|---|---|---|---|---|---|
| gpt-6.1-sol | 57.8 (61.4) | 68.4 (68.8) | 3.40 (60.1) | 4.07 ( 76.8 ) | 69.11 |
| gpt-6-astra | 58.5 (59.9) | 69.5 ( 70.0 ) | 3.49 ( 62.2 ) | 4.07 (76.7) | 69.24 |
| kimi-k3 | 42.3 (43.4) | 64.9 (65.0) | 3.35 (58.7) | 3.84 (71.1) | 25.86 |
| glm-5.3 | 23.5 (17.2) | 54.9 (55.8) | 2.88 (47.1) | 3.51 (62.6) | 35.78 |
| Pair | Pres. diff | Play. diff |
|---|---|---|
| gpt-6.1-sol – gpt-6-astra | ||
| gpt-6.1-sol – kimi-k3 | ||
| gpt-6.1-sol – glm-5.3 | ||
| gpt-6-astra – kimi-k3 | ||
| gpt-6-astra – glm-5.3 | ||
| kimi-k3 – glm-5.3 |
| Question | Test | Result |
|---|---|---|
| Is it stable? | Rate the same moments twice | identical in all cases |
| Swap the per-level examples for a different set, or drop them | game scores almost unchanged ( ) | |
| Does each score track its own property? | Remove the game’s hit and action feedback effects | control drops; flow and presentation unchanged |
| Repaint the scene structure in one flat color | presentation drops; control and flow unchanged | |
| Why two formats? | Rate presentation one moment at a time | order agreement drops (23 / 8 / 2); the repaint goes unnoticed |
| Judge playability by pairwise comparison | the repaint also lowers control and flow |
| Configuration ID | Short name | Harness | Model |
|---|---|---|---|
| codex-gpt-6.1-sol | gpt-6.1-sol | Codex CLI 0.167.0 | GPT-6.1 Sol |
| codex-gpt-6-astra | gpt-6-astra | Codex CLI 0.167.0 | GPT-6 Astra |
| zcode-glm-5.3 | glm-5.3 | Zcode 0.16.9 | GLM-5.3 |
| kimi-k3 | kimi-k3 | Kimi CLI 2.6.1 | Kimi K3 |
| All builds | Large tasks | |||
|---|---|---|---|---|
| Configuration | GR | GR † | GR | GR † |
| gpt-6-astra | 100.00 | 100.00 | 100.00 | 100.00 |
| gpt-6.1-sol | 99.66 | 99.94 | 100.00 | 100.00 |
| kimi-k3 | 96.06 | 99.28 | 94.51 | 97.64 |
| glm-5.3 | 100.00 | 100.00 | 100.00 | 100.00 |