cs.AINov 13, 2026

OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation

Authors: Kehang Li, Xinlan Xu, Yulin Jin, Heng Cao, Xuchen Li, Houxian Ren, Zihao Ji, Qiang Luo, 

Organizations: Shanghai Jiao Tong University · Zhongguancun Academy · Shandong University · Rezona Lab

Abstract

Complete game generation requires agents to coordinate rules, interactions, states, and presentation over a long development process. A single aggregate score obscures which capabilities are implemented and which requirements remain unmet. We present OCTAVE, a benchmark for complete HTML5 game generation organized around eight Capability Dimensions, with 180 tasks and 1,800 Evaluation Requirements drawn from real player requests, commercial-mechanics reconstruction, and human expert design. It covers PC Web, Mobile Web, 2D, and 3D. Requirements distinguish Completeness, assessing publicly specified core implementation, from Richness, assessing hidden, deeper system relationships. Static source-code judgments provide requirement-level diagnosis across capabilities and task conditions. Evaluation of 11 model-harness combinations yields 33.2% Completeness and 32.5% Richness; 19.9% of generation records are static gate failures or missing artifacts. Excluding same-model self-evaluation, even the best combination remains below 50% on both public and hidden requirements. Richness is not harder than Completeness under every task condition. Systems perform relatively better on Core Rules and Numerical Design, while AI/NPC and Feedback and Game Feel show more prominent gaps. Requirement difficulty offers coarse stratification but is only weakly negatively correlated with pass rates. A limited human cross-check finds that requirements judged satisfied by humans but failed by the automated evaluator are the main direction of disagreement. OCTAVE extends evaluation beyond aggregate scores to diagnosis localized by requirement, capability, and task condition.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

    Jun 16, 2026Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22game generationAI Agent Benchmarks

  2. WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

    May 17, 2026Wenyu Zhang, Guoliang You, Tianlun +8Web Application GenerationAI Coding Agents

  3. GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development

    Sep 18, 2026Xiuhui Zhang, Yi Chen, Shusheng Xu +4Code Generation BenchmarksAI Agent Benchmarks