GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers

Shufan Jiang; Chios Chen; Zhiyang Chen

cs.SEApr 3, 2026

GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers

Authors: Shufan Jiang, Chios Chen, Zhiyang Chen

Organizations: The University of Hong Kong · Datawhale Org. · Independent Researcher · Westlake University

Abstract

The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.

Figures & tables

Figure 1: Evolution of the software development paradigm in the LLM era. (a) Traditional human-driven iterative workflow. (b) Human–LLM collaborative coding, where a coding agent assists development under human supervision. (c) Toward a fully autonomous coding system which can generate code, detect bugs and fix them without human-in-the-loop. While existing benchmarks primarily focus on code generation and fixing, our benchmark emphasizes autonomous bug discovery and quality assurance part within the development cycle.

Figure 2: Overview of GBQA. Dataset is constructed using a multi-agent game builder that generates 30 game environments with 124 implanted bugs, which are annotated and categorized into three difficulty levels (Easy, Medium, Hard) by human QA experts. During evaluation, a QA agent autonomously interacts with the game environment through ReAct loops, and produces structured bug reports. Then, a critic agent verifies reported bugs by matching them against human-annotated ground truth to compute quantitative metrics.

Model	Player Exploring Mode				Quality Assurance Mode				Best Performance
Model	50	100	200	500	50	100	200	500	Best Performance
LLMs in Instruct Mode
Claude-4.6-Opus	14.52	20.97	25.81	31.45	22.58	28.23	31.45	37.90	37.90
Claude-4.5-Sonnet	11.29	16.13	18.55	20.97	17.74	25.00	28.23	32.26	32.26
GPT-5.2	7.26	10.48	12.90	14.52	11.29	16.94	19.35	22.58	22.58
Kimi-K2.5-1T-A32B	6.45	9.68	11.29	13.71	10.48	15.32	17.74	20.97	20.97

Table 1: GBQA Leaderboard. We report Recall (%) under two testing modes across four step budgets. Bold values indicate the highest score. Best Performance denotes the highest score achieved by each model across all settings.

Figure 4: Ablation study of memory module. Each cluster corresponds to a session, and vertical arrows indicate performance gains as the step budget increases. The four trend lines illustrate the aggregated trend for same memory settings across sessions.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Architectural overview of the Game Environment Builder. The Producer Agent orchestrates the end-to-end pipeline, coordinating three specialized teams (Design, Programming, Art) across isolated workspaces. Each team follows a structured planning–execution loop, where role-specific worker agents are dynamically instantiated, execute atomic subtasks, and report results for iterative validation. A shared utility platform provides cross-functional capabilities upon agent initialization. This multi-agent architecture enables automated, scalable, and modular environment generation.

Model	SWE-Bench Verified	GBQA	Source
Claude-4.6-Opus	81.4%	48.39%	( Anthropic, 2026 )
Gemini-3.1-Pro	80.6%	33.06%	( Google DeepMind, 2026 )
GPT-5.2	80.0%	22.58%	( OpenAI, 2025b )
Claude-4.5-Sonnet	77.2%	32.26%	( Anthropic, 2025b )
Qwen3-Coder-Next	70.6%	–	( Cao et al., 2026 )
DeepSeek-R1	57.6%	37.90%	( Guo et al., 2025a )

Appendix

Table 4: Performance comparison of frontier models on SWE-Bench Verified and GBQA. The pronounced performance gap highlights the increased complexity of autonomous bug discovery, which necessitates capabilities extending well beyond standard code resolution.

Figure 6: Distribution of game genres across the 30 games in GBQA.

Figure 7: Screenshots of representative game environments within GBQA.

Room	Primary role in progression	Key objects	Mechanic stressed during QA
Hall	Start state and final exit gate	sealed door, matches, candlestick	Initial observation, pickup behavior, final win-condition verification
Corridor	Routing hub between early branches	portrait, torch bracket	Navigation consistency and branching exploration
Bedroom	Early hidden-item branch	bed, bedside drawer, diary, small key	Hidden information, container visibility, item discovery
Kitchen	Utility branch for later access	stove, bucket, ladder	Portable tool acquisition and cross-room dependency
Storage	Locked side room unlocked by bedroom key	toolbox, rope, oil lamp, key fragment B	Lock semantics, container interaction, item gating
Library	Knowledge branch before attic access	bookshelves, reading desk, scroll	Reading clues, information retrieval, ladder placement dependency

Appendix

Table 5: Room structure of the CASTLE environment.

ID	Bug type	Difficulty	Minimal reproduction	Observed fault
1	logic error	Easy	Collect any two key fragments and execute combine .	The player can assemble the complete key with only two fragments, instead of all three.
2	description flaw	Easy	Enter the bedroom and execute look before opening the bedside drawer.	The room description reveals the hidden small key before the drawer has been opened.
3	data inconsistency	Medium	Pick up any portable item, move to a room, execute drop , then execute look .	The dropped object does not appear in the updated room description, so the textual state fails to reflect the backend change.

Appendix

Table 6: Human-verified bugs dataset within the CASTLE environment.

Session	QA Findings	Claude Code Repair Outcome	Verification / Session Result	Discovery Rate / Fixing Rate
1	Newly discovered BUG-2 and BUG-3	Repair the hidden-key leakage in the bedroom description and refresh the room description after drop .	Both reported issues are patched and scheduled for QA verification in Session 2.	Discovery: 2/3 Fixing: pending verification
2	Verify BUG-2 and BUG-3 as fixed; discover and report BUG-1	Patch the fragment-combination logic after the new BUG-1 report is submitted.	QA confirms BUG-2 and BUG-3 behave normally after repair. BUG-1 remains the only unresolved defect entering Session 3.	Discovery: 3/3 Fixing: 2/3
3	Verification-focused session for BUG-1; no additional bugs reported	Correct the erroneous conditional on the “fewer than three fragments” path so key combination is allowed only with all three fragments.	QA confirms BUG-1 is fixed and no released CASTLE bug is reproduced on the targeted verification paths.	Discovery: 3/3 Fixing: 3/3
Total	BUG-001, BUG-2, and BUG-3 all discovered across the three sessions	Claude Code successfully repairs all reported bugs.	Final verification confirms that all three human-verified CASTLE bugs are fixed.	Discovery: 3/3 (100%) Fixing: 3/3 (100%)

Appendix

Table 7: Session-level trajectory of the autonomous defect discovery and remediation loop on CASTLE.

Automated Software Engineering

Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.

See all 81 papers

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
Aug 22, 2026Kun Chen, Haorong Hong, Peizhong Gao +7AI Coding Agents AI Agent Benchmarks
Hunyuan Team, Tencent · MAIS, Institute of Automation, Chinese Academy of Sciences · Lightspeed Studios, Tencent +3
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
May 26, 2026Hwiwon Lee, Jiawei Liu, Dongjun Kim +3Software Vulnerability Detection Software Security
University of Illinois Urbana-Champaign
FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs
Aug 25, 2026Ze Sheng, Aleksandar Kezic, Zhicheng Chen +1Software Engineering Benchmarks Automated Software Testing
Texas A&M University · University of Novi Sad

Aug 22, 2026J/K move · Enter open · S save

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin+6

Hunyuan Team, Tencent · MAIS, Institute of Automation, Chinese Academy of Sciences · Lightspeed Studios, Tencent+3

AI Coding Agents AI Agent Benchmarks Agent Evaluation Software Engineering Agents

Open paper AskPDF arXiv

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.

Abstract

Automated Software Engineering

Explore similar work

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs