CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Shutong Wu; Kevin Calderone; Andy Tsen

cs.AISep 19, 2026

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Authors: Shutong Wu, Kevin Calderone, Andy Tsen

Organizations: University of Rochester · RamenVR

Abstract

Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.

Figures & tables

Figure 1: Sample benchmark tasks across the three deliverable groups. The screenshots illustrate the glide task ( gp-glide-stamina-cpp ) and mud-wading task ( t1-mud-wade-bp ). The editor-scripting panel shows the kp-spawn-level-actors task, which requires creating and saving a level containing six named objects.

Figure 2: Task construction, agent authoring, and evaluation of the saved submission in a fresh project. Each task declares its required checks. Reference-pass and unchanged-project-fail are admission criteria; § 3.5 describes task authoring and validation.

Task group	Tasks	Build	Runtime	Assets
C++	33	33	33	0
BP	25	25	15	20
Python	12	12	0	12

Table 1: Required checks in the measured suite. Runtime and asset columns overlap; each paired BP gameplay task requires both.

Configuration	C++ tasks	BP and Python tasks
File/shell baseline	Read, write, edit, shell; no editor MCP	Not evaluated
MCP1	File/shell tools and editor MCP	File/shell tools and editor MCP
MCP2	File/shell tools and editor MCP	Read/search and editor MCP; no native write/edit/shell

Table 2: Agent tool permissions. Both editor configurations can perform several kinds of project edits; tool availability is part of the evaluated configuration.

Configuration	C++ completion	BP completion
File/shell baseline	45/70 (64.3%)	—
MCP1	46/70 (65.7%)	25/70 (35.7%)
MCP2	50/70 (71.4%)	20/70 (28.6%)

Table 3: Completion within 40 minutes on the same ten gameplay tasks. Seven models run each task, giving 70 runs per evaluated configuration and variant.

Figure 3: C++ and BP completions within 40 minutes on the ten paired gameplay tasks. Each line connects the same model’s totals under one MCP configuration, with one attempt per task variant.

Observed method in BP runs	MCP1	MCP2
Task-asset readback	149/175	165/175
Compilation request	131/175	132/175
PIE start request	38/175	44/175
Image-capture request	15/175	10/175
PIE state readback	24/175	30/175

Table 4: Checking actions in BP trajectories, counted over full authoring sessions. Rows overlap. Request counts include unsuccessful attempts; readback counts require returned values. PIE starts include tool calls and starts within scripts. Image requests can capture either the editor or gameplay. Appendix D.1 defines the categories and their coverage.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Archived development/validation traces for the glide task, separate from the study’s agent runs. In the shaded post-request interval, the passing trace slows descent while Power drains; the failing trace drains Power without slowing descent. The plotted values and verdict labels are transcribed from a retained development capture. The Manny screenshot is a separate scene illustration, not a frame from either trace.

Display label	Requested model identifier
Claude Sonnet 5	claude-sonnet-5
DeepSeek V4 Pro	deepseek/deepseek-v4-pro-0813
Gemini 3.7 Flash	google/gemini-3.7-flash
GPT-5.6 Luna	openai/gpt-5.6-luna
GPT-5.6 Sol	openai/gpt-5.6-sol
Grok 4.6	x-ai/grok-4.6

Appendix

Table 5: Requested agent model identifiers.

C++
Model	File/shell baseline	MCP1	MCP2
Claude Sonnet 5	27/33 (81.8%)	30/33 (90.9%)	30/33 (90.9%)
DeepSeek V4 Pro	26/33 (78.8%)	28/33 (84.8%)	27/33 (81.8%)
Gemini 3.7 Flash	23/33 (69.7%)	29/33 (87.9%)	31/33 (93.9%)
GPT-5.6 Luna	21/33 (63.6%)	29/33 (87.9%)	27/33 (81.8%)
GPT-5.6 Sol	29/33 (87.9%)	30/33 (90.9%)	30/33 (90.9%)

Appendix

Table 6: Full-task completions within 40 minutes, with counts and percentages shown separately for each surface. The file/shell baseline is evaluated only on C++.

Figure 5: Task completion within 40 minutes, by model and configuration. Each model has three stacked bars: Baseline, MCP1, and MCP2. Colored segment lengths indicate the counts of completed C++, Blueprint, and Python tasks. Labels inside each segment give completed tasks out of 33 C++, 25 Blueprint, or 12 Python tasks. The baseline uses file/shell tools and was evaluated only on C++, so its total bar length is not comparable with those of the MCP configurations. Table 6 gives the same counts.

Model	MCP1		MCP2
	C++	BP	C++	BP
Claude Sonnet 5	8/10	5/10	8/10	3/10
DeepSeek V4 Pro	6/10	5/10	6/10	2/10
Gemini 3.7 Flash	6/10	5/10	8/10	3/10
GPT-5.6 Luna	6/10	1/10	6/10	4/10
GPT-5.6 Sol	8/10	6/10	8/10	4/10

Appendix

Table 7: Per-model completion within 40 minutes on the ten paired gameplay specifications.

C++ task subset	Baseline	MCP1	MCP2
Ten paired gameplay tasks	45/70	46/70	50/70
Other 23 C++ tasks	135/161	153/161	153/161

Appendix

Table 8: C++ completion within 40 minutes, split by whether the specification also has a BP version. The split follows the task definitions. It is not selected from outcomes.

Figure 6: BP completion by task type. Row labels give the number of tasks and runs per configuration (seven models per task). Groups follow the task instructions and differ in content, starting state, and required checks.

Configuration	C++ full	BP full	BP runtime pass	Gap (pp)
MCP1	46/70	25/70	25/70	30.0
MCP2	50/70	20/70	20/70	42.9

Appendix

Table 9: Completion on the ten paired gameplay tasks, with and without the BP asset checks. Each task group has 70 attempts per configuration. All counts require authoring to finish within 40 minutes. Counting recorded BP runtime passes leaves the full-task counts and the C++–BP gap unchanged; the original submissions are used in both counts. Gap is measured in percentage points.

	PIE start request		PIE state readback		Image return
Model	MCP1	MCP2	MCP1	MCP2	MCP1	MCP2
Sonnet	17	18	12	11	1	5
DeepSeek	4	6	2	4	2	0
Gemini	2	4	1	3	0	0
Luna	0	2	0	1	0	0
Sol	0	0	0	0	0	0

Appendix

Table 10: BP inspection methods by model, out of 25 runs per configuration. PIE start requests include identified scripts. PIE state readbacks require returned measurements from the running game. An image must appear in the agent’s conversation. Rows use the definitions in Appendix D.1 .

Observed sequence	All (95)	Same tasks (49)	$\leq 40$ min (79)
Accepted write, then changed same-graph write	30	16	20
Readback between those writes	27	15	17
Compilation return between those writes	21	11	12
Both readback and compilation return	18	10	9
Error, then immediate same-graph change	37	20	26
Acceptance, then immediate same-graph change	4	1	4

Appendix

Table 11: Graph revisions in MCP1 BP runs. Columns cover all direct graph-writing runs, the seven tasks on which every model directly writes graphs, and sessions ending within 40 minutes. Rows overlap. A compilation return need not report successful compilation. Immediate means the next agent tool request after the preceding write returns; 58 runs contain errors, and 30 revise accepted writes.

Sequence	Runs	On-time pass	Final pass
Uses direct graph writing	95	55	62
Receives graph-writing error	58	27	34
Changed same-graph request accepted	55	26	32
Every error has a later accepted change	50	23	29
Previous row, session ends within 40 min	39	23	23

Appendix

Table 12: Graph-writing errors and subsequent revisions in MCP1 BP runs. Rows successively restrict the 175-run panel. Accepted revisions return without a reported tool error. On-time passes finish authoring within 40 minutes; final passes include later submissions.

Failure after passing asset checks	MCP1	MCP2
Initial Health or Power	3	4
Required effect or resource use	8	6
Pose, display, or object indicator	5	8
Restoration or preserved behavior	5	6
Poison damage ratio	3	4
Total attempts	24/70	28/70

Appendix

Table 13: BP submissions that pass asset checks but fail a runtime check. Each of the 52 selected attempts has one observed failure category; subsequent checks may not execute if a previous check failed.

Selected BP submissions	MCP1	MCP2
On-time submissions passing asset checks	45	40
Runtime checks pass	25	20
Explicit runtime assertion failure	19/45 (42.2%)	20/40 (50.0%)
Runtime automation failure	1	0

Appendix

Table 14: Runtime outcomes among BP submissions on the ten gameplay specifications that pass asset checks and finish authoring within 40 minutes. The last three rows partition this selected subset; automation failures are separate from failed assertions.

Figure 7: C++ completion and median authoring time on the same 33 tasks per configuration. Markers show median duration on a logarithmic scale; labels give completion counts. All runs enter the duration calculation.

Model	Pairs	MCP1	MCP2	Charge ratio	Higher
Claude Sonnet 5	0/33	—	—	—	—
DeepSeek V4 Pro	20/33	18/20	19/20	0.21 $\times$	6/20
Gemini 3.7 Flash	32/33	28/32	31/32	1.86 $\times$	23/32
GPT-5.6 Luna	29/33	27/29	25/29	2.48 $\times$	29/29
GPT-5.6 Sol	33/33	30/33	30/33	2.69 $\times$	33/33
Grok 4.6	31/33	28/31	29/31	1.80 $\times$	29/31

Appendix

Table 15: C++ completion and billed inference on the matched subset. Completion columns count passes within 40 minutes. Charge ratio is the median paired MCP2/MCP1 ratio; Higher counts pairs with larger MCP2 charges. Pair counts give billing coverage out of 33 tasks per model.

Row	Task ID	Starting project	Build	Runtime	Inspect
C++ source tasks
C01	gp-crafting-queue	CraftBench template	$\checkmark$	$\checkmark$	—
C02	gp-dot-aoe-burn-cpp	Third Person	$\checkmark$	$\checkmark$	—
C03	gp-double-jump-stamina-cpp	Third Person	$\checkmark$	$\checkmark$	—
C04	gp-glide-stamina-cpp	Third Person	$\checkmark$	$\checkmark$	—
C05	gp-harvestable-regrow	CraftBench template	$\checkmark$	$\checkmark$	—

Appendix

Table 16: The 70 study tasks, with their starting projects and required grading gates. IDs refer to task directories. Build, runtime, and inspect denote L1, L2, and L2I;

\checkmark

marks a required check and — marks one not required. The C++, BP, and Python groups contain 33, 25, and 12 tasks.

Task family	Requested behavior	Catalog rows
Area burn	Apply periodic damage inside a bounded area and stop after its duration.	C02 / B03
Double jump	Reverse a fall with an upward impulse; debit Power once and reject insufficient Power.	C03 / B04
Glide	Slow descent while draining Power and restore normal falling when Power is exhausted.	C04 / B05
Healing over time	Restore Health repeatedly for a limited duration without exceeding MaxHealth.	C06 / B06
Health operations	Initialize writable Health and provide repeatable damage and healing abilities.	C07 / B07
Stacking poison	Apply periodic damage with up to three stacks and refresh the duration on reapplication.	C09 / B08

Appendix

Table 17: The ten gameplay specifications with C++ and Blueprint counterparts. Each pair names the same starting project, runtime map, and test class in its task definitions. Both counterparts require build and runtime checks; the Blueprint counterpart also requires artifact inspection. These summaries describe the requested behavior; the task prompts specify the acceptance tolerances and deliverables.

Benchmark	Setting and task source	Agent’s deliverable	Grading evidence
GameDevBench ( Chi et al., 2026 )	Godot; 333 tasks from web/video tutorials	Scripts, scenes, resources	In-engine tests of behavior and scene state
GameEngineBench ( La et al., 2026 )	Unreal; 110 tasks from nine projects	C++ changes in existing projects	Build and runtime tests, followed by an LLM behavioral judge
OpenGameEval ( Zhang et al., 2025 )	Roblox; expert-authored tasks (47 in the initial release)	Scripts and changes to existing experiences	Executable tests in Roblox Studio
GBQA ( Jiang et al., 2026 )	Web games; 124 human-verified injected bugs in 30 agent-built games	Bug reports from interactive exploration	LLM critic matches reports to ground truth; recall of verified bugs
CutsceneBench ( He et al., 2026 )	Unreal; 65 authored cutscene scenarios	Level Sequence assets	Tool and sequence checks; model-judged quality on 25 sampled videos per model
AutoUE ( Yin et al., 2026 )	Unreal; complete game generation	Game projects with scenes and gameplay	Automated play-testing; LLM-judged game quality

Appendix

Table 18: Benchmark designs, using the cited releases. The OpenGameEval count refers to its initial release; its task bank has since expanded. CraftBench-UE also includes 10 C++/BP gameplay pairs and evaluates two editor MCP configurations.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

Jul 3, 2026

GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.

Brian La, Sejoon Chang, Ben Kim +5

Nitrode · Nexon Intelligence Labs · Dartmouth College +3

Coding Agents Software Engineering Benchmarks Code Generation

Ask PDF

Oct 19, 2026

From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

A game ships when a player can launch it, finish it, and return to a saved world, none of which a successful build guarantees, yet game-creation agents are mostly tested on one engine or on one-prompt browser games. We present CEGame-Bench, the first long-horizon benchmark for complete-game creation across five production engines: Defold, Godot, Roblox Studio, Unity, and Unreal Engine. Agents build each game from an empty project and a design document in runs averaging up to 295 steps and 65M tokens; the document doubles as a contract, and a hidden evaluator accepts a game only by real-input play. Under four scaffolds, agents earn most integration points but usually under half the gameplay points, and their games reach the ending in only 16% of runs, while references built by engineers with coding agents score near the maximum, which places this artifact-to-play gap in the agents. No two engines rank configurations alike (Kendall's tau 0.14-0.62), and the same game usually scores higher on Unity and Unreal Engine, which check more before play. We release Game Completion Boost, an agent skill for requirement-driven playtesting and repair that raises the execution-based score of three models by 14.1-24.3 points.

Zixuan Dong, Yanghai Wang, Hanyan Bian +8

National University of Defense Technology · Nanjing University · The University of Tokyo +2

Automated Game Testing Software Engineering Agents Long-Horizon Agent Evaluation

Ask PDF

Jun 16, 2026

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22

1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5

AI Agent Benchmarks Agentic Code Generation Code Generation Evaluation

Ask PDF

Abstract

Coding Agents

Explore similar work

GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?