UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

Yijun Zhou; Minghao Shao; Ke Zhang; Renkai Yuan; Hongbo Yuan; Bin Hu; Bozhao Qian; Hongxia Shen

cs.SEOct 26, 2026

UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

Authors: Yijun Zhou, Minghao Shao, Ke Zhang, Renkai Yuan, Hongbo Yuan, Bin Hu, Bozhao Qian, Hongxia Shen,

Organizations: Unity China · New York University · University of Chinese Academy of Sciences · Xiamen University ·

Abstract

Game development stresses coding agents in ways general software engineering does not: correctness depends on how engine APIs, serialized scene state, and runtime behavior interact, not on source code alone. Existing benchmarks mainly target platforms such as Godot and heavily rely on LLM judges or coarse task-level outcomes. We present UnityBench-Mini3D, a benchmark of 77 project-level Unity tasks organized around seven expert-defined skill categories rather than tutorials or genres, with three difficulty tiers. Each task pairs a complete Unity project with natural-language requirements and is evaluated using 1,178 held-out, human-authored milestone tests covering scene structure, component configuration, and runtime behavior. This in-engine evaluation provides deterministic partial credit, localizes failures, and distinguishes partial implementations from complete solutions without relying on subjective judges. Across 21 models and 9 coding harnesses, the average pass rate reaches 80.9% on easy tasks but falls to 35.2% on hard tasks, with just 57.7% overall. UnityBench-Mini3D provides a reproducible, diagnostic testbed distinguishing systems across difficulty levels.

Figures & tables

Benchmark	Engine	Src.	Scope	Tk./Ts.		D./P.		Cat.
V-GameGym [17]	Pygame	Repos.	File	2,219 /	–	✗ /	✗	100 (C)
WebGame [18]	Browser	Web/user	Proj.	111 /	–	✗ /	✗	7 (G)
GameDev [4]	Godot	Tutorials	Proj.	333 /	–	✓ /	✗	4 (S)
GameCraft [12]	Godot	Human	Proj.	140 /	–	✗ /	✓	15 (G)
JamBench [13]	Godot	Jams	Proj.	300 /	–	✓ /	✗	–
GameLogic [1]	Godot	Repos./Games	Proj.	72 /	1,451	✓ /	✗	7 (S)

Table 1: Game development benchmark comparison. Src.: task source; Tk./Ts.: task and test counts; D./P.: judge-free deterministic evaluation and partial scoring; Cat.: clustered ( C ), genre-based ( G ), skill-based ( S ), or pattern-based ( P ).

Category	# Tasks	# Tests	Skills exercised	Primary requirement
Scene construction	19	365	Transforms, hierarchy, prefabs, terrain, cameras, NavMesh	Assemble and organize scene content under spatial, hierarchy, and navigation constraints.
Animation	12	179	Animation clips, Animator, events, layers, Avatar Masks, IK, Playables	Configure animation assets and controllers to produce coordinated runtime motion and character poses.
Particles & shaders	12	155	Particle System, VFX Graph, Shader Graph, shaders, materials	Construct and configure visual effects, and implement or adapt shaders and materials to the required behavior and rendering pipeline.
Physics	11	168	Rigidbodies, colliders, physics materials, joints, WheelCollider, Cloth, procedural meshes	Implement and configure physical motion, collisions, and interactions under specified runtime constraints.
User interface	9	111	UGUI, Canvas, RectTransform, UI Toolkit, Sprite Editor, RenderTexture	Prepare UI assets and implement layouts, input handling, and visual feedback.
Editor operations	11	142	Hierarchy organization, AssetDatabase, importers, Undo, project settings, ScriptableObject, audio integration	Organize and configure scenes, assets, and project settings, and integrate functionality while preserving existing references and behavior.

Table 2: UnityBench-Mini3D task taxonomy, example tasks, and required skills. The bottom row shows one representative task from each core category (except for Miscellaneous).

Figure 1: UnityBench-Mini3D overview. Agents build fully functional projects from initial challenge templates, with final project states evaluated by milestone tests.

Panel of Table 3 — Table 3: Performance of normal coding and CUA harnesses on UnityBench-Mini3D. O.a. is the overall pass rate and C. and T. denote average per-task API cost and wall-clock time.

Figure 2: Tool usage. Panel (b) includes only models evaluated with at least two harnesses; values on the right report average calls per task.

Configuration			Task category (%)								Difficulty (%)			Efficiency
Setting	Harness	Model	UI	Anm.	Edtr.	Scn.	P.&S.	Phys.	Misc.	O.a. (%)	Easy	Med.	Hard	C. ($)	T. (m)
w/o [-0.35ex] MM	OpenCode	Kimi K3	60.46 $\uparrow\!11.77$	69.32 $\uparrow\!6.49$	65.30 $\uparrow\!5.62$	52.91 $\downarrow\!3.44$	83.16 $\uparrow\!3.80$	62.72 $\downarrow\!2.99$	31.99 $\downarrow\!26.81$	63.42 $\uparrow\!1.46$	91.51 $\uparrow\!6.88$	64.83 $\downarrow\!2.83$	31.11 $\uparrow\!0.98$	13.64 $\downarrow\!3.35$	34.78 $\uparrow\!3.81$
	Codex	GPT 5.6 Sol	65.19 $\downarrow\!0.61$	73.38 $\downarrow\!1.27$	88.08 $\uparrow\!15.51$	61.99 $\uparrow\!0.30$	87.14 $\downarrow\!2.90$	74.66 $\downarrow\!4.08$	58.43 $\downarrow\!0.37$	73.46 $\uparrow\!0.97$	92.31 $\uparrow\!2.06$	67.33 $\uparrow\!0.36$	60.69 $\uparrow\!0.56$	18.46 $\uparrow\!6.42$	14.90 $\uparrow\!4.19$
	Codely	Kimi K3	65.24 $\uparrow\!5.78$	72.24 $\uparrow\!2.25$	78.35 $\uparrow\!5.61$	50.57 $\downarrow\!1.99$	83.16 $\uparrow\!0.63$	73.02 $\uparrow\!3.29$	55.10 $\downarrow\!0.28$	68.09 $\uparrow\!1.89$	94.10 $\uparrow\!5.78$	68.32 $\downarrow\!0.28$	39.53 $\uparrow\!0.40$	16.64 $\uparrow\!0.65$	34.37 $\uparrow\!3.00$
w/o [-0.35ex] MCP	Codely	GPT 5.6 Sol	51.32 $\uparrow\!6.04$	62.55 $\uparrow\!1.96$	73.37 $\uparrow\!5.69$	44.01 $\downarrow\!6.80$	58.33 $\downarrow\!24.73$	64.45 $\downarrow\!7.70$	29.63 $\downarrow\!25.19$	56.54 $\downarrow\!5.79$	80.42 $\downarrow\!4.15$	53.40 $\downarrow\!5.78$	34.54 $\downarrow\!7.59$	25.44 $\downarrow\!6.30$	26.42 $\uparrow\!12.25$
		GLM 5.2	56.84 $\downarrow\!7.01$	55.44 $\downarrow\!10.67$	71.07 $\downarrow\!6.34$	33.26 $\downarrow\!9.06$	67.13 $\downarrow\!9.28$	62.20 $\downarrow\!8.78$	51.77 $\uparrow\!22.60$	55.01 $\downarrow\!7.44$	82.66 $\downarrow\!1.44$	51.60 $\downarrow\!11.75$	29.26 $\downarrow\!8.54$	11.31 $\uparrow\!0.96$	24.64 $\uparrow\!5.73$

Table 5: Ablation study. Results cover multimodal input, engine tools, and CUA;

\uparrow

\downarrow

and colors indicate improvement or degradation relative to the corresponding baseline.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

CHALLENGE LIST 77 tasks · 7 skill categories
User Interface 9 tasks · 111 milestone tests
0025-FPSCompassHUDCalibration Easy 6 tests Compass HUD Direction Calibration Initial Challenge Template Completed Golden Reference
0050-UIToolkitSnake Easy 23 tests Snake Game Implemented with Unity UI Toolkit Initial Challenge Template Completed Golden Reference
0063-MinimapRealization Easy 7 tests Minimap UI Implementation Initial Challenge Template Completed Golden Reference
0010-SplitTheUI Medium 11 tests UI Sprite Sheet Slicing Initial Challenge Template Completed Golden Reference
0023-UIRedoBench Medium 9 tests GoldenBench UI Recreation Initial Challenge Template Completed Golden Reference

Appendix

Table 6: Challenge list. The 77 UnityBench-Mini3D tasks, grouped by skill category. Each entry identifies the initial project and the corresponding completed reference project; difficulty and held-out milestone-test count are shown in the header.

SPECIFICATIONS goal · context · constraints · resources
User Interface 9 tasks · 111 milestone tests
0025-FPSCompassHUDCalibration Easy 6 tests Compass HUD Direction Calibration
Goal	Fix the direction display of the compass HUD in the provided CompassHUD.cs.
Context	CompassHUD already contains a compass viewport, repeating texture, and scrolling logic, but its direction mapping is incorrect.
Constraints	• Only modify the direction calculation and the NorthCalibrationDegrees calibration value in CompassHUD.cs. Do not modify the compass UI creation logic, texture repetition method, stripRoot.anchoredPosition displacement formula, or smooth scrolling logic.
	• When the player’s view faces world Z+, the compass center line should align with N.

Appendix

Table 7: Task specifications. Agent-visible natural-language specifications for all 77 tasks. Each specification separates the requested goal, initial context, 780 explicit constraints, and relevant Unity resources.

TEST CASES 1,178 independent pass/fail checks
User Interface 9 tasks · 111 milestone tests
0025-FPSCompassHUDCalibration Easy 6 tests Compass HUD Direction Calibration
Test 01	RuntimeHeading_Earns1Of6_WhenCardinalDirectionsAreWithin6Degrees Checks that runtime behavior heading earns 1 of 6 when cardinal directions are within 6 degrees.
Test 02	RuntimeHeading_Earns2Of6_WhenCardinalDirectionsAreWithin5Degrees Checks that runtime behavior heading earns 2 of 6 when cardinal directions are within 5 degrees.
Test 03	RuntimeHeading_Earns3Of6_WhenCardinalDirectionsAreWithin4Degrees Checks that runtime behavior heading earns 3 of 6 when cardinal directions are within 4 degrees.
Test 04	RuntimeHeading_Earns4Of6_WhenCardinalDirectionsAreWithin3Degrees Checks that runtime behavior heading earns 4 of 6 when cardinal directions are within 3 degrees.

Appendix

Table 8: Milestone test cases. The 1,178 held-out, human-authored checks used to evaluate the 77 tasks. Every row names one independently scored test and summarizes the property it verifies.

Figure 3: Task difficulty and discrimination. Each point is one of the 77 tasks. The horizontal axis is its mean score across 32 configurations. The vertical axis is the score gap between the top and bottom 16 configurations, where configuration strength is recomputed as the task-equal mean over the other 76 tasks to avoid circularity. Marker size reflects cross-configuration variation, color denotes nominal difficulty, and shaded regions indicate potential floor and ceiling effects. This is a descriptive benchmark diagnostic; configurations sharing a harness are not independent samples.

Figure 4: Task-pair correlation and redundancy. Panel (a) compares raw Spearman correlations for all 2,926 task pairs with descriptive correlations adjusted for each configuration’s task-equal performance on the remaining 75 tasks. Panel (b) shows the five positive, non-floor/ceiling pairs whose raw correlations exceed the family-wise threshold estimated from 50,000 whole-matrix permutations.

Figure 5: Failure composition by benchmark skill. Each row gives the distribution of non-passing milestone outcomes within one skill and sums to 100%. The rightmost values report milestone counts rather than task runs. C1–C5 follow the failure taxonomy defined in the main paper.

Figure 6: Failure composition by task difficulty. Each row reports the distribution of non-passing milestone outcomes pooled across the 32 baseline configurations and sums to 100%; the value at right is the corresponding milestone count. C1–C5 follow the taxonomy defined in the main paper. Because each row is conditioned on non-passing outcomes, the figure compares failure composition rather than overall failure frequency.

Figure 7: Tool-use profiles across task difficulty. The value at right reports mean calls per task, and each stacked bar reports the pooled composition of those calls across the 32 baseline configurations. Percentages sum to 100% within each difficulty level;

n

denotes the number of benchmark tasks.

Figure 8: Leading-tier stability under task resampling. The left panel reports task-equal overall scores and 95% intervals for the leading 14 configurations; the right panel reports bootstrap frequencies of ranking first or within the top three, together with median rank. The 100,000 paired, category-stratified resamples preserve observed category counts and equal task weights; they capture benchmark-composition rather than run-to-run uncertainty.

Figure 9: Leaderboard robustness across scoring rules. The matrix reports Spearman rank correlations across five scoring rules and 32 configurations. Blue segments span ranks under the four partial-credit rules; navy and orange markers indicate task-equal headline and task-equal binary full-pass ranks, respectively. Span is the best-to-worst rank range across all five rules.

Figure 10: Observed score–cost trade-off. Points represent the 32 baseline model–harness configurations; ablations are excluded. Blue points lie on the empirical point-estimate frontier: no observed configuration has both a no-higher estimated API cost and a no-lower task-equal overall score, with at least one strict improvement. Gray points show the remaining configurations. The line connects observed frontier configurations and does not represent a fitted relationship. Costs and scores are point estimates from this evaluation; the figure does not incorporate score uncertainty or non-API deployment costs.

Figure 23: Breadth of the 77 task workspaces. The benchmark ranges from compact scenes to asset-rich Unity projects. The right panel summarizes the per-task distributions of files, disk footprint, file types, code, scene nodes, and images; each row uses an independent log scale, and colored marks show the interquartile range, median, and largest workspace.

Figure 24: Coverage of selected Unity project artifacts. Bars report the number and percentage of task workspaces containing each non-exclusive artifact type. Scene files combine .scene and .unity ; other composite modalities are unions of the corresponding extensions listed in the released profiling data. Each task is counted at most once per row.

Harness: Codely CLI version: 1.0.0-nightly.51
Tool	Description
activate_skill	Activates a specialized agent skill by name.
analyze_multimedia	Analyzes images, videos, and audio files using a multimodal model.
apply_patch	Apply a structured patch document to edit files.
cron_create	Schedule a prompt to be enqueued at a future time.
cron_delete	Cancel a cron job previously scheduled with cron_create.

Appendix

Table 9: Tool availability for Codely. Tools exposed to the benchmark main agent.

Harness: Codex CLI CLI version: 0.147.0
Tool	Description
apply_patch	Apply a structured patch to create, modify, or delete files.
current_time	Retrieve the current time when the current-time capability is enabled.
exec_command	Execute shell commands in the local execution environment.
followup_task	Create a follow-up task for a sub-agent.
get_context_remaining	Retrieve information about the remaining context budget.

Appendix

Table 10: Tool availability for Codex CLI. Tools exposed to the benchmark main agent.

Harness: Claude Code CLI version: 2.1.233 Models: Claude Opus 5 and Claude Sonnet 5
Tool	Description
Task	Launch or resume a subagent for a delegated task.
Bash	Execute shell commands in the current workspace.
CronCreate	Create a one-shot or recurring scheduled job.
CronDelete	Delete a scheduled job by its identifier.
CronList	List scheduled jobs in the current session.

Appendix

Table 11: Tool availability for Claude Code. Tools exposed to the benchmark main agent.

Harness: Grok Build CLI version: 1.0.5
Tool	Description
ask_user_question	Ask the user structured multiple-choice questions.
enter_plan_mode	Enter a read-only planning phase before implementation.
exit_plan_mode	Present a completed plan for user approval and leave plan mode.
get_command_or_subagent_output	Retrieve status and output from a background command, monitor, or subagent.
grep	Search file contents with regular expressions.

Appendix

Table 12: Tool availability for Grok Build. Tools exposed to the benchmark main agent.

Harness: DeepSeek Harness CLI version: 0.1.1-rc.2
Tool	Description
create_goal	Create a persistent same-session goal for autonomous continuation.
edit	Replace exact literal text in an existing UTF-8 file.
exit_plan_mode	Present a complete plan for user review and approval.
get_goal	Read the current goal, revision, phase, limits, and blockers.
glob	Find file paths that match a glob pattern.

Appendix

Table 13: Tool availability for DeepSeek Harness. Tools exposed to the benchmark main agent by the default Windows headless profile.

Harness: OpenCode CLI version: 1.18.23
Tool	Description
apply_patch	Apply patches to files.
bash	Execute shell commands in your project environment.
edit	Modify existing files using exact string replacements.
glob	Find files by pattern matching.
grep	Search file contents using regular expressions.

Appendix

Table 14: Tool availability for OpenCode. Tools exposed to the benchmark main agent.

Harness: Qwen Code CLI version: 0.21.13 / 0.21.14
Tool	Description
agent	Use agent to launch a specialized subagent to handle complex, multi-step tasks autonomously.
computer_use__kill_app	Terminate a desktop application by process ID.
computer_use__list_apps	List running and installed desktop applications with their process IDs.
computer_use__list_windows	List the on-screen windows of a desktop application.
edit	Edit replaces text within a file.

Appendix

Table 15: Tool availability for Qwen Code. Tools exposed to the benchmark main agent.

Harness: Antigravity CLI version: 1.1.19
Tool	Description
find_by_name	Find files and directories by name pattern, with extension, type, depth, and exclude filters.
grep_search	Search file contents for a query or regular expression, with case-insensitive and file-include filters.
list_dir	List the contents of a directory.
manage_task	Inspect and control background tasks: list them, query status, or kill by task ID.
read_url_content	Fetch and read the content of a URL.

Appendix

Table 16: Tool availability for Antigravity. Tools exposed to the benchmark main agent.

Harness: Kimi Code CLI version: 0.36.1
Tool	Description
Agent	Launch or resume one subagent for a delegated task.
AgentSwarm	Launch or resume multiple subagents in parallel.
AskUserQuestion	Ask the user structured questions with selectable options.
Bash	Execute shell commands in the foreground or background.
CreateGoal	Create a durable goal that persists across turns.

Appendix

Table 17: Tool availability for Kimi Code. Tools exposed to the benchmark main agent.

Coding Agent Benchmarks

See all 47 papers

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Sep 19, 2026Shutong Wu, Kevin Calderone, Andy TsenCoding Agents AI Agent Evaluation
University of Rochester · RamenVR
Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes
Sep 29, 2026Xiaokang Ye, Siddhant Hitesh Mantri, Zimeng Chen +6AI Coding Agents 3D Scene Editing
UC San Diego · UC Berkeley
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Multimodal Agent Evaluation Software Engineering Agents
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA

Sep 19, 2026J/K move · Enter open · S save

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Shutong Wu, Kevin Calderone, Andy Tsen

University of Rochester · RamenVR

Coding Agents AI Agent Evaluation Code Generation Benchmarks Benchmark Design

Open paper AskPDF arXiv

Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.

Abstract

Coding Agent Benchmarks

Explore similar work

CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

GameDevBench: Evaluating Agentic Capabilities Through Game Development