Grounding Machine Creativity in Game Design Knowledge Representations: Empirical Probing of LLM-Based Executable Synthesis of Goal Playable Patterns under Structural Constraints

Hugh Xuechen Liu; Kıvanç Tatar

cs.AIMar 7, 2026

Grounding Machine Creativity in Game Design Knowledge Representations: Empirical Probing of LLM-Based Executable Synthesis of Goal Playable Patterns under Structural Constraints

Authors: Hugh Xuechen Liu, Kıvanç Tatar

Organizations: Chalmers University of Technology and University of Gothenburg

Abstract

Creatively translating complex gameplay ideas into executable artifacts (e.g., games as Unity projects and code) remains a central challenge in computational game creativity. Gameplay design patterns provide a structured representation for describing gameplay phenomena, enabling designers to decompose high-level ideas into entities, constraints, and rule-driven dynamics. Among them, goal patterns formalize common player-objective relationships. Goal Playable Concepts (GPCs) operationalize these abstractions as playable Unity engine implementations, supporting experiential exploration and compositional gameplay design. We frame scalable playable pattern realization as a problem of constrained executable creative synthesis: generated artifacts must satisfy Unity's syntactic and architectural requirements while preserving the semantic gameplay meanings encoded in goal patterns. This dual constraint limits scalability. Therefore, we investigate whether contemporary large language models (LLMs) can perform such synthesis under engine-level structural constraints and generate Unity code (as games) structured and conditioned by goal playable patterns. Using 26 goal pattern instantiations, we compare a direct generation baseline (natural language -> C# -> Unity) with pipelines conditioned on a human-authored Unity-specific intermediate representation (IR), across three IR configurations and two open-source models (DeepSeek-Coder-V2-Lite-Instruct and Qwen2.5-Coder-7B-Instruct). Compilation success is evaluated via automated Unity replay. We propose grounding and hygiene failure modes, identifying structural and project-level grounding as primary bottlenecks.

Figures & tables

Panel of Figure 1 — Figure 1: Baseline and IR-conditioned pipelines. Data (white), LLM (light grey), compile target (dark grey), schema injection (dashed). Both share the same Unity batch-replay harness and error-log aggregation.

Configuration	Target	Additional prompt content (beyond shared header and <PATTERN_MD> )
no_schema	C#	Direct Unity Editor C# generation; “Output only raw C# code.”
with_schema_free	IR JSON $\to$ C#	IR step: no schema constraints; “Output ONLY valid JSON.”
with_schema_min	IR JSON $\to$ C#	IR step: free + skeleton listing the 7 required top-level fields with shape annotations.
with_schema_full	IR JSON $\to$ C#	IR step: min + type annotations + 4 hard referential-integrity constraints enumerated inline.

Table 1: Prompt differences across configurations. All four share a header ( [pattern: <PATTERN_ID>] , [method: <METHOD>] ) and ingest <PATTERN_MD> . Full verbatim templates: Appendix: Prompt Templates (Verbatim) .

Code	Compiler message	NS	Free	Min	Full	Total
CS0029	Cannot implicitly convert X (e.g., string ) to Y (e.g., string[] )	2	1	0	0	3
CS0101	Namespace already contains a definition for X (e.g., PlayerController )	25	18	15	22	80
CS0103	The name X (e.g., whatToConceal ) does not exist in the current context	14	16	14	15	59
CS0111	Member X (e.g., Update ) already defined with same parameter types	16	14	9	18	57
CS0115	X (e.g., EvadeGoal.IsCompleted() ) is not a suitable method for override	40	0	0	0	40
CS0116	Namespace cannot directly contain members such as fields or methods	0	0	0	1	1

Table 2: All observed C# compiler error codes, message templates, and log count by configuration. G = grounding failure; H = hygiene failure. NS = no schema. Placeholders X , Y , T stand for observed symbols in the raw compiler output. Representative examples from our logs include pattern-named types ( EvadeGoal , GuardGoal , RescueGoal.Rescue() ) which appear exclusively in G rows as types hallucinated by the model. Other representative examples include script members ( currentCount , interferableGoal ), namespace paths ( UnityEngine.Animations ), and primitive/engine types ( string , Vector2ArrayField ) which occur in both G and H rows depending on the error class.

		Qwen				DeepSeek
Class	Code	NS	Free	Min	Full	NS	Free	Min	Full
G	CS0115	20	0	0	0	20	0	0	0
	CS0117	2	2	0	1	0	0	0	0
	CS0122	0	0	0	0	17	0	0	0
	CS0234	20	0	0	0	0	0	0	0
	CS0239	4	0	0	0	0	0	0	0

Table 3: Per-model log_count by error code and configuration. G = grounding failure; H = hygiene failure. NS = no_schema, Free = with_schema_free, Min = with_schema_min, Full = with_schema_full.

Pattern	Tot	G	H	Top G	Tot	G	H	Top G	Tot	G	H	Top G	Tot	G	H	Top G
	no_schema				with_schema_free				with_schema_min				with_schema_full
1_Ownership	18	0	18	—	31	9	22	CS0246(6), CS1061(3)	71	14	57	CS0246(11), CS1061(3)	49	16	33	CS0246(15), CS0117(1)
2_Collection	23	8	15	CS0115(2), CS0234(2)	47	0	47	—	50	0	50	—	47	8	39	CS0246(6), CS1061(2)
3_Eliminate	37	0	37	—	46	2	44	CS0117(2)	43	7	36	CS0246(4), CS8121(2)	43	0	43	—
4_Capture	41	27	14	CS0234(16), CS0115(11)	47	0	47	—	38	0	38	—	35	0	35	—
5_Overcome	5	1	4	CS0234(1)	25	0	25	—	44	15	29	CS0246(7), CS1061(5)	54	10	44	CS0246(5), CS1061(5)

Table 4: Pattern-level errors by configuration (both models combined, 40 logs per pattern). G = grounding failure; H = hygiene failure. Timeout rates: NS 37–51%, Free 86–92%, Min 96%, Full 97–99%.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

#	Iteration	What Changed	Why	Impact
1	v0 static draft	Defined six top-level fields: scene , objects , scripts , params , links , rules .	Minimal structured representation between pattern description and Unity code generation.	Established baseline schema consumed by all pipeline stages.
2	MVP narrowing	Deferred params extraction; pipeline operates on objects , scripts , links , rules only. params emitted as {} .	params requires GUID resolution (scene $\to$ prefab $\to$ script $\to$ .cs ), blocking the initial pipeline.	Unblocked end-to-end generation without serialized-field extraction.
3	Runtime extension	Added PrefabInstance / PrefabAsset object types, script-defined rules , and runtime_params field.	Static .unity parsing misses prefab-driven gameplay; core behaviour emerges from prefab instantiation and runtime script logic.	IR captures the actual gameplay loop; generation can reason about spawned entities and runtime configuration.
4	Per-instance constraint	scripts[].object_id must reference a real objects[].id . No implicit aggregate placeholders (e.g. circle_all ).	Aggregate placeholders create ambiguous references unresolvable during code generation or evaluation.	Enforces 1:1 script-to-object binding; enables automatic referential integrity validation.
5	Evidence-aware semantics	Conditional relation labels (e.g. can_trigger_game_win_if_aligned ). Required evidence_type on every rules[] entry. Optional confidence field.	Unconditional labels over-assert determinism for conditional code paths. Evidence attribution improves trust calibration in generated output.	Generation produces more accurate causal claims; evaluation can filter or weight rules by evidence type and confidence.

Appendix

Table 5: IR schema iteration history from initial draft to frozen v0.2-runtime-evidence.

Tier	Item	Coverage
Top-level fields
Core	scene , objects , scripts , params , runtime_params , links , rules	26/26
Object types
Core	GameObject , PrefabInstance	26/26
Optional	PrefabAsset	4/26
Script classes

Appendix

Table 6: IR schema frequency tiers across all 26 patterns (v2). Thresholds: Core

\geq 80\%

, Common

\geq 40\%

, Optional

<40\%

Pattern	Objects	Scripts	Links	Rules
1 Ownership	9	4	12	3
2 Collection	7	4	13	3
3 Eliminate	7	5	13	3
4 Capture	7	6	14	3
5 Overcome	8	5	14	4
6 Evade	10	7	16	3

Appendix

Table 7: Full set of 26 goal patterns with IR structural statistics (v0.2-runtime-evidence). All 26 scenes pass referential integrity and evidence-type validation.

Pattern	Total	G	H	Top 2 G codes
1_Ownership	18	0	18	—
2_Collection	0	0	0	—
3_Eliminate	36	0	36	—
4_Capture	12	0	12	—
5_Overcome	4	0	4	—
6_Evade	30	30	0	CS0115(15), CS0246(15)

Appendix

Table 8: Pattern-level errors: no_schema, DeepSeek-Coder-V2-Lite (timeout 37–51%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	0	0	0	—
2_Collection	23	8	15	CS0115(2), CS0234(2)
3_Eliminate	1	0	1	—
4_Capture	29	27	2	CS0234(16), CS0115(11)
5_Overcome	1	1	0	CS0234(1)
6_Evade	20	2	18	CS0115(1), CS0234(1)

Appendix

Table 9: Pattern-level errors: no_schema, Qwen2.5-Coder-7B (timeout 37–51%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	11	9	2	CS0246(6), CS1061(3)
2_Collection	19	0	19	—
3_Eliminate	24	0	24	—
4_Capture	19	0	19	—
5_Overcome	8	0	8	—
6_Evade	14	0	14	—

Appendix

Table 10: Pattern-level errors: with_schema_free, DeepSeek-Coder-V2-Lite (timeout 86–92%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	20	0	20	—
2_Collection	28	0	28	—
3_Eliminate	22	2	20	CS0117(2)
4_Capture	28	0	28	—
5_Overcome	17	0	17	—
6_Evade	32	0	32	—

Appendix

Table 11: Pattern-level errors: with_schema_free, Qwen2.5-Coder-7B (timeout 86–92%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	36	13	23	CS0246(10), CS1061(3)
2_Collection	19	0	19	—
3_Eliminate	19	0	19	—
4_Capture	20	0	20	—
5_Overcome	23	11	12	CS0246(6), CS1061(5)
6_Evade	20	4	16	CS0246(2), CS1061(2)

Appendix

Table 12: Pattern-level errors: with_schema_min, DeepSeek-Coder-V2-Lite (timeout 96%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	35	1	34	CS0246(1)
2_Collection	31	0	31	—
3_Eliminate	24	7	17	CS0246(4), CS8121(2)
4_Capture	18	0	18	—
5_Overcome	21	4	17	CS0246(1), CS0311(1)
6_Evade	18	0	18	—

Appendix

Table 13: Pattern-level errors: with_schema_min, Qwen2.5-Coder-7B (timeout 96%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	25	10	15	CS0246(10)
2_Collection	19	0	19	—
3_Eliminate	22	0	22	—
4_Capture	17	0	17	—
5_Overcome	30	4	26	CS0246(2), CS1061(2)
6_Evade	18	0	18	—

Appendix

Table 14: Pattern-level errors: with_schema_full, DeepSeek-Coder-V2-Lite (timeout 97–99%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Pattern	Total	G	H	Top 2 G codes
1_Ownership	24	6	18	CS0246(5), CS0117(1)
2_Collection	28	8	20	CS0246(6), CS1061(2)
3_Eliminate	21	0	21	—
4_Capture	18	0	18	—
5_Overcome	24	6	18	CS0246(3), CS1061(3)
6_Evade	19	0	19	—

Appendix

Table 15: Pattern-level errors: with_schema_full, Qwen2.5-Coder-7B (timeout 97–99%; 20 logs per pattern). G = grounding failure; H = hygiene failure

Computational Creativity

Computational creativity studies how machines can generate, evaluate, and collaborate on novel, valuable ideas and artifacts.

See all 114 papers

Jul 11, 2026

Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

Large language models (LLMs) write Unity C# for game scenes. Yet nearly all demonstrations rest on an iterative repair loop that regenerates code until it compiles, conflating what the model writes with what the loop fixes. We remove the loop and evaluate a single pass, where the first draft is final. This isolates the model's parametric knowledge, the most stringent test of unaided generation. Models instantiate Goal Playable Concepts, playable counterparts of goal patterns, across 10,400 generations (four open-weight models, 7B--30B; two generation modes; four intermediate-representation (IR) conditioning levels; 26 goal patterns; 20 seeds). None compiled into a runnable scene, leaving no survivorship bias. To understand how the generated C# scripts fail, we categorize the 99 error codes behind 90{,}673 compiler-error occurrences as Grounding (invented or misused Unity types and APIs) or Hygiene (structural defects needing no Unity knowledge). The split differs sharply by goal pattern (e.g., Stealth fails mostly on invented engine references; Capture on plain C# structure). Larger models, stricter IRs, and different generation modes move the errors but never yield a compiling scene. The bottleneck is missing engine-specific knowledge. The census orders goal patterns by that demand, showing designers where single-pass generation breaks.

Hugh Xuechen Liu, Kıvanç Tatar

Chalmers University of Technology and University of Gothenburg, SE-412 96 Göteborg, Sweden

LLM-Based Program Synthesis Procedural Content Generation Code Generation Evaluation LLM Code Generation

Ask PDF

Oct 2, 2026

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in web front-end execution, with browser-based game generation emerging as a particularly prominent frontier. While previous efforts frequently rely on complex multi-turn workflows or focus on static game evaluation benchmarks, this work targets direct end-to-end real-world game synthesis driven by coding agents. However, generating complex games directly from sparse user queries often forces coding agents to make underspecified assumptions, yielding incomplete mechanics, disconnected gameplay flows, and limited visual aesthetics. To resolve this issue, this paper presents GameGo, a scalable framework that systematically transforms brief game seeds into comprehensive Product Requirements Documents grounded in industry game-development practices. To retain core gameplay constraints without restricting design exploration, GameGo uses task-specific dynamic compression to maximize information density while preserving instruction following. Based on this pipeline, GameGoData is constructed with 55,060 development trajectories across 2D, 2.5D, and 3D games, alongside GameGoBench, a benchmark comprising 124 diverse game queries. Training GameGoCoder on GameGoData yields a model that outperforms matched baselines and is comparable to frontier models across gamedev benchmarks. All code, datasets, and models will be made publicly available.

Haoyue Yang, Jingyao Li, Zhengfan Wu +3

Institute of Automation, Chinese Academy of Sciences · Baidu Inc., Beijing, China

Agentic Code Generation Synthetic Data Generation for Language Models LLM Agent Trajectory Synthesis AI Agent Benchmarks

Ask PDF

May 8, 2026

Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate

Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') -- compile success, runtime success, structural fidelity, and mechanism adherence -- applied to 858 generation attempts across four open-weight LLMs (7B--30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C# generation achieves the highest runtime-pass rate (43% mean) yet produces structurally vacuous scenes (mechanism

F_1 \approx 0.12

). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure (

F_1

up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar

p = 1.0

), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the benchmark, replay logs, and per-record metrics for independent verification.

Hugh Xuechen Liu, Kıvanç Tatar

Chalmers University of Technology and University of Gothenburg

Code Generation Code Generation Evaluation Code Generation Benchmarks LLM Code Generation

Ask PDF

Abstract

Computational Creativity

Explore similar work

Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate