GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation
Authors: Wei Zeng, Fengwei An, Zhen Liu, Jian Zhao
Organizations: School of Microelectronics, Southern University of Science and Technology (SUSTech), Shenzhen, China · Zhongguancun Academy, Beijing, China · School of Marxism, Tsinghua University, Beijing, China
Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process. We present GameUIAgent, an LLM-powered agentic framework that translates natural language descriptions into editable Figma designs via a Design Spec JSON intermediate representation. A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction with guaranteed non-regressive quality. Evaluated across 110 test cases, three LLMs, and three UI templates, cross-model analysis establishes a game-domain failure taxonomy (rarity-dependent degradation; visual emptiness) and uncovers two key empirical findings. A Quality Ceiling Effect (Pearson r=-0.96, p<0.01) suggests that RC improvement is bounded by headroom below a quality threshold -- a visual-domain counterpart to test-time compute scaling laws. A Rendering-Evaluation Fidelity Principle reveals that partial rendering enhancements paradoxically degrade VLM evaluation by amplifying structural defects. Together, these results establish foundational principles for LLM-driven visual generation agents in game production.
Figures & tables
Fig. 1: Overall architecture of GameUIAgent. Stages 1–5 constitute the forward generation pass; Stage 6 (Reflection Controller, green) closes the agentic loop by converting VLM quality scores into targeted repair prompts and re-invoking Stage 2 if Savg<θ . The Design Spec JSON intermediate representation (dashed box) separates creative generation from deterministic enhancement.
Fig. 2: Rarity progression (N to UR) generated by GameUIAgent with DeepSeek V3. Progressively richer visual treatments are injected automatically: simple gray borders (N) → colored theme borders (R) → gradient fills with glow (SR) → multi-layered borders with star badges (SSR) → golden ornate frames (UR). All five cards share the Fire element theme, demonstrating consistent thematic adaptation across the rarity hierarchy.
Model
V (%)
T (s)
Tokens
SL
SC
SR
SP
SA
Savg
$/call
DeepSeek V3 †
98.0
47.0
4,534
8.1 ± 0.4
8.9 ± 0.4
7.9 ± 0.5
7.1 ± 0.6
8.1 ± 0.5
8.0 ± 0.3
0.008
Gemini 2.0 Flash ‡
88.0
14.7
5,141
6.6 ± 0.6
7.2 ± 0.8
5.6 ± 0.7
6.1 ± 0.9
6.4 ± 0.6
6.4 ± 0.6
0.001
GPT-4o-mini ‡
56.0
35.2
4,485
6.9 ± 0.4
7.3 ± 0.8
6.0 ± 0.5
6.5 ± 0.8
6.6 ± 0.5
6.7 ± 0.4
0.012
† VLM on Figma renders ( n=49 ). ‡ VLM on preview renders (Gemini n=44 , GPT-4o-mini n=28 ).
TABLE I: Cross-Model Comparison on Character Card Generation ( n=50 ). Note : DeepSeek V3 scores are from Figma renders; other models use preview renders. The RQ3 renderer comparison (Table III ) estimates a Figma advantage of up to +1.58 points, implying that cross-model quality gaps reflect both model capability and renderer differences.
Fig. 3: Cross-model analysis. (A) Failure cases (left panel): (a) Gemini UR-tier degradation ( V=60% ); (b–c) GPT-4o-mini visual emptiness on Item Thumbnail and Skill Panel; (d) DeepSeek V3 UR reference. (B) R-tier comparison (right panel, enlarged for clarity): DeepSeek V3 (left, S=8.0 ) produces coherent themed borders, readable color-coded stats, and a well-structured portrait region; GPT-4o-mini (right, S=6.7 ) yields a syntactically valid design but with low-contrast text, blank portrait region, and element overlap.
b Auto Layout simulation eliminates overlaps before rasterization.
TABLE III: Controlled Renderer Comparison: Same 27 Skill Panel Designs Evaluated Across Three Configurations (paired Wilcoxon signed-rank)
Fig. 4: Rendering-Evaluation Fidelity: the Gradient renderer decreases VLM score by −1.00 ( p=0.0017 , d=−0.81 ) relative to the Flat baseline; the Layout-aware renderer recovers and surpasses it ( Δ=+2.52 , p<0.001 , d=2.74 ). n=27 Skill Panel designs evaluated across three renderer configurations with identical Design Spec JSONs.
Fig. 5: Quality Ceiling Effect: Pearson r=−0.96 ( p<0.01 ) between mean initial VLM score and RC improvement ( Δ ) across five independent conditions ( ntotal=93 ; DeepSeek V3 and GPT-4o-mini; five tier mixes). RC gain is universally bounded by headroom below θ , not by model capability or rarity complexity.
Fig. 6: K≤3 N-tier ablation: RC score trajectories for representative cases. CC-001 exhibits a plateau-breaking pattern ( 7.0→7.0→6.4→8.0 ) inaccessible at K≤2 ; CC-007 converges at K=1 ; best-result tracking prevents CC-006’s K=3 regression. Dashed line: θ=7.5 .
AI models excel at creating content, but typically render it with static, predefined interfaces. Specifically, the output of LLMs is often a markdown "wall of text". Generative UI is a long standing promise, where the model generates not just the content, but the interface itself. Until now, Generative UI was not possible in a robust fashion. We demonstrate that when properly prompted and equipped with the right set of tools, a modern LLM can robustly produce high quality custom UIs for virtually any prompt. When ignoring generation speed, results generated by our implementation are overwhelmingly preferred by humans over the standard LLM markdown output. In fact, while the results generated by our implementation are worse than those crafted by human experts, they are at least comparable in 50% of cases. We show that this ability for robust Generative UI is emergent, with substantial improvements from previous models. We also create and release PAGEN, a novel dataset of expert-crafted results to aid in evaluating Generative UI implementations, as well as the results of our system for future comparisons. Interactive examples can be seen at https://generativeui.github.io
Large Language Model (LLM)-based UI agents show great promise for UI automation but often hallucinate in long-horizon tasks due to their lack of understanding of the global UI transition structure. To address this, we introduce AGENT+P, a novel framework that leverages symbolic planning to guide LLM-based UI agents. Specifically, we model an app's UI transition structure as a UI Transition Graph (UTG), which allows us to reformulate the UI automation task as a pathfinding problem on the UTG. This further enables an off-the-shelf symbolic planner to generate a provably correct and optimal high-level plan, preventing the agent from redundant exploration and guiding the agent to achieve the automation goals. AGENT+P is designed as a plug-and-play framework to enhance existing UI agents. Evaluation on the AndroidWorld benchmark demonstrates that AGENT+P improves the success rates of state-of-the-art UI agents by up to 14.32% and reduces the action steps by 37.70%. Our code is available at: https://github.com/KiteFlyKid/agentp.
Shang Ma, Xusheng Xiao, Yanfang Ye
University of Notre Dame · Arizona State University
Game-Creation-Agents (GCA) turns a natural-language request into a runnable Phaser 4 browser game. The pipeline first builds a multimodal game design document that couples mechanics and scene contracts to grounded visual references. It then projects that document into a runtime scaffold, machine-readable contracts, and deterministic tests. Eight specialized agents next carry out production: scenes are implemented serially, assets are produced in parallel, and one reduced manifest joins the two branches into a single build. Build-linked real-UI checks and a code-blind critic then verify that build and decide whether the game may ship. Each blocker returns to the agent that owns the affected artifact for a bounded repair, and the game is released once all the gates pass. We evaluate finished games separately, through their rendered interfaces alone, over prompt-matched tasks run under a common protocol. GCA achieves stronger requirement satisfaction, core mechanics, content depth, and functional visuals than a comparable open pipeline. In a side-by-side human study, raters also score GCA's games higher on every dimension and prefer them overall.