GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

Wei Zeng; Fengwei An; Zhen Liu; Jian Zhao

cs.AIMar 16, 2026

GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

Authors: Wei Zeng, Fengwei An, Zhen Liu, Jian Zhao

Organizations: School of Microelectronics, Southern University of Science and Technology (SUSTech), Shenzhen, China · Zhongguancun Academy, Beijing, China · School of Marxism, Tsinghua University, Beijing, China

Abstract

Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process. We present GameUIAgent, an LLM-powered agentic framework that translates natural language descriptions into editable Figma designs via a Design Spec JSON intermediate representation. A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction with guaranteed non-regressive quality. Evaluated across 110 test cases, three LLMs, and three UI templates, cross-model analysis establishes a game-domain failure taxonomy (rarity-dependent degradation; visual emptiness) and uncovers two key empirical findings. A Quality Ceiling Effect (Pearson r=-0.96, p<0.01) suggests that RC improvement is bounded by headroom below a quality threshold -- a visual-domain counterpart to test-time compute scaling laws. A Rendering-Evaluation Fidelity Principle reveals that partial rendering enhancements paradoxically degrade VLM evaluation by amplifying structural defects. Together, these results establish foundational principles for LLM-driven visual generation agents in game production.

Figures & tables

Fig. 1: Overall architecture of GameUIAgent. Stages 1–5 constitute the forward generation pass; Stage 6 (Reflection Controller, green) closes the agentic loop by converting VLM quality scores into targeted repair prompts and re-invoking Stage 2 if $S_{\text{avg}}\!<\!\theta$ . The Design Spec JSON intermediate representation (dashed box) separates creative generation from deterministic enhancement.

Fig. 2: Rarity progression (N to UR) generated by GameUIAgent with DeepSeek V3. Progressively richer visual treatments are injected automatically: simple gray borders (N) $\to$ colored theme borders (R) $\to$ gradient fills with glow (SR) $\to$ multi-layered borders with star badges (SSR) $\to$ golden ornate frames (UR). All five cards share the Fire element theme, demonstrating consistent thematic adaptation across the rarity hierarchy.

Model	$V$ (%)	$T$ (s)	Tokens	$S_{L}$	$S_{C}$	$S_{R}$	$S_{P}$	$S_{A}$	$S_{\text{avg}}$	$/call
DeepSeek V3 †	98.0	47.0	4,534	8.1 $\pm$ 0.4	8.9 $\pm$ 0.4	7.9 $\pm$ 0.5	7.1 $\pm$ 0.6	8.1 $\pm$ 0.5	8.0 $\pm$ 0.3	0.008
Gemini 2.0 Flash ‡	88.0	14.7	5,141	6.6 $\pm$ 0.6	7.2 $\pm$ 0.8	5.6 $\pm$ 0.7	6.1 $\pm$ 0.9	6.4 $\pm$ 0.6	6.4 $\pm$ 0.6	0.001
GPT-4o-mini ‡	56.0	35.2	4,485	6.9 $\pm$ 0.4	7.3 $\pm$ 0.8	6.0 $\pm$ 0.5	6.5 $\pm$ 0.8	6.6 $\pm$ 0.5	6.7 $\pm$ 0.4	0.012
† VLM on Figma renders ( $n\!=\!49$ ). ‡ VLM on preview renders (Gemini $n\!=\!44$ , GPT-4o-mini $n\!=\!28$ ).

TABLE I: Cross-Model Comparison on Character Card Generation (

n\!=\!50

). Note : DeepSeek V3 scores are from Figma renders; other models use preview renders. The RQ3 renderer comparison (Table III ) estimates a Figma advantage of up to

+1.58

points, implying that cross-model quality gaps reflect both model capability and renderer differences.

Fig. 3: Cross-model analysis. (A) Failure cases (left panel): (a) Gemini UR-tier degradation ( $V\!=\!60\%$ ); (b–c) GPT-4o-mini visual emptiness on Item Thumbnail and Skill Panel; (d) DeepSeek V3 UR reference. (B) R-tier comparison (right panel, enlarged for clarity): DeepSeek V3 (left, $S\!=\!8.0$ ) produces coherent themed borders, readable color-coded stats, and a well-structured portrait region; GPT-4o-mini (right, $S\!=\!6.7$ ) yields a syntactically valid design but with low-contrast text, blank portrait region, and element overlap.

Config.	$V$ (%)	$T$ (s)	Tok.	$S_{\text{avg}}$	$S_{L}$	$S_{R}$	$S_{A}$
Full Pipeline †	98.0	47	4,534	8.0	8.1	7.9	8.1
No Few-Shot ‡	96.0	79	3,557	7.9	8.0	7.9	7.5
No Post-Proc. ‡	96.0	47	4,534	6.5	6.7	5.7	6.5
No-Schema ‡	100.0	20	728	4.8	4.9	4.7	4.8
$\Delta^{\ddagger}$	—	—	—	+3.1	+3.1	+3.2	+2.7
† Figma renders. ‡ Preview renders. $\Delta^{\ddagger}$ : No-FS minus No-Schema (matched renders).

TABLE II: Ablation and Baseline Study (DeepSeek V3,

n\!=\!50

)

Renderer	$S_{\text{avg}}$	$S_{R}$	$S_{C}$	$\Delta$ vs. prev	$p$	$d$
Flat (solid colors only)	4.42 $\pm$ 1.01	4.46	5.38	—	—	—
Gradient (+grad./shadows, no layout)	3.44 $\pm$ 0.63	2.74	4.44	$-$ 1.00 a	0.0017	$-$ 0.81
Layout-aware (+Auto Layout simulation)	5.96 $\pm$ 0.84	4.82	6.93	$+$ 2.52 b	$<$ 0.001	2.74
Net Flat $\to$ Layout-aware	—	—	—	$+$ 1.58 c	0.0001	1.16
a Gradient amplifies overlap salience ( $S_{R}$ : $-$ 1.73, $p\!<\!0.001$ , $d\!=\!-1.38$ ).
b Auto Layout simulation eliminates overlaps before rasterization.

TABLE III: Controlled Renderer Comparison: Same 27 Skill Panel Designs Evaluated Across Three Configurations (paired Wilcoxon signed-rank)

Fig. 4: Rendering-Evaluation Fidelity: the Gradient renderer decreases VLM score by

-1.00

(

p\!=\!0.0017

d\!=\!-0.81

) relative to the Flat baseline; the Layout-aware renderer recovers and surpasses it (

\Delta\!=\!+2.52

p\!<\!0.001

d\!=\!2.74

n\!=\!27

Skill Panel designs evaluated across three renderer configurations with identical Design Spec JSONs.

Metric	N ( $n\!=\!10$ )	R ( $n\!=\!9$ )	SR ( $n\!=\!9$ )	All ( $n\!=\!28$ )
Init $S_{\text{avg}}$	5.30 $\pm$ 0.48	5.56 $\pm$ 0.78	7.00 $\pm$ 0.50	5.93 $\pm$ 0.98
Final $S_{\text{avg}}$	6.40 $\pm$ 0.70	6.67 $\pm$ 1.00	7.67 $\pm$ 0.50	6.89 $\pm$ 0.92
Avg $\Delta$	+1.10	+1.11	+0.67	+0.96
Improved / Same	9/1	7/2	5/4	21/7
Regression-free	10/10	9/9	9/9	28/28 (100%)
Final $\geq\!7.5$	0/10	2/9	6/9	8/28 (28.6%)

TABLE IV: Reflection Controller Results by Rarity Tier (DeepSeek V3, Gradient renderer,

K\!\leq\!2

\theta\!=\!7.5

; Wilcoxon:

W\!=\!0

p\!<\!0.001

d\!=\!1.30

Fig. 5: Quality Ceiling Effect: Pearson

r\!=\!-0.96

(

p\!<\!0.01

) between mean initial VLM score and RC improvement (

\Delta

) across five independent conditions (

n_{\text{total}}\!=\!93

; DeepSeek V3 and GPT-4o-mini; five tier mixes). RC gain is universally bounded by headroom below

\theta

, not by model capability or rarity complexity.

Fig. 6:

K\!\leq\!3

N-tier ablation: RC score trajectories for representative cases. CC-001 exhibits a plateau-breaking pattern (

7.0\!\to\!7.0\!\to\!6.4\!\to\!8.0

) inaccessible at

K\!\leq\!2

; CC-007 converges at

K\!=\!1

; best-result tracking prevents CC-006’s

K\!=\!3

regression. Dashed line:

\theta\!=\!7.5

Vision-Language Agents

Vision-language agents use visual and language models to perceive environments, reason about goals, and take actions through interactive interfaces.

See all 45 papers

Generative UI: LLMs are Effective UI Generators
Feb 24, 2026Yaniv Leviathan, Dani Valevski, Matan Kalman +10Web Application Generation
Cheenu · Google Research
Agent+P: Guiding UI Agents via Symbolic Planning
Oct 7, 2025Shang Ma, Xusheng Xiao, Yanfang YeGUI Agents Mobile GUI Agents
University of Notre Dame · Arizona State University
Game-Creation-Agents: Autonomous Game Production
Oct 12, 2026Tencent Multimodal RL Team, Xiangxin Zhou, Jiaqi Tang +15AI Coding Agents Software Engineering Agents
Multimodal Reinforcement Learning Team, Tencent

Feb 24, 2026J/K move · Enter open · S save

Generative UI: LLMs are Effective UI Generators

Yaniv Leviathan, Dani Valevski, Matan Kalman, Danny Lumen+9

Cheenu · Google Research

Web Application Generation

Open paper AskPDF arXiv

AI models excel at creating content, but typically render it with static, predefined interfaces. Specifically, the output of LLMs is often a markdown "wall of text". Generative UI is a long standing promise, where the model generates not just the content, but the interface itself. Until now, Generative UI was not possible in a robust fashion. We demonstrate that when properly prompted and equipped with the right set of tools, a modern LLM can robustly produce high quality custom UIs for virtually any prompt. When ignoring generation speed, results generated by our implementation are overwhelmingly preferred by humans over the standard LLM markdown output. In fact, while the results generated by our implementation are worse than those crafted by human experts, they are at least comparable in 50% of cases. We show that this ability for robust Generative UI is emergent, with substantial improvements from previous models. We also create and release PAGEN, a novel dataset of expert-crafted results to aid in evaluating Generative UI implementations, as well as the results of our system for future comparisons. Interactive examples can be seen at https://generativeui.github.io

Abstract

Vision-Language Agents

Explore similar work

Generative UI: LLMs are Effective UI Generators

Agent+P: Guiding UI Agents via Symbolic Planning

Game-Creation-Agents: Autonomous Game Production