cs.HCMar 18, 2026

SPRITE: From Static Mockups to Engine-Ready Game UI

Authors: Yunshu Bai, RuiHao Li, Hao Zhang, Chien Her Lim, Ming Yan, Mengtian Li

Organizations: Shanghai University, Shanghai, China · MiAO Worlds, Singapore, Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Game UI implementation requires translating stylized mockups into interactive engine entities. However, current "Screenshot-to-Code" tools often struggle with the irregular geometries and deep visual hierarchies typical of game interfaces. To bridge this gap, we introduce SPRITE, a pipeline that transforms static screenshots into editable engine assets. By integrating Vision-Language Models (VLMs) with a structured YAML intermediate representation, SPRITE explicitly captures complex container relationships and non-rectangular layouts. We evaluated SPRITE against a curated Game UI benchmark and conducted expert reviews with professional developers to assess reconstruction fidelity and prototyping efficiency. Our findings demonstrate that SPRITE streamlines development by automating tedious coding and resolving complex nesting. By facilitating rapid in-engine iteration, SPRITE effectively blurs the boundaries between artistic design and technical implementation in game development. Project page: https://baiyunshu.github.io/sprite.github.io/

Figures & tables

Figure 1. The SPRITE System.Transforming a raw GameUI screenshot (left) into editable engine assets (right). Unlike standard app interfaces, Game UIs are characterized by highly stylized aesthetics, irregular geometries, and complex visual hierarchies, making manual reconstruction a labor-intensive task that requires specialized expertise. By automating this reconstruction, SPRITE effectively bridges the gap between artistic vision and engine implementation to enable rapid gameplay iteration. A pipeline overview of the SPRITE system. The left side shows a raw, highly stylized game user interface screenshot with ornate buttons and HUD elements. An arrow points to the right side, which displays the reconstructed version inside a game engine editor. The reconstruction highlights editable components, irregular geometries, and layout structures, demonstrating the automated transition from a static image to functional engine assets.
Figure 2. SPRITE Our system transforms mockups into engine assets via three stages: (1) Semantic Scaffolding , VLM infers a schema-guided hierarchical YAML; (2) Precision Grounding , utilizing 2D models for pixel-perfect extraction and geometric calibration; and (3) Engine-Native Synthesis , where an MLLM generates executable UXML/USS code and interaction logic. SPRITE pipeline: (1) VLM converts mockup to YAML; (2) 2D models extract and calibrate elements; (3) MLLM generates UXML/USS and interaction logic.
Figure 3. YAML template structure for UI scaffolding. A code snippet illustrating a YAML structure. It shows root-level container panels with parent set to null, and nested interactive elements referencing their parent. Each element includes fields for label, 2D bounding box coordinates, and a segmentation prompt.
Figure 4. System Prompt: UI Master Persona A text box showing the system prompt for the UI Master Persona. It instructs the vision-language model to act as an expert Game UI developer, analyze a screenshot, and output a multi-level hierarchical UI scaffold. It specifies two types of elements: container panels and interactive elements, requiring outputs in YAML format with bounding boxes, segmentation prompts, and parent references.
Figure 5. Qualitative Comparison. While VLMs (a-b) are limited to bounding boxes and the baseline (c) suffers from fragmentation, SPRITE (d) leverages semantic scaffolding to extract pixel-perfect assets for irregular shapes. Visual comparison of four methods: (a-b) VLMs show bounding boxes; (c) baseline shows fragmented segments; (d) SPRITE extracts precise, irregular assets.
Figure 6. Visual representation of the GAMEUI Benchmark gallery. These representative samples demonstrate the system’s ability to faithfully reconstruct a wide array of challenging layouts across different game genres. Ranging from high-density RPG inventory grids to complex, multi-level navigation systems, this gallery highlights the functional diversity and zero-shot generalizability explicitly targeted by SPRITE. Visual representation of the GAMEUI Benchmark gallery.
More in this topic

Image-to-Code Generation

See all 17 papers

Explore similar work

CardsList

  1. GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

    Mar 16, 2026Wei Zeng, Fengwei An, Zhen Liu +1Vision-Language AgentsLLM Agent Self-Improvement

    School of Microelectronics, Southern University of Science and Technology (SUSTech), Shenzhen, China · Zhongguancun Academy, Beijing, China · School of Marxism, Tsinghua University, Beijing, China

  2. From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

    Aug 12, 2026Houston H. Zhang, Tao Zhang, Li Gu +5Image-to-Code GenerationCode Generation

    McMaster University · University of Toronto · Concordia University

  3. GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

    Oct 2, 2026Haoyue Yang, Jingyao Li, Zhengfan Wu +3Agentic Code GenerationSynthetic Data Generation for Language Models

    Institute of Automation, Chinese Academy of Sciences · Baidu Inc., Beijing, China