UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning

Hao Yang; Shuoyang Liu; Xiaoyu Lin; Zhou Fang; Maji Huang; Zhaosong Huang; Ruichao Mao; Teng Guo

cs.CVOct 31, 2026

UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning

Authors: Hao Yang, Shuoyang Liu, Xiaoyu Lin, Zhou Fang, Maji Huang, Zhaosong Huang, Ruichao Mao, Teng Guo,

Organizations: Ant Group

Abstract

Mobile UI understanding and generation require both fine-grained component perception and reliable structured synthesis. Inspired by the unified direction introduced by UI-UG, we present UI-UG 2, a compact 4B-parameter multimodal large language model (MLLM) that advances mobile UI Understanding and Generation with broader task coverage, a front-end-aligned taxonomy, and online render-aware reinforcement learning. UI-UG 2 constructs a multi-source understanding corpus from RICO, VINS, Android in the Wild, and Android Control, annotated with an 80-category two-level UI taxonomy aligned with front-end component semantics, and pairs it with over 5,000 requirement-screenshot-JSON tuples for progressive structured UI generation through a domain-specific language (DSL). For understanding, UI-UG 2 introduces decomposed rewards for category prediction, OCR fidelity, color recognition, dense grounding, and GUI grounding. Unlike the offline Direct Preference Optimization (DPO)-style generation optimization in UI-UG, UI-UG 2 optimizes generation through end-to-end online Group Relative Policy Optimization (GRPO): sampled DSL outputs are rendered with a Playwright-based server-side pipeline and evaluated using LLM-as-a-Judge and contrastive preference ranking. Experiments show that UI-UG 2 achieves the strongest referring and dense-grounding results among the evaluated models on our 80-category benchmark, and achieves the best composite generation score among compared models at a compact scale (4B). More importantly, we observe a consistent understanding-to-generation transfer: adding fine-grained UI perception supervision improves generation quality under a fixed training budget, suggesting that component-level semantics provide a shared representational foundation for both tasks.

Figures & tables

Figure 1: Overview of UI-UG 2. UI-UG 2 unifies fine-grained mobile UI understanding and structured UI generation with a front-end-aligned taxonomy, joint training, task-specific rewards, and online render-aware optimization.

Figure 2: Pipeline for constructing the UI generation dataset. Each tuple consists of a requirement, a reference image, and a corresponding renderable UI JSON DSL.

Hyperparameter	SFT	GRPO
Base Model	Qwen3-VL-4B-Instruct
Frozen Modules	ViT Encoder
Trainable Modules	LLM + VL Adapter
Max Learning Rate	1e-5	1e-6
LR Schedule	Warmup-Stable-Decay	Warmup-Stable-Decay
Warmup Ratio	0.05	0.03

Table 1: Training hyperparameters for SFT and GRPO stages.

Closed-source
		Referring			Dense Grounding				GUI Grounding
Model	Params	Category	OCR	Color	F1	Precision	Recall	mIoU	SS-V1-Mobile	SS-V2-Mobile
GPT-5.2 [ 32 ]	$-$	0.415	0.551	0.611	0.048	0.045	0.052	0.656	0.492	0.352
Claude Sonnet 4.5 [ 2 ]	$-$	0.484	0.830	0.628	0.368	0.415	0.330	0.757	0.187	0.232
Gemini 3.1 Pro [ 13 ]	$-$	0.535	0.690	0.501	0.471	0.461	0.482	0.759	0.900	0.951
Open-source

Table 2: UI understanding results. Best results are in bold ; second-best are underlined .

-

: unknown;

\dagger

: unsupported by the native output interface. Unsupported entries are excluded from task rankings.

Closed-source
Model	JSON	Rend	Col	Str	Txt	Int	Sum
GPT-5.2 [ 32 ]	0.99	0.96	0.58	0.62	0.58	0.60	4.33
Claude Sonnet 4.5 [ 2 ]	0.96	0.94	0.69	0.70	0.71	0.70	4.70
Gemini 3.1 Pro [ 13 ]	1.00	0.97	0.42	0.53	0.48	0.49	3.89
Open-source
Qwen3-VL-4B [ 3 ]	0.98	0.96	0.59	0.59	0.60	0.58	4.30

Table 3: UI generation results. JSON: format compliance; Rend: render success; Col/Str/Txt/Int: four judge dimensions; Sum: composite score (maximum 6). Best results are in bold ; second-best are underlined .

Figure 3: RL strategy ablation. Each cell shows improvement over SFT; sequential multi-stage training yields larger gains.

Configuration	Precision	Recall	F1	mIoU
SFT (no RL)	0.798	0.693	0.742	0.865
$+R_{\text{fmt}}$	0.803	0.697	0.745	0.868
$+R_{\text{fmt}}+R_{\text{pre}}$	0.836	0.712	0.763	0.877
$+R_{\text{fmt}}+R_{\text{rec}}$	0.754	0.742	0.727	0.836
$+R_{\text{fmt}}+R_{\text{pre}}+R_{\text{rec}}$	0.832	0.729	0.777	0.894

Table 4: Ablation of dense grounding reward components. Each row specifies the reward combination applied to the SFT baseline. Precision, Recall, and F1 are averaged over samples; therefore, the reported F1 is not necessarily the harmonic mean of the averaged Precision and Recall.

Figure 4: Effect of understanding-data ratio during SFT. Generation peaks at U20, while understanding metrics increase monotonically.

Reward	$\Delta$ Col	$\Delta$ Str	$\Delta$ Txt	$\Delta$ Int
$+R_{\text{cls}}$ (Category)	$+$ 1.31%	$+$ 0.86%	$+$ 1.31%	$-$ 1.49%
$+R_{\text{col}}$ (Color)	$+$ 1.60%	$-$ 0.39%	$-$ 0.18%	$+$ 0.41%
$+R_{\text{ocr}}$ (Text)	$-$ 0.11%	$-$ 1.45%	$+$ 2.48%	$-$ 0.13%
$+R_{\text{dense}}$ (Dense Gnd.)	$+$ 1.64%	$+$ 4.72%	$+$ 0.25%	$+$ 1.74%
$+R_{\text{cap}}$ (GUI Gnd.)	$+$ 0.20%	$-$ 0.27%	$-$ 0.25%	$-$ 0.17%

Table 5: Relative change in generation scores after applying each understanding reward independently in RL training. Notable Gain is shown in bold (>1%).

Figure 5: Cosine similarity of text-to-vision attention patterns across understanding and generation tasks. Generation and dense grounding show the strongest alignment.

Figure 6: Visual comparison of UI generation quality on two representative cases (poster card and login interface). Each column corresponds to one model under the same prompt condition. UI-UG 2-4B (rightmost) shows better layout stability, hierarchy consistency, and style-control fidelity than representative MLLMs.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Parent Class	Subcategories (complete list)	AntD Correspondence
Card (9)	InfoCard, CommodityCard, MenuCard, InfoCardTruncated, CommodityCardTruncated, Banner, BannerTruncated, NoticeBar, Swiper	AntD Card (functional/truncation variants); AntD NoticeBar; AntD Swiper; custom Banner pattern
Collapse (9)	CollapseFold, CollapseUnFold, Popup, FloatingPanel, List, ListItem, Table, Form, FormItem	AntD Collapse (state-level split); AntD Popup/FloatingPanel; AntD List/List.Item; AntD Form/Form.Item; custom Table
Chart (7)	LineChart, HorizontalBarChart, VerticalBarChart, CircleChart, RadarChart, DashBoardChart, OtherChart	Not in AntD; referenced from AntV/F2, ECharts, and Recharts; custom OtherChart for uncommon types
Navigation (7)	Tabs, NavBar, TabBar, SideBar, IndexBar, TabsItemSelected, TabsItemUnselected	AntD Tabs/NavBar/TabBar/SideBar/IndexBar; custom item-state split (Selected/Unselected)
Progress (6)	ProgressBar, ProgressCircle, Selector, Steps, StepsItem, Scroll	AntD ProgressBar/ProgressCircle; AntD Selector; AntD Steps/Steps.Step; custom Scroll indicator
Button (10)	CommonButton, SideButton, CustomButton, TextButton, CloseButton, MoreButton, HomeButton, ReturnButton, Switch, Dropdown	AntD Button (functional variants); AntD Switch; AntD Dropdown

Appendix

Table 6: Complete UI taxonomy summary (80 classes = 9 parents + 71 subcategories). AntD Mobile is used here as the reference for front-end component correspondence.

Dataset	Categories	OCR / Color	Level	FW-Aligned	# Images
RICO Deka et al. [2017]	24	× / ×	Element	×	$\sim$ 72k
VINS Bunian et al. [2021]	11	× / ×	Element	×	$\sim$ 4k
AITW Rawles et al. [2023]	—	× / ×	Action	✓	$\sim$ 715k
AC Li et al. [2024a]	—	× / ×	Action	×	$\sim$ 1.5M
Ours	80 (9 + 71)	✓	Element+Action	✓	>30k

Appendix

Table 7: UI element taxonomy coverage comparison. FW-Aligned indicates whether the labels are aligned with front-end component libraries such as Ant Design, Material UI, and Vant.

Panel of Figure 7 — Figure 7: Visualization examples of dataset annotations (2 $\times$ 4). The order in each row is RICO, VINS, AITW, and AC. Overlaid annotations illustrate both UI element categories and bounding-box locations.

Figure 8: Scene category distribution across AC, AITW, RICO, and VINS. The four sources exhibit complementary scene biases that improve overall diversity.

Figure 9: Complexity-level distribution (Simple/Medium/High) for each source dataset. Most samples are simple/medium, with source-dependent medium/high tails.

Figure 10: A real example from our generation dataset. Left: rendered UI card. Right: corresponding JSON DSL (truncated). The full DSL contains nested layout, dynamic data bindings, and Tailwind styling tokens.

Figure 11: Overall metric comparison on the referring task across models.

Figure 12: Per-category precision comparison under strict and relaxed type matching protocols.

Figure 13: Overall grounding metric comparison across models under strict and macro-label protocols (F1@50/F1@75 and macro_F1@50/macro_F1@75).

Figure 14: Per-category grounding F1@50 comparison across models. Common categories are generally easier, while complex or confusable UI categories remain challenging for most baselines.

Panel of Figure 15 — Figure 15: Visual comparison of grounding results on two dense UI examples (2 $\times$ 7). Each row follows the same model order: GPT-5.2, Gemini 3.1 Pro, Claude Sonnet 4.5, Qwen3-VL-235B-A22B, Kimi K2.5, UI-Venus-Ground-72B, and UI-UG 2-4B (ours).

Dataset	Icon	Text	Overall
ScreenSpot-V1 Mobile	0.887	0.916	0.902
ScreenSpot-V2 Mobile	0.912	0.944	0.929

Appendix

Table 8: UI-UG 2-4B GUI grounding accuracy on ScreenSpot mobile benchmarks by element type.

Figure 16: Attention maps demonstrate task-dependent focus, with highlights concentrated on the corresponding target regions.

Judge Model	$\Delta_{\text{Col}}$	$\Delta_{\text{Str}}$	$\Delta_{\text{Txt}}$	$\Delta_{\text{Int}}$	$\Delta_{\text{Overall}}$	Variance
Gemini 3.1 Pro	0.07	0.08	0.09	0.08	0.08	0.032
Qwen3-VL-235B-A22B	0.15	0.17	0.13	0.20	0.17	0.046
GPT-5.2	0.12	0.11	0.14	0.12	0.13	0.049
Claude Sonnet 4.5	0.21	0.24	0.23	0.26	0.24	0.050

Appendix

Table 9: LLM-as-a-Judge reliability on 100 samples.

\Delta

is the mean absolute deviation from human reference (lower is better). Variance is computed on residuals against the human reference.

Model	Size	Size-Based Factor	Estimated Time (s)
UI-UG 2-4B	4B	1.00 $\times$	0.9
Qwen3-VL-30B-A3B	A3B	0.75 $\times$	0.7
UI-Venus-Ground-7B	7B	1.75 $\times$	1.6
Qwen3-VL-8B	8B	2.00 $\times$	1.8
UI-Venus-Ground-72B	72B	18.00 $\times$	16.0
Qwen3-VL-235B-A22B	A22B	5.50 $\times$	4.9

Appendix

Table 10: Estimated UI generation latency using a size-based scaling factor for larger open-source baselines.

Figure 17: Progressive rendering of generated UI specifications.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,114 papers

Feb 24, 2026

Generative UI: LLMs are Effective UI Generators

AI models excel at creating content, but typically render it with static, predefined interfaces. Specifically, the output of LLMs is often a markdown "wall of text". Generative UI is a long standing promise, where the model generates not just the content, but the interface itself. Until now, Generative UI was not possible in a robust fashion. We demonstrate that when properly prompted and equipped with the right set of tools, a modern LLM can robustly produce high quality custom UIs for virtually any prompt. When ignoring generation speed, results generated by our implementation are overwhelmingly preferred by humans over the standard LLM markdown output. In fact, while the results generated by our implementation are worse than those crafted by human experts, they are at least comparable in 50% of cases. We show that this ability for robust Generative UI is emergent, with substantial improvements from previous models. We also create and release PAGEN, a novel dataset of expert-crafted results to aid in evaluating Generative UI implementations, as well as the results of our system for future comparisons. Interactive examples can be seen at https://generativeui.github.io

Yaniv Leviathan, Dani Valevski, Matan Kalman +10

Cheenu · Google Research

Web Application Generation

Ask PDF

Jun 11, 2026

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.

Ruichao Mao, Zhou Fang, Teng Guo +9

Ant Group

Multimodal Reasoning Multimodal Model Evaluation

Ask PDF

Jun 4, 2026

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model's own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) a reflective prompt-evolution step from the model's own failure traces. Across multiple domains and datasets, PerceptUI achieves human-level realism, generalizes to unseen questions and personas, and yields population-level response distributions.

Nicolas Bougie, Xiaotong Ye, Gian Maria Marconi +1

1Woven by Toyota

User Modeling User Simulation Human-Computer Interaction

Ask PDF

Abstract

Multimodal Large Language Models

Explore similar work

Generative UI: LLMs are Effective UI Generators

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation