SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation

Jiarui Dong; Yin Cai; Zhouhong Gu; Chenmou Wu; Ci Tao; Yiran Chen; Jialing Li; Xiaoran Shi

cs.CLAug 23, 2026

SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation

Authors: Jiarui Dong, Yin Cai, Zhouhong Gu, Chenmou Wu, Ci Tao, Yiran Chen, Jialing Li, Xiaoran Shi,

Organizations: Shanghai University of Engineering Science · Fudan University

Abstract

Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challenging due to uncontrolled data distributions, noisy annotations, and limited layout scenario coverage. To address this, we propose SchemaGUI, a template-based benchmark for controllable GUI generation evaluation. By synthesizing paired natural language instructions and deterministic function-call references from parameterized interface schemas, SchemaGUI can generate thousands of deterministically annotated tasks in seconds without human labeling. Based on 1,000 evaluated instances per scenario and language across six representative bilingual scenarios, we benchmark five mainstream models, including the Qwen3.5 family, Qwen3-Coder-30B, and DeepSeek-R1. Our extensive analysis reveals three key insights. First, precise geometric spatial control remains an important bottleneck; while scaling Qwen3.5 from 4B to 27B improves Schema Feasibility from 91.56% to 99.63%, the Geometry score improves more modestly (from 67.05% to 75.30%). Second, generation difficulty is highly sensitive to layout complexity, with current LLMs excelling at simple sequential arrangements but suffering severe coordinate drift in dense grids and multi-region compositions. Third, thinking mode increases token consumption while generally reducing GUI Score, particularly for smaller models.

Figures & tables

Figure 1: Comparison of different data construction paradigms for GUI generation evaluation. (Left) Web-Crawled : data distribution is uncontrolled with expensive manual annotations. (Middle) LLM-Synthesized : the data distribution remains unknown while outputs suffer from model noise and hallucinated references. (Right) SchemaGUI (Ours) : parametric slot configurations enable enumerable distributions and deterministic, exact annotations.

Figure 2: Overview of the SchemaGUI benchmark. Stage 1 samples a GUI instance $s$ from the schema-driven design space $\mathcal{S}$ . Stage 2 constructs the aligned triplet $(x,y^{T},y^{L})$ through instruction realization, deterministic reference generation, and model prediction. Stage 3 assigns zero to schema-infeasible predictions and computes the GUI Score for feasible predictions over five evaluation dimensions.

Model	Lang.	Calculator	Login Form	Music Player	Doc Editor	Image Editor	Info Card	Overall
Qwen3.5-4B	EN	33.25	84.58	77.84	87.34	74.97	82.59	73.43
Qwen3.5-4B	ZH	56.26	89.92	84.20	78.79	65.35	84.71	76.54
Qwen3.5-9B	EN	43.40	85.14	87.51	88.88	64.26	83.77	75.49
Qwen3.5-9B	ZH	63.22	90.25	88.24	79.53	67.24	78.74	77.87
Qwen3.5-27B	EN	71.93	93.07	88.49	88.85	80.84	84.51	84.61
Qwen3.5-27B	ZH	74.82	93.10	86.03	81.38	70.81	84.55	81.78

Table 1: Main GUI Score results (%) across six GUI generation scenarios. EN and ZH denote the English and Chinese evaluation sets, respectively. Overall is the macro-average over the six scenarios, with each scenario weighted equally. Within each language, the best result in each column is shown in bold , and the second-best is underlined .

Figure 3: GUI Score across the six GUI generation scenarios, macro-averaged over the five evaluated models and two language settings. The horizontal lines indicate the minimum and maximum scores observed for each scenario.

English (EN)
Model	$\mathcal{G}$	$\mathcal{F}$	$\mathcal{T}$	$\mathcal{O}$	$\mathcal{H}$	Feas.
Qwen3.5-4B	68.52	95.55	86.34	84.38	70.10	86.07
Qwen3.5-9B	73.33	94.68	86.15	82.07	70.40	88.68
Qwen3.5-27B	77.37	99.57	88.19	85.34	74.97	99.37
Qwen3-Coder	72.98	99.99	87.08	82.20	72.12	99.83
DeepSeek-R1	79.09	99.20	87.01	84.03	74.28	87.65

Table 2: Submetric performance (%) and Schema Feasibility (Feas.) in EN and ZH. Submetrics are averaged over schema-feasible predictions. Best and second-best results within each language are shown in bold and underlined , respectively.

Model	Lang.	Calculator	Login Form	Music Player	Doc Editor	Image Editor	Info Card	Overall
Qwen3.5-4B -	EN	2525.9	283.4	591.6	318.7	297.7	192.9	701.7
Qwen3.5-4B -	ZH	1252.4	198.3	329.7	285.2	324.9	146.7	422.9
Qwen3.5-4B +	EN	9022.0	3625.2	4094.8	4570.7	5409.7	3232.2	4992.4
Qwen3.5-4B +	ZH	10181.8	4569.9	4607.1	5563.1	5612.4	4275.3	5801.6
Qwen3.5-9B -	EN	1991.9	309.6	393.1	340.1	298.0	192.9	587.6
Qwen3.5-9B -	ZH	1427.6	299.6	426.1	376.6	447.4	203.5	530.1

Table 3: Average raw response-token consumption (

T_{\mathrm{out}}

) across six GUI generation scenarios. EN and ZH denote the English and Chinese evaluation sets. Superscripts - and + indicate non-reasoning and reasoning-enabled configurations, respectively. Overall is the macro-average across the six scenarios. Within each language, the lowest and second-lowest values are shown in bold and underlined.

Figure 4: Changes in GUI Score after enabling thinking mode for the Qwen3.5 models. Each value denotes $\Delta=\mathrm{GS}_{\mathrm{thinking}}-\mathrm{GS}_{\mathrm{no\text{-}thinking}}$ in percentage points; negative values indicate performance degradation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	Controlled variation
Calculator	Calculator type, grid structure, display configuration, operation set, and component scale.
Login Form	Form configuration, field composition, optional controls, layout style, window size, and spacing.
Music Player	Player layout, interface attributes, artwork, track metadata, playback controls, and feature set.
Doc Editor	Panel organization, directory structure, document content, toolbar actions, and interface attributes.
Image Editor	Canvas-panel layout, tool set, layer structure, selected layer, property fields, and interface attributes.
Info Card	Card layout, avatar, profile content, optional contact fields, window size, and spacing.

Appendix

Table 4: Main sources of controlled variation across the six SchemaGUI scenarios.

Figure 5: Illustration of the Geometry score and its three penalty sources.

Scenario	$\mathcal{G}$ at $0.5$	Min. $\mathcal{G}$	Max. $\mathcal{G}$	Range
Calculator	0.51070	0.51060	0.51074	0.00014
Login Form	0.82953	0.82947	0.82953	0.00007
Music Player	0.67393	0.67304	0.67398	0.00094
Doc Editor	0.81896	0.81704	0.81896	0.00191
Image Editor	0.66628	0.66598	0.66661	0.00063
Info Card	0.67729	0.67620	0.67730	0.00110

Appendix

Table 5: Sensitivity of Geometry to the severe-overlap threshold. The threshold is varied from

0.1

0.9

in increments of

0.1

. The symbol

\mathcal{G}

denotes the average Geometry score. Results are averaged over the five evaluated models in both EN and ZH settings, with

1{,}000

samples per model, language, and scenario.

Figure 6: Representative error cases in the Calculator scenario. (a) A schema-infeasible output receives a GUI Score of zero. (b) A schema-feasible prediction suffers from grid collapse, severe out-of-bounds placement, and duplicated controls. (c) A structurally complete and usable prediction retains a relatively high GUI Score despite coordinate and size deviations from the deterministic reference.

Model	Lang.	Calculator	Login Form	Music Player	Doc Editor	Image Editor	Info Card
Qwen3.5-4B -	EN	692–6778	18–3406	18–3574	192–500	291–325	149–238
Qwen3.5-4B -	ZH	112–5741	141–400	239–3649	196–521	297–515	109–189
Qwen3.5-4B +	EN	5433–11715	0–7832	2570–7713	2302–7815	3556–7686	1161–7812
Qwen3.5-4B +	ZH	5242–12480	2438–8735	3172–8521	3741–8418	3615–8194	2334–8951
Qwen3.5-9B -	EN	664–4463	162–525	225–3453	192–620	291–326	149–238
Qwen3.5-9B -	ZH	650–7199	199–579	335–515	278–540	434–596	153–259

Appendix

Table 6: Observed minimum–maximum ranges of raw response-token counts. This table includes all generations, including failed, empty, and timed-out outputs. A minimum of zero indicates that at least one sample produced no recorded raw response tokens. Superscripts - and + denote non-reasoning and reasoning-enabled configurations, respectively.

Model	Lang.	Calculator	Login Form	Music Player	Doc Editor	Image Editor	Info Card	Overall
Qwen3.5-4B -	EN	1278.6	182.3	285.3	254.5	297.7	192.9	415.2
Qwen3.5-4B -	ZH	873.8	194.7	309.0	264.5	324.9	146.7	352.3
Qwen3.5-4B +	EN	193.4	154.2	175.6	220.0	183.6	178.8	184.2
Qwen3.5-4B +	ZH	1293.6	173.3	282.2	322.8	369.7	145.4	431.1
Qwen3.5-9B -	EN	1118.6	185.6	291.1	256.4	298.0	192.9	390.4
Qwen3.5-9B -	ZH	1241.3	278.4	426.1	376.6	447.4	203.5	495.5

Appendix

Table 7: Average clean JSON-token counts for successfully parsed GUI outputs. Clean-token statistics include only valid generations that can be parsed into GUI specifications, while parse-failed, empty, and timed-out outputs are excluded. Overall is the macro-average across the six scenarios.

Model	Render	Fidelity	Layout	Visual	Human
Qwen3.5-4B	72.50	3.86	3.38	2.91	49.06
Qwen3.5-9B	88.30	4.02	3.34	2.68	59.10
Qwen3.5-27B	94.19	4.16	3.93	3.42	72.28
Qwen3-Coder	100.00	4.41	3.69	3.18	75.20
DeepSeek-R1	90.80	3.70	3.33	2.91	60.18

Appendix

Table 8: Human evaluation results across five models. The best and second-best results are highlighted in bold and underlined, respectively.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 396 papers

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI Grounding Multimodal Model Evaluation
Sapienza University of Rome · Warmwind AG
Generative UI: LLMs are Effective UI Generators
Feb 24, 2026Yaniv Leviathan, Dani Valevski, Matan Kalman +10Web Application Generation
Cheenu · Google Research
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3

Oct 22, 2026J/K move · Enter open · S save

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling

Sapienza University of Rome · Warmwind AG

GUI Grounding Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.

Abstract

Language Model Generation Evaluation

Explore similar work

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Generative UI: LLMs are Effective UI Generators

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark