Organizations: HKUST (Guangzhou), Guangzhou, China · Alibaba Group, Hangzhou, China · HKUST, Hong Kong, China · University of Electronic Science, Chengdu, China · Zhejiang Lab, Hangzhou, China
While Vision Language Models (VLMs) have shown promise in Design-to-Code generation, they suffer from a "holistic bottleneck-failing to reconcile high-level structural hierarchy with fine-grained visual details, often resulting in layout distortions or generic placeholders. To bridge this gap, we propose DOne, an end-to-end framework that decouples structure understanding from element rendering. DOne introduces (1) a learned layout segmentation module to decompose complex designs, avoiding the limitations of heuristic cropping; (2) a specialized hybrid element retriever to handle the extreme aspect ratios and densities of UI components; and (3) a schema-guided generation paradigm that bridges layout and code. To rigorously assess performance, we introduce HiFi2Code, a benchmark featuring significantly higher layout complexity than existing datasets. Extensive evaluations on the HiFi2Code demonstrate that DOne outperforms exiting methods in both high-level visual similarity (e.g., over 10% in GPT Score) and fine-grained element alignment. Human evaluations confirm a 3 times productivity gain with higher visual fidelity.
Figures & tables
Figure 1: Reconstruction comparison (a-b) and illustrative examples of baseline failures (c-d).
Figure 2: The DOne Framework. DOne employs layout segmentation and visual element detection, followed by schema-guided code generation to ensure structural and visual fidelity.
Metric
Min
Max
Avg
Boxes/page
2
7
3.53
Elements/page
1
64
4.66
Elements/box
0
16
1.11
Tokens/box
0
1518
69.97
Box-to-page Ratio
0%
99%
25%
Table 1: Statistics of WebSeg.
Metric
Design2Code
Web2Code
HiFi2Code
Avg. boxes/page
3.88
2.84
5.32
Avg. elements/page
2.59
1.43
14.39
Avg. elements/box
0.53
0.51
2.80
Element coverage
81.4%
66.9%
100%
Large elements
20.7%
17.7%
44.2%
Table 2: Comparison of evaluation datasets.
Figure 3: Example of layout segmentation.
Figure 4: Example of visual element detection.
Table 7
Configuration
CLIP
GPT
Direct Generation
0.6475
-
+ Layout Segmentation
0.7226
0.701
+ Element Detection
0.6975
0.656
+ Schema Guidance
0.6615
0.554
+ All Techniques ( DOne )
0.7435
0.723
Table 5: Ablation study on DOne components.
Method
Vote Percentage
S2C
15.5%
CoT2C
23.3%
DCGen
2.60%
DOne
58.6%
Table 6: User preference distribution.
Figure 5: Pairwise win/tie/loss rates vs. S2C.
Metric
Control
DOne (Exp)
Time (min)
9.81
3.18
CLIP Score
0.499
0.665
Table 7: Average efficiency and fidelity metrics.
Figure 6: Task completion time distribution.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Value
mAP 50:95
0.5316
mAP 50
0.8203
Precision
0.7763
Recall
0.7293
Appendix
Table 8: Performance metrics
Model
Large Objects
Small Objects
Overall
CO-DETR
0.6823
0.7545
0.7356
YOLOv10
0.6139
0.7914
0.7212
CO-DETR+YOLOv10
0.6751
0.7812
0.7512
Appendix
Table 9: Comparison of mAP 75 across different model combinations.
Figure 8: Prompt of schema generation.
Figure 9: Prompt of code generation with schema.
Figure 10: Qualitative comparison of generated webpage results.