WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development

Shanchao Liang; Nan Jiang; Shangshu Qian; Lin Tan

cs.SEOct 24, 2024

WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development

Authors: Shanchao Liang, Nan Jiang, Shangshu Qian, Lin Tan

Organizations: Purdue University

Abstract

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.

Figures & tables

Figure 1: Removing the children of the element <div id = "left-column"> highlighted in yellow does not affect the structure of the visual layout of itself or its sibling element <div id="right-column"> .

Figure 2: Overview of Waffle , including training data mutation, structure-aware attention, and contrastive learning.

CSS						HTML	Total
Color	Size	Margin	Font	Display	Position	HTML	Total
12	11	19	10	1	2	8	63

Table 1: Most frequent causes of failures.

Figure 3: Example of structure-aware attention. Yellow : self-attention. Green : parent-attention. Teal : sibling-attention. Purple : masked-out Attention.

Figure 4: Different metrics on a pair of images.

Backbones	Techniques	WebSight-Test				Design2Code
Backbones	Techniques	HTML-Match (%) $\uparrow$	CW-SSIM $\uparrow$	CLIP $\uparrow$	LLEM (%) $\uparrow$	CW-SSIM $\uparrow$	CLIP $\uparrow$	LLEM (%) $\uparrow$
Gemini 1.5 Pro	Prompting	9.40	0.3385	88.55	90.16	0.2652	87.76	87.17
GPT-4o mini	Prompting	10.20	0.3055	87.72	87.54	0.2304	86.06	78.84
GPT-4o	Prompting	11.40	0.3666	89.03	92.18	0.2776	83.67	75.98
Moondream2	Standard FT	21.60	0.4233	89.92	90.59	0.1348	46.63	40.71
Moondream2	Waffle	27.60	0.4486	89.98	91.72	0.2142	79.62	67.83

Table 2: Main results on the WebSight-Test and Design2Code dataset.

Figure 5: Example from WebSight-Test dataset, with the generated images by GPT-4o, Standard FT (VLM-WebSight), and Waffle (VLM-WebSight).

Backbones	Techniques	WebSight-Test				Design2Code
Backbones	Techniques	HTML-Match (%) $\uparrow$	CW-SSIM $\uparrow$	CLIP $\uparrow$	LLEM (%) $\uparrow$	CW-SSIM $\uparrow$	CLIP $\uparrow$	LLEM (%) $\uparrow$
Moondream2	Standard FT	21.60	0.4233	89.92	90.59	0.1348	46.63	40.71
	Waffle ${}_{\texttt{-attn}}$	23.60	0.4311	90.47	91.34	0.1821	67.73	56.49
	Waffle ${}_{\texttt{-contra}}$	26.00	0.4296	89.55	91.21	0.2100	76.63	65.82
	Waffle	27.60	0.4486	89.98	91.72	0.2142	79.62	67.83
VLM-WebSight	Standard FT	28.00	0.5023	93.30	92.73	0.2518	82.35	73.00

Table 3: Ablation studies on the two test datasets. LLEM refers to the averaged Low-Level Element Matching.

Techniques	Rank 1 $\uparrow$	Rank 2 $\uparrow$	Rank 3 $\uparrow$	Avg Rankings $\downarrow$
Standard FT	0 9 \| 34 (43)	18 \| 13 (46)	34 \| 25 (54)	2.88 \| 2.37 (2.62)
Waffle ${}_{\texttt{-attn}}$	22 \| 23 (45)	0 11 \| 30 (41)	22 \| 26 (71)	2.67 \| 2.45 (2.56)
Waffle ${}_{\texttt{-contra}}$	60 \| 33 (93)	0 13 \| 17 (30)	11 \| 26 (42)	1.78 \| 2.41 (2.10)
Waffle	46 \| 54 ( 100 )	33 \| 22 (55)	10 \| 0 15 (26)	1.85 \| 1.79 ( 1.82 )

Table 4: Human evaluation on two datasets using VLM-WebSight as the backbone. The numbers are shown as “ x | y (x+y)”, where x is the result on WebSight-Test and y is the result on Design2Code.

Techniques	Prior	Current	Drop (%)
Waffle ${}_{\texttt{-attn}}$	0.8002	0.5797	27.55
Waffle	0.8291	0.7932	4.34

Table 5: CW-SSIM on 20 samples using the VLM-WebSight backbone. “Prior” refers to “without intermediate mistakes”, and “Current” to “with intermediate mistakes”.

Techniques	$d(\overline{v^{i}},\overline{t^{i}})\downarrow$	$sim(\overline{v^{i}},\overline{t^{i}})\uparrow$
Standard FT	1.3395	0.1027
Waffle ${}_{\texttt{-attn}}$	0.8447	0.6244

Table 6: Distance (

d

) and similarity (

sim

) between averaged image embeddings

\overline{v^{i}}

and text embeddings

\overline{t^{i}}

, using Moondream2 as the backbone.

Figure 6: t-SNE plots of the text and image embeddings, computed by Moondream2 fine-tuned with Standard FT and Waffle

{}_{\texttt{-attn}}

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Class	Failure Type	Specification
CSS	Color	Random Color in Range [ #000000, #FFFFFF ]
	Size	Random Size in [0, 500] pixels
	Margin	Random Size in [0, 100] pixels
	Font	Random Size in [0, 40] pixels
	Display	Random Keyword for text-align , display ,
		flex-direction , and justify-content

Appendix

Table 7: Specification for Mutation Rules to construct the Contrastive dataset.

Figure 7: Existing MLLM generates identical HTML and CSS code in (b) given the different webpage screenshots in (a) and (c). The MLLM fails to generate the correct 2fr highlighted in red in (d).

Figure 8: Illustration of the tuning process of the parameter that controls the effect of structure-aware attention. In (b), the green line almost overlaps with the blue line.

Techniques	$d(\overline{v^{i}},c)\uparrow$	$sim(\overline{v^{i}},c_{g})\downarrow$
Standard FT	0.1224	0.9910
Waffle ${}_{\texttt{-attn}}$	0.7590	0.6202

Appendix

Table 8: Distance (

d

) and similarity (

sim

) between each averaged image embeddings

\overline{v^{i}}

with the corresponding centroid

c

of the group of mutants, with Moondream2 backbone.

Figure 9: Selected Attention Weight Visualization of Waffle on a simplified example.

Figure 10: Figures of Design2Code task 42.

Figure 11: Figures of WebSight-Test task 45.

Image-to-Code Generation

See all 17 papers

EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation
Sep 15, 2025Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan +3Efficient Multimodal Inference Code Generation
The Chinese University of Hong Kong, Hong Kong, China · Huazhong University of Science and Technology, Wuhan, China · Singapore Management University, Singapore, Singapore +1
Vision-Guided Iterative Refinement for Frontend Code Generation
Apr 7, 2026Hannah Sansford, Derek H. C. Law, Wei Liu +3Code Generation Vision-Language Model-as-a-Judge
School of Mathematics, University of Bristol, UK · Amazon AGI
Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
Mar 27, 2026Zehai He, Wenyi Hong, Zhen Yang +4Web Agent Benchmarks Web Application Generation
Tsinghua University · Zhipu AI

Sep 15, 2025J/K move · Enter open · S save

EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation

Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan, Yintong Huo+2

The Chinese University of Hong Kong, Hong Kong, China · Huazhong University of Science and Technology, Wuhan, China · Singapore Management University, Singapore, Singapore+1

Efficient Multimodal Inference Code Generation Multimodal Token Compression Visual Token Compression

Open paper AskPDF arXiv

Multimodal Large Language Models have demonstrated exceptional performance in UI2Code tasks, significantly enhancing website development efficiency. However, these tasks incur substantially higher computational overhead than traditional code generation due to the large number of input image tokens and extensive output code tokens required. Our comprehensive study identifies significant redundancies in both image and code tokens that exacerbate computational complexity and hinder focus on key UI elements, resulting in excessively lengthy and often invalid HTML files. We propose EfficientUICoder, a compression framework for efficient UI code generation with three key components. First, Element and Layout-aware Token Compression preserves essential UI information by detecting element regions and constructing UI element trees. Second, Region-aware Token Refinement leverages attention scores to discard low-attention tokens from selected regions while integrating high-attention tokens from unselected regions. Third, Adaptive Duplicate Token Suppression dynamically reduces repetitive generation by tracking HTML/CSS structure frequencies and applying exponential penalties. Extensive experiments show EfficientUICoder achieves a 55%-60% compression ratio without compromising webpage quality and delivers superior efficiency improvements: reducing computational cost by 44.9%, generated tokens by 41.4%, prefill time by 46.6%, and inference time by 48.8% on 34B-level MLLMs. Code is available at https://github.com/WebPAI/EfficientUICoder.

Techniques	Rank 1 $\uparrow$	Rank 2 $\uparrow$	Rank 3 $\uparrow$	Avg Rankings $\downarrow$
Standard FT	0 9 \| 34 (43)	18 \| 13 (46)	34 \| 25 (54)	2.88 \| 2.37 (2.62)
Waffle ${}_{\texttt{-attn}}$	22 \| 23 (45)	0 11 \| 30 (41)	22 \| 26 (71)	2.67 \| 2.45 (2.56)
Waffle ${}_{\texttt{-contra}}$	60 \| 33 (93)	0 13 \| 17 (30)	11 \| 26 (42)	1.78 \| 2.41 (2.10)
Waffle	46 \| 54 ( 100 )	33 \| 22 (55)	10 \| 0 15 (26)	1.85 \| 1.79 ( 1.82 )

Abstract

Image-to-Code Generation

Explore similar work

EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation

Vision-Guided Iterative Refinement for Frontend Code Generation

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification