Multimodal large language models (MLLMs) achieve strong performance in translating WebUI screenshots into HTML/CSS. However, when they faced multiple frameworks (React/Vue/Angular), they often suffer from negative transfer that leads to compilation failures. These failures commonly stem from mixing framework specific syntax and violating framework constraints. This paper studies supervised fine-tuning for multi-framework WebUI code generation and proposes a phase-wise MLLM tuning method driven by compatibility and heterogeneity signals across frameworks. We first apply a normalization mapping that preserves each framework's native syntax while suppressing noise from variable naming and literal content. We then estimate cross framework heterogeneity and compatibility offline, and automatically derive training groups and their ordering by maximizing a phased objective. Finally, we perform phase-wise adapter tuning to reduce cross framework interference. Experiments on multiple benchmarks show that our proposed method improves both generation quality and compilation success rate compared with baselines.
Figures & tables
Framework
Train
Val
Test
Avg. Tokens
Vanilla
2,814
312
300
1,023
React
2,773
308
295
1,187
Vue
2,681
298
284
1,142
Angular
2,594
288
277
1,309
Total
10,862
1,206
1,156
1,165
Table 1: Dataset statistics per framework.
Method
Vanilla
React
Vue
Angular
Avg.
CLIP
CSR
CLIP
CSR
CLIP
CSR
CLIP
CSR
CLIP
CSR
Claude-3.7
0.7832
1.0000
0.7891
0.9492
0.7934
0.6655
0.6223
0.6029
0.7470
0.8044
GPT-4o
0.7635
1.0000
0.6763
0.9119
0.7734
0.6092
0.5867
0.5379
0.7000
0.7648
Gemini-2.0
0.7412
1.0000
0.7611
0.8407
0.6997
0.1408
0.6006
0.4838
0.7007
0.6163
Llama-3.2-11B
0.6304
1.0000
0.6401
0.4915
0.3241
0.5810
0.5834
0.5415
0.5445
0.6535
Qwen2.5-VL-7B
0.7411
1.0000
0.6875
0.1085
0.6452
0.1197
0.5186
0.0866
0.6481
0.3287
Table 2: Results on DesignBench Generation task. Best results are bold ; second best are underlined .
Method
Block Match
Text
Position
Color
CLIP
GPT-4o
93.0
98.2
85.5
84.1
90.4
Claude-3.7
90.2
97.5
77.9
71.4
87.0
GPT-4V
85.8
97.4
80.5
73.3
86.9
Design2Code-18B
78.5
96.4
74.3
67.0
85.8
ScreenCoder
84.3
97.1
79.2
75.6
87.3
EfficientUICoder
81.7
96.8
77.4
72.3
86.1
Table 3: Results on Design2Code benchmark. Best results are bold ; second best are underlined .
Table 4Table 5Figure 6
Figure 3: Qualitative code generation examples stratified by ( Δ , S ). Each row shows our method (center, green border) vs. Shared+Specific no-gate baseline (right, red border).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Layer
CLIP (Avg.)
CSR (Avg.)
Synergy Std.
Layer 8
0.7763±0.0079
0.8714±0.0061
0.0082
Layer 10
0.7798±0.0065
0.8741±0.0051
0.0071
Layer 12
0.7878±0.0049
0.8802±0.0043
0.0063
Layer 14
0.7924±0.0033
0.8917±0.0030
0.0041
Layer 15 (ours)
0.7950±0.0027
0.9007±0.0024
0.0031
Layer 18
0.7823±0.0038
0.8771±0.0035
0.0048
Appendix
Table 8: Effect of gradient computation layer (mean ± std over 3 seeds).
Distance Function
CLIP (Avg.)
CSR (Avg.)
Offline Comp.
MMD
0.7912±0.0033
0.8901±0.0029
1.4×
Sinkhorn-Wasserstein
0.7931±0.0030
0.8921±0.0028
3.7×
Ours (JS Divergence)
0.7950±0.0027
0.9007±0.0024
1.0×
Appendix
Table 9: Discrepancy distance function ablation (mean ± std over 3 seeds).
Normalization
CLIP (Avg.)
CSR (Avg.)
Weak (format only)
0.7812
0.8747
Strong (remove framework syntax)
0.7769
0.8694
Ours (moderate)
0.7950
0.9007
Appendix
Table 10: Code normalization strength ablation.
Method
Vanilla
React
Vue
Angular
Avg.
Claude-3.7
7.42
7.83
7.61
5.94
7.20
GPT-4o
7.23
7.41
7.39
5.61
6.91
Qwen2.5-VL-7B
6.81
5.12
5.34
1.03
4.58
ScreenCoder
7.34±0.08
7.52±0.11
7.48±0.09
6.12±0.14
7.12±0.10
EfficientUICoder
7.12±0.09
7.23±0.13
7.19±0.11
5.87±0.16
6.85±0.12
Ours
7.68±0.07
7.94±0.09
7.87±0.08
6.89±0.12
7.60±0.09
Appendix
Table 11: MLLM-as-Judge scores on the DesignBench Editing task (0–10 scale; mean ± std over 3 seeds for fine-tuned methods).
Method
Vanilla
React
Vue
Angular
Avg.
Claude-3.7
7.91
8.12
7.98
7.03
7.76
GPT-4o
7.78
7.93
7.82
6.87
7.60
Qwen2.5-VL-7B
7.34
6.23
6.47
2.14
5.55
ScreenCoder
7.62±0.08
7.81±0.10
7.74±0.09
6.58±0.13
7.44±0.10
EfficientUICoder
7.43±0.10
7.61±0.12
7.52±0.11
6.34±0.15
7.23±0.12
Ours
8.03±0.06
8.24±0.08
8.17±0.07
7.52±0.11
7.99±0.08
Appendix
Table 12: MLLM-as-Judge scores on the DesignBench Repair task (0–10 scale; mean ± std over 3 seeds for fine-tuned methods).
Generating runnable front-end code from UI screenshots is a long-standing goal in automated software engineering. Existing MLLM-based methods predominantly focused on HTML/CSS, leaving multi-framework generation for React/Vue/Angular underexplored. Naively modifying prompts leads to substantial performance gaps across multi-framework and highly framework-specific error modes. To address this, we propose MulFCoder, a framework-conditioned multi-agent method that explicitly encodes framework constraints to bring multi-framework differences into a decidable rule space. MulFCoder orchestrates four agents: Grounder constructs an ElementTable, ContentTable, and macro-layout regions from detected UI elements; Planner builds a DOM-like hierarchical layout tree, produces a task schedule, and derives a framework-specific file Contract; Writer generates structured file writes or patches within a restricted edit window; Judger enforces lightweight, framework-conditioned constraints to accept or reject updates and trigger bounded repairs, preventing drift and deadlocks without expensive builds. Experiments demonstrate that MulFCoder substantially improves compilation success rate and reduces framework-specific errors, with particularly pronounced gains on Angular.
Jie Wu, Haoran Ma, Shisong Tang +3
Communication University of China · Harvard University · Tsinghua university +3
Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.
Generating codes from UI screenshots has recently benefited from MLLMs, yet existing methods centered on HTML targets and transfer poorly to multi-frameworks such as React, Vue, and Angular. However, unlike HTML, which does not entail compilation issues, generating executable code in multi-framework settings is more challenging due to framework-specific differences in syntax and failure modes, as well as the need to maintain global consistency across files. To address this problem, we study executable multi-framework front-end code generation and propose MESCoder, a tree-structured self-corrective generation framework. Our method first extracts a structural scaffold from the screenshot and then constructs sub-trees and assembly relations step by step, transforming large project generation into a sequence of localized decisions. On top of this scaffold, we introduce a unified Project Agent that continuously revises its own partial project under runtime sandbox feedback and dynamically chooses between local repair and cluster level repair. For training, we adopt a two-stage strategy that first initializes the policy offline to learn basic multi-framework generation and repair priors, and then refines it with online reinforcement learning to optimize long horizon executability and final page quality. The resulting framework models multi-framework executability as a conditioned project state transition process and improves the stability and quality of complex front end generation through self correction. Experiments show that our method improves executability, while narrowing the visual-fidelity gap to HTML-based generation. Code is available at https://github.com/leo-lilinxiao/MESCoder.
Linxiao Li, Haoran Ma, Chenyue Wang +3
The University of Sydney · Harvard University · Meituan +2