Generating codes from UI screenshots has recently benefited from MLLMs, yet existing methods centered on HTML targets and transfer poorly to multi-frameworks such as React, Vue, and Angular. However, unlike HTML, which does not entail compilation issues, generating executable code in multi-framework settings is more challenging due to framework-specific differences in syntax and failure modes, as well as the need to maintain global consistency across files. To address this problem, we study executable multi-framework front-end code generation and propose MESCoder, a tree-structured self-corrective generation framework. Our method first extracts a structural scaffold from the screenshot and then constructs sub-trees and assembly relations step by step, transforming large project generation into a sequence of localized decisions. On top of this scaffold, we introduce a unified Project Agent that continuously revises its own partial project under runtime sandbox feedback and dynamically chooses between local repair and cluster level repair. For training, we adopt a two-stage strategy that first initializes the policy offline to learn basic multi-framework generation and repair priors, and then refines it with online reinforcement learning to optimize long horizon executability and final page quality. The resulting framework models multi-framework executability as a conditioned project state transition process and improves the stability and quality of complex front end generation through self correction. Experiments show that our method improves executability, while narrowing the visual-fidelity gap to HTML-based generation. Code is available at https://github.com/leo-lilinxiao/MESCoder.
Figures & tables
Figure 1: Compilation failure rates of six state-of-the-art MLLMs under one-shot generation on React, Vue, and Angular. Failures are decomposed into local errors and cluster errors .
Figure 2: The two-stage learning process of MESCoder.
React
Vue
Angular
HTML
Executor
Method
CSR
CLIP
CSR
CLIP
CSR
CLIP
CLIP
GPT-4o
HTML Ref.
–
–
–
–
–
–
0.784
One-Shot
97.3
0.758
94.9
0.769
71.1
0.591
–
MESCoder
99.1
0.766
97.8
0.779
89.2
0.661
–
Claude-3.7-Sonnet
HTML Ref.
–
–
–
–
–
–
0.853
One-Shot
95.4
0.813
97.5
0.827
68.7
0.597
–
Table 1: Main results on DesignBench. HTML Ref. reports the CLIP of the same executor generating a single HTML file, serving as a compilation-free visual reference. Best multi-framework result per executor in bold . All metrics are averaged over three independent runs.
Figure 3: Left : Average CSR on the validation set during the online RL stage under three initialization conditions: behavior-cloning (offline) initialization, hand-coded rule initialization (rule policy used as a parameter seed, then fine-tuned with PPO), and random initialization. Offline initialization converges faster and plateaus higher, confirming the value of the offline prior. Right : Action distribution on stagnated-cluster states before and after RL fine-tuning. RL substantially increases the PatchCluster probability while reducing spurious PatchLocal decisions on these hard states.
Figure 4: Left : CSR gain over Rule Repair per error family (GPT-4o, injected-error subset). Right : Per-cell CSR across framework–error combinations. Angular shows the steepest difficulty gradient.
Variant
Avg CSR
Δ vs. Full
Full MESCoder
95.4
–
Architecture ablations
w/o scaffold (flat generation)
80.8
− 14.6
replace MLP policy w/ rule policy
82.6
− 12.8
w/o PatchCluster action
85.5
− 9.9
w/o Assemble action
89.7
− 5.7
Table 2: Ablation study on average CSR (%) across React, Vue, and Angular (GPT-4o executor). Each row removes or replaces one design component from the full MESCoder system.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Easy CSR
Med CSR
Hard CSR
Easy CLIP
Med CLIP
Hard CLIP
One-Shot
95.1
87.4
77.8
0.712
0.669
0.583
Rule Repair
96.3
90.1
80.3
0.726
0.683
0.601
RLEF
96.8
92.6
85.2
0.741
0.697
0.629
Offline-Only
96.6
81.6
70.0
0.737
0.674
0.598
MESCoder
98.6
95.3
91.7
0.739
0.701
0.632
Appendix
Table 3: Results stratified by page complexity (GPT-4o executor, averaged across React / Vue / Angular). Complexity is defined by scaffold tree depth D and inter-file dependency edge count E .
Method
Hard React
Hard Vue
Hard Angular
One-Shot
88.6
86.3
58.4
Rule Repair
90.2
88.7
62.1
RLEF
93.4
91.8
70.3
Offline-Only
82.6
80.1
47.2
MESCoder
96.3
95.1
83.7
Appendix
Table 4: Hard-tier CSR (%) per framework (GPT-4o executor).
Renames a prop or variable reference to an undefined name within a single component.
Ccont
Component Contract
Changes a component’s exported interface without updating its parent call site.
Casm
Assembly & Hierarchy
Removes or misroutes an import connecting a child component to its parent.
Cmod
Module Dep. & Import
Renames or removes an export without updating downstream import sites, producing missing-export or name-mismatch errors.
Cfw
Framework Specific
Removes an Angular component from @NgModule , or introduces a Vue setup() hook incompatible with Options API.
Appendix
Table 5: Corruption operators used in offline data construction.
Method
Escal. Rate (%)
CSR (%)
Avg Steps
Inv. Patches
Always-Local
0.0
21.7
15.2
6.12
Always-Cluster
100.0
44.8
9.3
1.89
Rule Repair
26.3
39.2
12.4
3.71
Offline-Only
57.8
58.3
10.8
2.43
MESCoder
90.3
76.2
11.3
1.12
Appendix
Table 6: Scope escalation analysis on the stagnated-cluster diagnostic subset.
Method
Metric
React ( n=18 )
Vue ( n=17 )
Angular ( n=52 )
Rule Repair
Escal. Rate (%)
27.8
29.4
25.0
CSR (%)
44.4
47.1
34.6
Offline-Only
Escal. Rate (%)
66.7
64.7
52.3
CSR (%)
72.2
70.6
49.4
MESCoder
Escal. Rate (%)
88.9
88.2
91.4
CSR (%)
88.9
82.4
69.8
Appendix
Table 7: Per-framework escalation rate and CSR on the stagnated-cluster subset (GPT-4o executor). Episode counts reflect the frequency of stagnated-cluster episodes per framework.
Method
Avg CSR
Avg Steps
Sandbox Calls
LLM Calls
Wall Time (s)
One-Shot
87.8
1.0
1.0
1.0
18.3
Rule Repair
89.4
4.1
8.6
4.1
52.7
RLEF
91.8
7.3
7.3
7.3
94.2
Offline-Only
84.1
8.4
8.4
8.4
109.1
MESCoder
95.4
10.2
14.7
10.2
134.6
Appendix
Table 8: Inference efficiency on DesignBench averaged across React / Vue / Angular.
Figure 5: Qualitative rendering results of MESCoder across three UI test cases. For each case, we show the original reference screenshot (Original), an HTML generation baseline (HTML), and MESCoder outputs for three component-based frameworks (React, Vue, Angular). MESCoder faithfully reproduces the reference layout across all target frameworks.
Generating runnable front-end code from UI screenshots is a long-standing goal in automated software engineering. Existing MLLM-based methods predominantly focused on HTML/CSS, leaving multi-framework generation for React/Vue/Angular underexplored. Naively modifying prompts leads to substantial performance gaps across multi-framework and highly framework-specific error modes. To address this, we propose MulFCoder, a framework-conditioned multi-agent method that explicitly encodes framework constraints to bring multi-framework differences into a decidable rule space. MulFCoder orchestrates four agents: Grounder constructs an ElementTable, ContentTable, and macro-layout regions from detected UI elements; Planner builds a DOM-like hierarchical layout tree, produces a task schedule, and derives a framework-specific file Contract; Writer generates structured file writes or patches within a restricted edit window; Judger enforces lightweight, framework-conditioned constraints to accept or reject updates and trigger bounded repairs, preventing drift and deadlocks without expensive builds. Experiments demonstrate that MulFCoder substantially improves compilation success rate and reduces framework-specific errors, with particularly pronounced gains on Angular.
Jie Wu, Haoran Ma, Shisong Tang +3
Communication University of China · Harvard University · Tsinghua university +3
Multimodal large language models (MLLMs) achieve strong performance in translating WebUI screenshots into HTML/CSS. However, when they faced multiple frameworks (React/Vue/Angular), they often suffer from negative transfer that leads to compilation failures. These failures commonly stem from mixing framework specific syntax and violating framework constraints. This paper studies supervised fine-tuning for multi-framework WebUI code generation and proposes a phase-wise MLLM tuning method driven by compatibility and heterogeneity signals across frameworks. We first apply a normalization mapping that preserves each framework's native syntax while suppressing noise from variable naming and literal content. We then estimate cross framework heterogeneity and compatibility offline, and automatically derive training groups and their ordering by maximizing a phased objective. Finally, we perform phase-wise adapter tuning to reduce cross framework interference. Experiments on multiple benchmarks show that our proposed method improves both generation quality and compilation success rate compared with baselines.
Haoran Ma, Linxiao Li, Chenyue Wang +2
Harvard University · The University of Sydney · University of Michigan +1
Code generation with large language models often relies on multi-stage human-in-the-loop refinement, which is effective but very costly - particularly in domains such as frontend web development where the solution quality depends on rendered visual output. We present a fully automated critic-in-the-loop framework in which a vision-language model serves as a visual critic that provides structured feedback on rendered webpages to guide iterative refinement of generated code. Across real-world user requests from the WebDev Arena dataset, this approach yields consistent improvements in solution quality, achieving up to 17.8% increase in performance over three refinement cycles. Next, we investigate parameter-efficient fine-tuning using LoRA to understand whether the improvements provided by the critic can be internalized by the code-generating LLM. Fine-tuning achieves 25% of the gains from the best critic-in-the-loop solution without a significant increase in token counts. Our findings indicate that automated, VLM-based critique of frontend code generation leads to significantly higher quality solutions than can be achieved through a single LLM inference pass, and highlight the importance of iterative refinement for the complex visual outputs associated with web development.
Hannah Sansford, Derek H. C. Law, Wei Liu +3
School of Mathematics, University of Bristol, UK · Amazon AGI