Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks. We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples. Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages. Agents must selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining their outputs. We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges. Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.
Figures & tables
Figure 1: Overview of RefCraft. (a) RefCraft spans a broad spectrum of practical web development domains and features visual references that vary in intensity and source. (b) An example task in which a new webpage borrows the color palette from source webpage A and the layout from source webpage B. The agent may iteratively render, inspect, and revise the webpage. (c) Overall score versus reported token usage for nine frontier models. The dashed line denotes the Pareto frontier. The results show that the task remains far from solved.
Figure 2: Construction and evaluation of RefCraft. (a) Reference webpages are captured across interaction states. (b) A specification spanning six visual dimensions and five requirement types guides natural-language query synthesis. (c) Task requirements are decomposed into program-verifiable and VLM-assessed keypoints. (d) Given the query and reference images, agents iteratively build, render, inspect, and revise a webpage in a shared harness. (e) Keypoints guide evidence collection from generated artifacts. (f) Complementary programmatic and VLM judges produce the base keypoint score B , core-function score C , and reference-following score R .
Table 1: Results on the 300-task RefCraft benchmark. We report Overall and its three component scores: keypoint completion (Keypts., B ), core functionality (Core, C ), and reference following (Ref., R ). Rollout statistics report mean agent turns, total tokens (Total Tok.), and tool calls per task. Bold and underlined values indicate the best and second-best entries in each column, respectively (higher scores and lower resource counts).
Figure 3: Fine-grained performance by (a) visual dimension and (b) requirement type, averaged equally across models.
Table 5
Figure 4: Two failure cases identified by RefCraft: one in reference following and the other in core functionality.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Composition and diversity of RefCraft.
Model A
Model B
Δ
A win (%)
Tie (%)
B win (%)
GPT-6 Astra
Qwen3.8-Max
3.17
38.3
24.7
37.0
GPT-6 Astra
Grok 4.6
3.29
42.7
21.7
35.7
GPT-6 Astra
Kimi K3
4.57
44.0
20.7
35.3
GPT-6 Astra
Claude Fable 5
7.51
46.7
20.7
32.7
GPT-6 Astra
GLM-5.3
9.52
47.3
23.7
29.0
GPT-6 Astra
DeepSeek-V4-Pro
15.03
51.0
25.3
23.7
Appendix
Table 4: Paired comparisons among the nine main-table models. Δ is Model A minus Model B.
Figure 6: Judge sensitivity on the fixed 32-task sample. Colored points and connecting lines show scores assigned by Gemini 3.1 Pro, GPT-6 Astra, and Claude Opus 5. Whiskers indicate illustrative uncertainty intervals. Relative scoring patterns are broadly consistent across the three models.
Figure 7: Protocol sensitivity and evidence coverage. (a) Changes in Overall score relative to the complete protocol: +42.3 under holistic rating and −4.6 under screenshot-only evaluation. (b) Observed rates of unresolved core cases across matched artifacts.
Model
Harness
Overall score
GPT-6 Astra
OpenCode (released)
49.2
GPT-6 Astra
Claude Code
51.3
GPT-6 Astra
Codex
53.0
Kimi K3
OpenCode (released)
48.2
Kimi K3
Claude Code
49.4
Kimi K3
Codex
43.1
Appendix
Table 7: Cross-harness sensitivity on the fixed 32-task sample. We report Overall scores across three harnesses.
Figure 8: Generated user query examples. English requests from the final task set illustrate casual, bulleted, and structured PRD forms. Task 132 is shown in full; Tasks 279 and 161 are excerpted, with [...] marking omissions. Original wording is retained.
Figure 9: An end-to-end development trajectory on Task 163. Boxes and enlarged screenshot crops highlight reference cues, responsive-layout revisions, and changes in the quote and booking state.
Figure 10: Prompt for user-query generation. Across all prompt figures, [...] denotes omitted passages. Formatting and line breaks are adjusted for presentation.
Figure 11: Instruction template for expert keypoint review. Reviewers check coverage, evidence grounding, reference-use consistency, and verifiability, then return revised keypoints with reasons for changes and unresolved issues.
Figure 12: Keypoint-construction prompt excerpts.
Figure 13: Keypoint-review prompt excerpts. The global pass checks coverage, and the local pass reviews individual items against the reference-use specification and supporting evidence.
Figure 14: Perceptual-judging prompt excerpts. The judge must record expected and observed values and their deviation before applying the four-level rubric.
Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited. To address this gap, we introduce Vision2Web, a hierarchical benchmark for visual website development, spanning from static UI-to-code generation, interactive multi-page frontend reproduction, to long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. To support flexible, thorough and reliable evaluation, we propose workflow-based agent verification paradigm based on two complementary components: a GUI agent verifier and a VLM-based judge. We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels, with state-of-the-art models still struggling on full-stack development.
We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents. Unlike prior code generation benchmarks that focus on algorithmic tasks, VISTA targets realistic UI-centric development, where agents must produce functional, visually coherent applications from underspecified inputs. We define five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint: (1) text only with free stack choice, (2) text with reference screenshots under three specified stacks, (3) text with reference screenshots under free stack choice, (4) text with screenshots and pruned Figma structure under a single specified stack, and (5) text with screenshots and pruned Figma structure under free stack choice. To enable robust evaluation, each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points, addressing the well-known limitations of script-based testing tools such as Playwright in open-ended code generation settings. Evaluation combines DOM-grounded reference matching, behavior-specific browser tests, and CLIP-based visual similarity, jointly measuring structural alignment, behavioral completeness, and overall visual fidelity. We use VISTA to assess four agent systems drawn from two model families and two harnesses, finding that visual fidelity and functional correctness are partially decoupled across both input conditions and agents, and that agent editing style varies sharply but is largely orthogonal to task quality. VISTA establishes a rigorous and reproducible foundation for advancing agent-based software engineering research. Code is available at https://github.com/kaboider/VISTA_Bench.
The rapid progress of Artificial Intelligence Generated Content (AIGC) tools enables images, videos, and visualizations to be created on demand for webpage design, offering a flexible and increasingly adopted paradigm for modern UI/UX. However, directly integrating such tools into automated webpage generation often leads to style inconsistency and poor global coherence, as elements are generated in isolation. We propose MM-WebAgent, a hierarchical agentic framework for multimodal webpage generation that coordinates AIGC-based element generation through hierarchical planning and iterative self-reflection. MM-WebAgent jointly optimizes global layout, local multimodal content, and their integration, producing coherent and visually consistent webpages. We further introduce a benchmark for multimodal webpage generation and a multi-level evaluation protocol for systematic assessment. Experiments demonstrate that MM-WebAgent outperforms code-generation and agent-based baselines, especially on multimodal element generation and integration. Code & Data: https://aka.ms/mm-webagent.
Yan Li, Zezi Zeng, Yifan Yang +12
Shanghai Jiao Tong University · Xi’an Jiaotong University · Microsoft Corporation +1