cs.CLOct 19, 2026

RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development

Authors: Yuchi Wang, Rongyao Fang, Weihao Yu, Xilin Wei, Tiancheng Gu, Dingkang Yang, Hang Zhang, Dayiheng Liu, 

Organizations: MMLab, CUHK · Alibaba Token Foundry · The Chinese Univerity of Hong Kong · Fudan University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks. We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples. Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages. Agents must selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining their outputs. We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges. Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.

Figures & tables

Figure 1: Overview of RefCraft. (a) RefCraft spans a broad spectrum of practical web development domains and features visual references that vary in intensity and source. (b) An example task in which a new webpage borrows the color palette from source webpage A and the layout from source webpage B. The agent may iteratively render, inspect, and revise the webpage. (c) Overall score versus reported token usage for nine frontier models. The dashed line denotes the Pareto frontier. The results show that the task remains far from solved.
Figure 2: Construction and evaluation of RefCraft. (a) Reference webpages are captured across interaction states. (b) A specification spanning six visual dimensions and five requirement types guides natural-language query synthesis. (c) Task requirements are decomposed into program-verifiable and VLM-assessed keypoints. (d) Given the query and reference images, agents iteratively build, render, inspect, and revise a webpage in a shared harness. (e) Keypoints guide evidence collection from generated artifacts. (f) Complementary programmatic and VLM judges produce the base keypoint score BB , core-function score CC , and reference-following score RR .
Table 1: Results on the 300-task RefCraft benchmark. We report Overall and its three component scores: keypoint completion (Keypts., BB ), core functionality (Core, CC ), and reference following (Ref., RR ). Rollout statistics report mean agent turns, total tokens (Total Tok.), and tool calls per task. Bold and underlined values indicate the best and second-best entries in each column, respectively (higher scores and lower resource counts).
Figure 3: Fine-grained performance by (a) visual dimension and (b) requirement type, averaged equally across models.
Table 5
Figure 4: Two failure cases identified by RefCraft: one in reference following and the other in core functionality.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Composition and diversity of RefCraft.
Model AModel BΔ\DeltaA win (%)Tie (%)B win (%)
GPT-6 AstraQwen3.8-Max3.1738.324.737.0
GPT-6 AstraGrok 4.63.2942.721.735.7
GPT-6 AstraKimi K34.5744.020.735.3
GPT-6 AstraClaude Fable 57.5146.720.732.7
GPT-6 AstraGLM-5.39.5247.323.729.0
GPT-6 AstraDeepSeek-V4-Pro15.0351.025.323.7
Appendix
Table 4: Paired comparisons among the nine main-table models. Δ\Delta is Model A minus Model B.
Figure 6: Judge sensitivity on the fixed 32-task sample. Colored points and connecting lines show scores assigned by Gemini 3.1 Pro, GPT-6 Astra, and Claude Opus 5. Whiskers indicate illustrative uncertainty intervals. Relative scoring patterns are broadly consistent across the three models.
Figure 7: Protocol sensitivity and evidence coverage. (a) Changes in Overall score relative to the complete protocol: +42.3+42.3 under holistic rating and −4.6-4.6 under screenshot-only evaluation. (b) Observed rates of unresolved core cases across matched artifacts.
ModelHarnessOverall score
GPT-6 AstraOpenCode (released)49.249.2
GPT-6 AstraClaude Code51.351.3
GPT-6 AstraCodex53.053.0
Kimi K3OpenCode (released)48.248.2
Kimi K3Claude Code49.449.4
Kimi K3Codex43.143.1
Appendix
Table 7: Cross-harness sensitivity on the fixed 32-task sample. We report Overall scores across three harnesses.
Figure 8: Generated user query examples. English requests from the final task set illustrate casual, bulleted, and structured PRD forms. Task 132 is shown in full; Tasks 279 and 161 are excerpted, with [...] marking omissions. Original wording is retained.
Figure 9: An end-to-end development trajectory on Task 163. Boxes and enlarged screenshot crops highlight reference cues, responsive-layout revisions, and changes in the quote and booking state.
Figure 10: Prompt for user-query generation. Across all prompt figures, [...] denotes omitted passages. Formatting and line breaks are adjusted for presentation.
Figure 11: Instruction template for expert keypoint review. Reviewers check coverage, evidence grounding, reference-use consistency, and verifiability, then return revised keypoints with reasons for changes and unresolved issues.
Figure 12: Keypoint-construction prompt excerpts.
Figure 13: Keypoint-review prompt excerpts. The global pass checks coverage, and the local pass reviews individual items against the reference-use specification and supporting evidence.
Figure 14: Perceptual-judging prompt excerpts. The judge must record expected and observed values and their deviation before applying the four-level rubric.
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,486 papers

Explore similar work

CardsList

  1. Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

    Mar 27, 2026Zehai He, Wenyi Hong, Zhen Yang +4Web Agent BenchmarksWeb Application Generation

    Tsinghua University · Zhipu AI

  2. VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

    May 22, 2026JunJia Guo, Yuhang Yao, Jiawei +2Web Application GenerationWeb Agent Benchmarks

    Joe · University of Arizona · Zoom +1

  3. MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation

    Apr 16, 2026Yan Li, Zezi Zeng, Yifan Yang +12Web Application GenerationWeb Agent Benchmarks

    Shanghai Jiao Tong University · Xi’an Jiaotong University · Microsoft Corporation +1