cs.SEOct 2, 2026
SaveWebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness
Organizations: Columbia University · Meta SuperIntelligence Labs
Abstract
Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan--act--observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures.
Figures & tables
Figure 1: Examples of WebUI outputs in WebUIProof , spanning general applications (landing pages, code IDE, social/profile pages, games) and 3D simulation interfaces (physics dynamics, planetary scenes). These examples illustrate the benchmark’s emphasis on both visual fidelity and interaction-driven functionality. Each task is evaluated with executable interaction tests that check user actions (click/type/drag) and the resulting UI behavior.
Figure 2: Benchmark instance format of WebUIProof . We show a structured spec. with paired test cases. Our benchmark construction pipeline first lets the UI agent collect a sequence of visual snapshots and HTML/DOM structure from the given seed webpage. Based on observations from webpages, the UI agent provides a website summary and test case recommendations. The red and blue boxes illustrate element-to-test alignment: a subset of entries in "element_requirement" is grouped and paired with executable test cases, making element-level coverage explicit.
| Benchmark | #Instr. | Instr. Form | Instr. Type | Instr. Length | Output Scope | UI Topic | UI Agent Eval. | #Tests | TCR(%) |
| DesignBench Xiao et al. (2025) | 900 | NL prompts | Gen./Edit/Repair | 80-150 | Single file (HTML) | General | ✗ | 0 | 0% |
| WebDev-Arena Zheng et al. (2024) | 10k | NL prompts | Gen. | 60 | Repo (App) | General | ✗ | 0 | 0% |
| FrontendBench Zhu et al. (2025) | 148 | NL prompts | Gen. | 50-100 | Single file (HTML) | General | ✓ | 148 | 90% |
| WebGen-Bench Liu et al. (2024b) | 101 | NL prompts | Gen. | 40-70 | Repo (App) | General | ✓ | 600 | 49.24% |
| WebUIProof (Ours) | 219 | Structured Spec. | Gen. | 270-550 | Repo (App) | General+3D | ✓ | 3k | 97.23% |
Table 1: Benchmarks comparison for WebUI code generation. We compare the benchmark based on instruction number, form, type, length, and output scope. UI Topic indicates general WebUIs (dashboards/tools/games) or additionally includes 3D simulation interfaces (classical mechanics/particles simulation). UI Agent Eval. denotes whether the evaluation involves a UI agent that executes test suites. Test coverage rate ( TCR ) measures element-level coverage. Our WebUIProof provides diverse UI topics (general+3D) with rich test suites and high TCR.
Figure 3: VisRL Overview : Given instruction prompts, the Policy LLM generates UI code candidates, which are built and rendered to produce screenshots; rewards combination (i) code validity with error fixing loop, (ii) vision-based screenshot scoring, and (iii) UI-agent interaction tests (element detection, task proposal, and execution success), and are used to update the policy via PPO.
| Model Source | Model Name / Training Variant | Build Failure Rate (%) | Build Success – UI-Agent Verified Task Outcome (%) | Accuracy (%) (Full + 0.5 Partial) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Proprietary (API) | Claude Sonnet 4 Anthropic (2025) | 30.37 | 12.59 | 22.10 | 35.05 | 46.10 |
| Gemini 2.5 Pro DeepMind (2025) | 32.21 | 15.02 | 20.38 | 32.67 | 42.86 | |
| Qwen 3 Coder 480B Alibaba Cloud (2025) | 33.05 | 13.92 | 20.76 | 31.62 | 41.99 | |
| DeepSeek R1 DeepSeek (2025) | 32.89 | 14.91 | 20.51 | 31.65 | 41.90 | |
| GPT 4.1 OpenAI (2025) | 36.58 | 11.67 | 20.30 | 31.25 | 41.40 | |
Table 2: General WebUI generation on WebUIProof. We report the build failure rate (% of generations that fail to build/render), and the UI-agent verified task outcome among build-successful generations: No/Partial/Full completion rate (%). Overall Accuracy is computed as , where build failures are counted as No completion. Rows are grouped into eight proprietary (API) models and two self-trained models, including Qwen2.5 14B Instruct and MIMO 7B under three training variants: SFT, PPO w/ vision agent, and VisRL (ours).
| Model Source | Model Name / Training Variant | Build Failure Rate (%) | Build Success – UI-Agent Verified Task Outcome (%) | Accuracy (%) (Full + 0.5 Partial) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Proprietary (API) | Claude Sonnet 4 Anthropic (2025) | 0.00 | 49.66 | 20.21 | 30.14 | 40.24 |
| Qwen 3 Coder 480B Alibaba Cloud (2025) | 1.43 | 51.37 | 17.12 | 29.11 | 37.67 | |
| Gemini 2.5 Pro DeepMind (2025) | 0.00 | 54.79 | 18.49 | 26.71 | 35.96 | |
| Kimi K2 Moonshot (2025) | 0.00 | 54.79 | 20.21 | 25.00 | 35.10 | |
| Llama 4 Maverick Meta (2025) | 0.00 | 59.93 | 15.75 | 24.43 | 32.19 | |
Table 3: 3D simulation WebUI generation on WebUIProof. Same protocol as Table 2 : we report build failure rate and UI-agent verified task outcomes (%), including No/Partial/Full completion rate, and compute Accuracy, counting build failures as No completion. Results compare eight proprietary (API) models and self-trained models, including Qwen2.5 14B Instruct and MIMO 7B under three training variants: SFT baseline, PPO w/ vision agent, and VisRL (ours).
Figure 6: UI-agent execution examples on WebUIProof. Top: general WebUIs (e.g., interactive dashboards and widgets). Bottom: 3D simulation WebUIs (e.g., physics-driven interactive scenes). In all cases, the UI agent executes interaction tests by grounding to DOM/UI elements and performing actions (click/type/drag) to validate functional behavior.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Next.js project structure
| Action Type | Detail |
| Click | Click any web element identified by its numerical label (e.g., Click [0]) |
| Type | Clear textbox and input content (automatically hits ENTER) (e.g., Type [0]) |
| Scroll | Scroll specific element or entire window (e,g, Scroll WINDOW up / down) |
| Hover | Hover for tooltips/effects, with precise coordinate positioning for charts (e.g., Hover [0:75,50] for 75% from left, 50% from top) |
| Wait | Pause for 5 seconds (for loading processes) |
| GoBack | Navigate to previous webpage |
Table 7: Action space of UI agent
Figure 8: System prompt for General WebUI task. Models generate React code generation with strict styling and output constraints.
Figure 9: System prompt for Three.js HTML simulation generation.
Figure 10: WebUI Category Distribution for WebUIProof.
Figure 11: Test Case Distribution for WebUIProof.
Figure 12: Benchmark Collection Pipeline.
Figure 13: Multimodal prompt (instructions + screenshots + HTML) used for WebUI test-case generation.
Figure 14: Visualization of SFT Training Curve
Figure 15: Training Infrastructure of VisRL.
Figure 16: Comparison of PPO training dynamics with and without a UI agent. Top: average reward (with 7-step moving average). Bottom: vision score and invalid action rate.
Figure 17: Radar plot for reward decompsition analysis
Figure 18: Sensitivity analysis of Configuration
| Model Name | Build Failure Rate (%) | Test Category | Build Sucess UI-Agent Verified Task (%) | Accuracy (Cat. %) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Claude Sonnet 4 Anthropic (2025) | Functionality Testing | 10.15 | 23.86 | 30.28 | 42.21 | |
| 30.37 | Data Display Testing | 15.04 | 15.76 | 43.41 | 51.29 | |
| UI Design Validation | 18.52 | 29.63 | 37.03 | 51.85 | ||
| Gemini 2.5 Pro DeepMind (2025) | Functionality Testing | 16.15 | 19.44 | 22.47 | 32.19 | |
| 32.21 | Data Display Testing | 12.89 | 17.19 | 51.58 | 60.17 | |
Table 10: Category accuracy of General WebUI task on Proprietary API.
| Training Variants of Qwen 2.5 14B Instruct | Build Failure Rate (%) | Test Category | Build Sucess UI-Agent Verified Task (%) | Accuracy (Cat. %) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Baseline SFT | Functionality Testing | 7.75 | 10.71 | 20.66 | 26.01 | |
| 56.25 | Data Display Testing | 14.33 | 14.33 | 21.49 | 28.65 | |
| UI Design Validation | 18.52 | 11.11 | 18.52 | 24.07 | ||
| PPO W/ Vision agent | Functionality Testing | 12.87 | 11.05 | 34.38 | 39.90 | |
| 41.42 | Data Display Testing | 14.33 | 7.30 | 28.65 | 32.31 | |
Table 11: Category accuracy of General WebUI task on Self-Trained Model.
| Model Name | Build Failure Rate (%) | Test Category | Build Sucess UI-Agent Verified Task (%) | Accuracy (Cat. %) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Claude Sonnet 4 Anthropic (2025) | Functionality Testing | 53.15 | 16.79 | 30.08 | 38.47 | |
| 0.00 | Data Display Testing | 21.43 | 42.86 | 35.71 | 57.14 | |
| UI Design Validation | 68.35 | 6.33 | 25.32 | |||
| Qwen 3 Coder 480B Alibaba Cloud (2025) | Functionality Testing | 55.25 | 18.87 | 27.27 | 36.71 | |
| 1.43 | Data Display Testing | 40.00 | 25.71 | 34.29 | 47.14 | |
Table 12: Category accuracy of 3D WebUI task on Proprietary API.
| Training Variants of Qwen 2.5 14B Instruct | Build Failure Rate (%) | Test Category | Build Sucess UI-Agent Verified Task (%) | Accuracy (Cat. %) | ||
| No Completion | Partial Completion | Full Completion | ||||
| Baseline SFT | Functionality Testing | 52.16 | 12.82 | 19.74 | 26.16 | |
| 0.00 | Data Display Testing | 48.57 | 25.71 | 25.71 | 38.57 | |
| UI Design Validation | 68.35 | 12.66 | 18.99 | 25.32 | ||
| PPO w/ Vision agent | Functionality Testing | 59.38 | 27.97 | 20.96 | 34.95 | |
| 0.00 | Data Display Testing | 21.43 | 35.71 | 28.57 | 46.43 | |
Table 13: Category accuracy of 3D WebUI task on Self-Trained Model.
Figure 19: WebUI interaction examples (sample 1 and 2). Each row shows a full UI state transition with agent-labeled elements and executed actions.
Figure 20: WebUI interaction examples (sample 3 to 5). Each row shows a full UI state transition with agent-labeled elements and executed actions.
Figure 21: Six simulation samples. Each subfigure shows the UI state and interaction annotation for one sample.
Figure 22: We present three webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left-most webpage is a music band showcase portal. The upper-right webpage is a codecraft editor. The lower-right webpage is a AI website showdown.
Figure 23: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. Both of them are the social media viewer.
Figure 24: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left side is a 2048 game interface and the right side is a chess game simulator.
Figure 25: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. Both of them are the stock report generator.
Figure 26: Three webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left side is a travel accommodation finder, the upper-right side is an AI agent workflow manager, and the lower-right side is a landing page for tech. company.
Figure 27: Four webpage demo generated from self-trained model Qwen 2.5 14B w/ VisRL. We provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D animation of a circular island that has multiple trees on it and a river flowing within. (2.) The upper-right demo: Create a 3D simulation of jelly cubes bouncing in a box. (3.) The lower-left demo: Create a 3D aquarium with fishes that swim around and avoid collisions with the tank walls. (4.) The lower-right demo: Create a 3D animation of a sail boat floating in the sea.
Figure 28: Same as prior figure, we provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D simulation of a mars rover moving on the surface of Mars. Users should be able to accelerate and control the direction of the car. Incorporate accurate suspension dynamics as the car moves on the rugged Mars surface. (2.) The upper-right demo: Create a visually engaging animation of a double pendulum using JavaScript. The animation should demonstrate the chaotic motion of two pendulums connected together. (3.) The lower-left demo: Simulate 3D paper lanterns rising into the night sky during a festival, with buoyancy and wind physics, above a reflective lake. (4.) The lower-right demo: Create a realistic 3D ocean scene with: dynamic water movement, accurate wave simulations, and reflective surface effects.
Figure 29: Same as prior figure, we provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D solar system simulation using Three.js with realistic planet orbits, textures, and lighting effects. (2.) The upper-right demo: Create a breakout-style game where a paddle at the bottom of the screen reflects a bouncing ball toward a grid of colored blocks above. The paddle moves left and right with arrow keys, and blocks disappear when hit by the ball. Each block is worth different points based on its color, and the game ends when all blocks are cleared or the ball falls below the paddle. (3.) The lower-left demo: Create a 3D animation of a top spinning. Use accurate physics to demonstrate its movements. (4.) The lower-right demo: Create a dinosaur running game where: (i) A dinosaur character can jump using spacebar/up arrow, (ii) Cacti and birds appear as obstacles from the right side, (iii) Ground moves continuously to create running effect (iv) Game ends when dinosaur hits an obstacle, (v) Simple pixel art style, and (vi) Day/night cycle every 500 points.
Figure 30: Breakdown of build-time errors across models in our WebUI generation benchmark. Bars show the number of failures by error type, including missing dependencies, syntax mistakes, reference issues, and invalid default exports. Higher stacks indicate more frequent build failures.
More in this topic
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 313 papers