cs.AIApr 13, 2026
SaveWebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
Organizations: Tencent BAC · Tsinghua University
Abstract
Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation that limits scalability. We present WebForge, the first fully automated framework that resolves this trilemma through a four-agent pipeline -- Plan, Generate, Refine, and Validate -- that produces interactive, self-contained web environments end-to-end without human annotation. A seven-dimensional difficulty control framework structures task design along navigation depth, visual complexity, reasoning difficulty, and more, enabling systematic capability profiling beyond single aggregate scores. Using WebForge, we construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels. Multi-model experiments show that difficulty stratification effectively differentiates model capabilities, while cross-domain analysis exposes capability biases invisible to aggregate metrics. Together, these results confirm that multi-dimensional evaluation reveals distinct capability profiles that a single aggregate score cannot capture. Code and benchmark are publicly available at https://github.com/yuandaxia2001/WebForge.
Figures & tables
Figure 1: Overview of the WebForge four-stage automated generation pipeline. The pipeline is executed sequentially by Plan Agent (blue), Generation Agent (green), Refinement Agent (orange), and Validation Agent (purple).
Figure 2: Plan Agent workflow. A dual-LLM generation process: high-temperature drafting ( ) produces diverse creative proposals, followed by low-temperature refinement ( ) for constraint verification and quality enhancement.
Figure 3: Generation Agent workflow. From analyzing the plan blueprint through resource collection to building a complete website with anti-cheating mechanisms.
Figure 4: Refinement Agent workflow. A four-stage process—Assess Quality, Plan Repairs, Execute Improvements, Verify & Deliver—guided by a comprehensive quality rules checklist.
Figure 5: Validation Agent workflow. The agent reads the solution file, replays browser actions in an Observe–Reason–Act loop (up to 50 steps), and produces a solvability verdict by comparing the final result against ground truth. Three failure modes are detected: ground-truth mismatch, reasoning logic flaws, and repeated action failures.
| Difficulty Level | Cross-Domain | ||||||||||
| Model | L1 | L2 | L3 | ALL | D1 | D2 | D3 | D4 | D5 | D6 | D7 |
| (a) Multimodal (Screenshot + DOM) | |||||||||||
| Closed-source | |||||||||||
| Gemini-3-Pro | 86.4 | 82.1 | 58.0 | 75.9 | 72.2 | 67.2 | 82.4 | 79.4 | 71.0 | 76.6 | 80.9 |
| Gemini-3-Flash | 82.4 | 73.5 | 44.0 | 67.1 | 65.2 | 61.6 | 66.4 | 62.5 | 74.0 | 66.0 | 74.8 |
| Gemini-2.5-Flash-Lite | 58.5 | 33.5 | 12.6 | 35.0 | 34.8 | 28.8 | 26.7 | 41.9 | 38.2 | 33.3 | 39.7 |
Table 1: Main results on WebForge-Bench (934 tasks). Left: accuracy (%) by difficulty level. Right: cross-domain accuracy (%). D1: Consumer Transaction/Service, D2: Content Moderation/Compliance, D3: Enterprise Process/Collaboration, D4: Info Retrieval/Analysis, D5: Platform Management/Ops, D6: Tool Usage, D7: Content Creation/Publishing. (a) Multimodal models receive screenshot + DOM. (b) Text-only models receive DOM only; rows marked (T) denote multimodal models evaluated with DOM only.
| Plan Refine | Refine Agent | Pass Rate (%) |
| ✗ | ✗ | 51.4 |
| ✗ | ✓ | 59.5 |
| ✓ | ✓ | 74.1 |
Table 2: Pipeline ablation. Validation pass rate under different pipeline configurations.
| Level 1 | Level 2 | Level 3 | ||||||||||
| Model | Turns | Acts | Prompt | Compl | Turns | Acts | Prompt | Compl | Turns | Acts | Prompt | Compl |
| (a) Multimodal (Screenshot + DOM) | ||||||||||||
| Closed-source | ||||||||||||
| Gemini-3-Pro | 7.9 | 12.2 | 133K | 4.2K | 13.8 | 21.6 | 307K | 5.9K | 26.9 | 44.6 | 1036K | 11.2K |
| Gemini-3-Flash | 8.0 | 12.3 | 159K | 5.5K | 13.1 | 19.3 | 304K | 6.5K | 25.3 | 39.1 | 962K | 15.3K |
| Gemini-2.5-Flash-Lite † | 12.0 | 6.6 | 224K | 4.6K | 16.5 | 11.5 | 254K | 3.4K | 26.1 | 21.9 | 520K | 5.6K |
Table 3: Runtime efficiency metrics on WebForge-Bench, averaged per task by difficulty level. “Turns” = LLM dialogue rounds; “Acts” = browser actions; “Prompt” and “Compl” = input/output tokens (in K). Models marked with † do not support the step-level logging mode, resulting in systematically lower token counts; they are excluded from cross-model efficiency comparisons.
| Jump Depth | Jump Breadth | Page Interact. | Visual Compl. | Info Compl. | Reason./Calc | Risk Factor | |||||||||||||||
| Model | L1 | L2 | L3 | L1 | L2 | L3 | L1 | L2 | L3 | L1 | L2 | L3 | L1 | L2 | L3 | L1 | L2 | L3 | L1 | L2 | L3 |
| (a) Multimodal (Screenshot + DOM) | |||||||||||||||||||||
| Closed-source | |||||||||||||||||||||
| Gemini-3-Pro | 86.5 | 78.9 | 60.2 | 84.8 | 79.9 | 51.2 | 84.0 | 74.9 | 65.0 | 90.8 | 78.9 | 55.8 | 84.7 | 75.7 | 53.2 | 91.4 | 74.6 | 58.3 | 80.6 | 70.3 | 23.1 |
| Gemini-3-Flash | 82.3 | 71.1 | 45.1 | 83.8 | 67.6 | 45.7 | 74.6 | 67.8 | 47.0 | 83.1 | 69.0 | 46.8 | 81.2 | 64.0 | 39.0 | 84.7 | 68.3 | 42.6 | 72.2 | 60.0 | 38.5 |
| Gemini-2.5-Flash-Lite | 57.3 | 33.2 | 13.5 | 56.0 | 34.3 | 13.0 | 52.1 | 33.3 | 9.0 | 54.7 | 34.2 | 13.0 | 50.4 | 28.6 | 13.5 | 56.8 | 31.7 | 12.8 | 42.7 | 23.7 | 0.0 |
Table 4: Per-dimension accuracy (%) on WebForge-Bench. For each of the seven difficulty dimensions, we report accuracy at dimension level L1, L2, and L3. Text-only models and rows marked (T) receive DOM only (no screenshots).
| Benchmark | Type | #Tasks | #Dom. | Diff. Ctrl | Noise | Auto | Reprod. | Eval Paradigm |
|---|---|---|---|---|---|---|---|---|
| Mind2Web [ 6 ] | Real | 2,350 | 137 † | ✗ | ✗ a | ✗ | ✗ b | Step-wise pred. |
| WebVoyager [ 11 ] | Real | 643 | 15 † | ✗ | Passive | ✗ | ✗ | LMM-judge |
| MMInA [ 30 ] | Real | 1,050 | 14 † | ✗ | Passive | ✗ | ✗ | Hop SR |
| WebArena [ 40 ] | Ctrl | 812 | 4 | ✗ | ✗ | ✗ | ✓ | Programmatic |
| VisualWebArena [ 13 ] | Ctrl | 910 | 3 | 2-dim c | ✗ | ✗ | ✓ | Hand-crafted |
| WorkArena++ [ 3 ] | Ctrl | 682 | 1 | 1-dim d | ✗ | ✗ | ✓ | Oracle fn |
Table 5: Comparison with representative browser agent benchmarks. “Diff. Ctrl” indicates whether task difficulty is controllable a priori during generation. “Noise” indicates systematic injection of real-web disturbances (pop-ups, cookie dialogs, network delays). “Auto” indicates fully automated environment and task generation with zero human annotation. “Reprod.” indicates whether the benchmark environment is self-contained and reproducible.
| Dimension | L1 (Easy) | L2 (Medium) | L3 (Hard) |
|---|---|---|---|
| Jump Depth (Page transitions from start to finish) | 1–2 page transitions | 3–5 page transitions | 6+ page transitions |
| Jump Breadth (Max options to evaluate on a single page) | 1–2 links/options | 3–5 links/options | 6+ links/options |
| Page Interaction (Actions per page: forms, popups, buttons) | 1–2 interactions (simple clicks, single field) | 3–5 interactions (multi-field forms, dropdowns) | 6+ interactions (complex forms, multi-step wizards) |
| Visual Complexity (Reliance on visual reasoning) | All information in text/DOM; no visual parsing needed | Simple chart/image reading; single visual element | Complex visual reasoning; correlating multiple charts/images |
| Info Complexity (Volume of information to process) | Key information is prominent and easy to find | Moderate density; requires scanning | High density; long documents; information buried in noise |
| Reasoning/Calc (Reasoning and calculation complexity) | Direct lookup; no calculation needed | Simple arithmetic; basic comparison/filtering | Multi-step reasoning; optimization; complex math |
Table 6: Seven difficulty dimensions and their level definitions. Each dimension captures a distinct axis of task complexity; L1 represents the easiest and L3 the hardest level.
| Dimension | L1 | L2 | L3 |
|---|---|---|---|
| Jump Depth | 288 (30.8%) | 380 (40.7%) | 266 (28.5%) |
| Jump Breadth | 191 (20.4%) | 581 (62.2%) | 162 (17.3%) |
| Page Interaction | 213 (22.8%) | 621 (66.5%) | 100 (10.7%) |
| Visual Complexity | 349 (37.4%) | 284 (30.4%) | 301 (32.2%) |
| Info Complexity | 373 (39.9%) | 420 (45.0%) | 141 (15.1%) |
| Reasoning/Calc | 301 (32.2%) | 398 (42.6%) | 235 (25.2%) |
Table 7: Dimension-level distribution across 934 tasks. Each cell shows count (%).
| Domain | L1 | L2 | L3 | Total |
|---|---|---|---|---|
| D1: Consumer Trans. | 39 (65.0) | 41 (68.3) | 35 (58.3) | 115 (63.9) |
| D2: Content Moder. | 39 (65.0) | 48 (80.0) | 38 (63.3) | 125 (69.4) |
| D3: Enterprise Proc. | 43 (71.7) | 42 (70.0) | 46 (76.7) | 131 (72.8) |
| D4: Info Retrieval | 53 (88.3) | 58 (96.7) | 49 (81.7) | 160 (88.9) |
| D5: Platform Mgmt. | 41 (68.3) | 50 (83.3) | 40 (66.7) | 131 (72.8) |
| D6: Tool Usage | 42 (70.0) | 51 (85.0) | 48 (80.0) | 141 (78.3) |
Table 8: Task count and pipeline pass rate (%) by domain and difficulty level. Each cell shows count ( rate ), where rate = count/60.
Figure 6: Spearman rank correlation matrix ( ) between the seven difficulty dimensions. The average off-diagonal , indicating moderate positive correlation driven by the overall difficulty constraints, while preserving sufficient independence for discriminative profiling.
| Model | J.Dep. | J.Bre. | Pg.Int. | Vis.C. | Inf.C. | Rea.C. | Risk | |
|---|---|---|---|---|---|---|---|---|
| Gemini-3-Pro | L1 | 86.5 | 84.8 | 84.0 | 90.8 | 84.7 | 91.4 | 80.6 |
| L3 | 60.2 | 51.2 | 65.0 | 55.8 | 53.2 | 58.3 | 23.1 | |
| 26.3 | 33.6 | 19.0 | 35.0 | 31.5 | 33.1 | 57.5 | ||
| Gemini-3-Flash | L1 | 82.3 | 83.8 | 74.6 | 83.1 | 81.2 | 84.7 | 72.2 |
| L3 | 45.1 | 45.7 | 47.0 | 46.8 | 39.0 | 42.6 | 38.5 | |
| 37.2 | 38.1 | 27.6 | 36.3 | 42.2 | 42.2 | 33.7 |
Table 9: Per-dimension accuracy drop (L1 L3, in pp) for three representative models. Larger = more discriminative.
| Aspect | Draft Plan | Refined Plan |
|---|---|---|
| Pricing tiers | 3 (Red/Yellow/Green) | 4 (Red/Orange/Yellow/Green) |
| Venue rental (Yellow) | $3,500 | $3,200 |
| Catering (Premium) | $85/pp | $90/pp |
| Service fee | None | 10% of subtotal |
| Saturday preference | Not mentioned | Explicitly required |
| Valid dates | May 15–18 (4 dates) | May 15–19 (5 dates, 1 Saturday) |
Table 10: Draft vs. Refined Plan comparison for the wedding venue booking task.
Figure 7: Cedar Lakes Estate generated homepage (upper-page comparison). The top of the real Cedar Lakes Estate homepage and the top of the generated CelebrationVenues homepage exhibit the same high-level homepage prior: a cinematic hero image, restrained luxury branding, and a structure that foregrounds visual atmosphere before denser task information.
Figure 8: Cedar Lakes Estate generated homepage (lower-page comparison). The bottom of the reference homepage and the bottom of the generated homepage both place substantial visual weight on lower-page promotional content and a prominent footer/contact region, showing that the Generation Agent transferred not only hero styling but also lower-page layout rhythm and brand treatment.
Figure 9: WeddingWire generated search results page. The Generation Agent borrows the search-results interaction pattern from WeddingWire: left-aligned thumbnails, stacked result cards, star-rating cues, price-tier markers, and strong venue-selection affordances. This is the clearest evidence that the generated site learned a retrieval/listing UI pattern from a real wedding marketplace.
Figure 10: MND Farm Westerlo generated informational page. Here the transfer signal is mainly about editorial tone and venue storytelling . The real site uses a soft estate narrative with image-backed property presentation; the generated CelebrationVenues page echoes this through a mission/story section, estate-style imagery, and a polished informational layout rather than a purely transactional interface.
| Category | Count | Details |
|---|---|---|
| HTML pages | 8 | index , search , venue_overview , venue_pricing , |
| venue_flora , venue_book , venue_review , venue_confirmation | ||
| Image assets | 10 | Scene photos (5), flower photos (3), data charts (2) |
| Code & data files | 3 | style.css , main.js , data.json |
| Total files | 21 |
Table 11: Generation Agent output summary. The agent produces a complete web environment comprising HTML pages, visual assets, and supporting code/data files.
Figure 11: All 10 image assets produced by the Generation Agent. Top row: scene and catering photographs used across venue pages. Bottom row: decorative flower images and two data-driven visualizations (a seasonal bloom chart and a pricing/availability heatmap). These images are produced via three tools: Banana (text-to-image generation), web image retrieval, and programmatic Python chart generation.
| Metric | Before (Gen. Agent) | After (Ref. Agent) |
|---|---|---|
| HTML pages | 8 | 18 (+10 new pages) |
| Image assets | 10 | 10 (unchanged) |
| Code & data files | 3 | 3 (unchanged) |
| Total files | 21 | 31 |
| Dead navigation links | Yes (Blog, About, Contact) | Resolved (all links functional) |
| alert() dialogs | Present (blocks DOM parsing) | Replaced with inline errors |
Table 12: Before vs. After Refinement Agent. The Refinement Agent expands the website from 21 to 31 files while injecting real-web noise and fixing solvability issues.
Figure 12: Blog page created by the Refinement Agent. The Generation Agent’s navigation bar included a “Blog” link that led nowhere. The Refinement Agent created a fully functional blog page with categorized articles (“Outdoor,” “Planning,” “Inspiration”), a sidebar with category filters and a newsletter subscription widget, and consistent site-wide navigation. Similar pages were created for About Us, Contact, Careers, Login, Register, Privacy Policy, and Terms of Service.
Figure 13: Alert dialog replacement. Left : The Generation Agent’s alert() dialog blocks DOM parsing, making the page unresponsive to browser agents. Right : The Refinement Agent’s inline error message (“ Please enter a contact name.”) appears as styled DOM content below the form field, maintaining full page interactivity and ensuring task solvability for automated agents.
Figure 14: Real-web noise injection. The “Schedule a Private Tour” promotional popup (center) appears after a stochastic 5–15s delay, while the cookie consent banner (bottom) appears 1s after page load. Together they simulate the real-world browsing distractions that agents must handle gracefully.
Figure 15: Generic venue placeholder page. When agents click “View Details” on non-target venues, they now see this “Content Unavailable” page instead of encountering a dead link. The page provides clear navigation options (“Contact Support,” “Back to Search”) and suggests similar venues, maintaining a realistic user experience while guiding agents back to productive paths.
| Condition | Code Returned | Correct? |
|---|---|---|
| Correct (May 16, 80 guests, Premium) | GEG-2026-05841 | ✓ |
| Valid date but not Saturday (May 15/17/18/19) | GEG-2026-05842 | |
| Wrong catering selection | GEG-2026-05991 | |
| Wrong guest count | GEG-2026-05118 | |
| Invalid date (outside May 15–19) | GEG-2026-05294 |
Table 13: Deceptive code mapping. Each common mistake pattern returns a unique, plausible-looking confirmation code. Only the fully correct booking yields the ground-truth code.
More in this topic
Web Agent Benchmarks
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.
See all 58 papers