WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
Authors: Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Jun Du, Wenchong Zeng, Han Li, Kun Gai
Organizations: Northeastern University · Kuaishou Technology
The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at https://github.com/friedrichor/WebTestBench.
Figures & tables
Benchmark
#Sample
Task
Env
Checklist
Inte.
L.H.
Func.
Cons.
Test Method
Application Quality Evaluation Methods Originate from Web Generation Tasks
Web2Code ( Yun et al., 2024 )
60K
Generation
Web
✗
✗
✗
✗
✗
Visual
Interaction2Code ( Xiao et al., 2024 )
127
Generation
Web
✗
✓
✗
✓
✗
Visual+Rule
Design2Code ( Si et al., 2025 )
484
Generation
Web
✗
✗
✗
✗
✗
Visual
DesignBench ( Xiao et al., 2025 )
900
Generation
Web
✗
✗
✗
✗
✗
Visual+Rule+MLLM
WebGen-Bench ( Lu et al., 2025b )
101
Generation
Web
✗
✓
✓
✓
✗
Agent
Table 1: Comparison with representative related benchmarks across several aspects: the number of samples ( #Sample ); the primary task ( Task ); the app environment ( Env ); whether the evaluation supports end-to-end testing without manually predefined test items ( Checklist ); whether testing requires dynamic interaction with the app ( Inte. ); whether it involves long - horizon interactions ( L.H. ); whether functional correctness is evaluated ( Func. ); whether latent logical constraints are evaluated ( Cons. ); and the approach used to execute tests ( Test Method ).
Figure 1: Overview of WebTestBench construction.
Figure 2: An illustrative example from WebTestBench. The web application is built from a given development instruction, typically comprising multiple pages with rich interactive functionality. The benchmark provides a gold checklist across four quality dimensions: Functionality, Constraint, Interaction, and Content. Each test case is annotated with a binary Pass/Fail verdict.
Model
#Turns
#Tokens
Functionality
Constraint
Interaction
Content
Overall
Cov.
F1
Cov.
F1
Cov.
F1
Cov.
F1
Cov.
P
R
F1
Open-Source LLMs
Minimax-M2.1
41.7
3.58M
77.9
12.3
40.4
15.8
42.2
19.9
47.1
7.7
60.1
22.3
14.6
15.2
Qwen3-Coder-Next
63.4
6.24M
77.6
14.1
48.3
23.8
42.7
11.4
35.9
4.3
60.4
27.8
15.8
17.3
GLM-4.7
41.6
3.47M
79.9
16.5
47.3
20.5
36.0
17.2
48.9
4.3
61.1
26.7
16.6
18.1
GLM-5
41.3
3.71M
79.7
11.9
50.1
26.9
41.4
20.9
50.6
3.4
63.1
30.4
15.6
19.0
Table 2: Web testing performance of representative LLM on WebTestBench under the WebTester framework. We report the Coverage metric (Cov.) for checklist generation, and the average number of iteration turns (#Turns), average context tokens (#Tokens) per instance, and Precision/Recall/F1 metrics for defect detection. Results in bold and underline denote the best and second-best performances.
Model
Presentation
Search
Tool
Commerce
DM
Workflow
UGC
Minimax M2.1
25.6/10.3/14.3
18.9/6.9/9.4
12.3/12.6/11.7
10.9/10.1/8.6
36.4/23.4/25.3
18.4/12.3/12.8
27.2 /20.3/17.8
Qwen3 Coder Next
16.4/16.4/14.3
21.5/11.1/13.7
33.8/12.2/16.6
26.8/16.8/19.1
42.6/20.1/23.2
18.4/15.8/15.0
27.0 /15.9/16.4
GLM 4.7
22.4/12.7/15.6
19.4/26.5/20.0
17.8/7.8/9.8
35.9 /19.1/24.1
41.8/22.1/24.7
22.6/15.7/16.3
21.7/16.2/16.3
GLM 5
30.8/16.7/21.1
11.1/4.6/6.5
24.7/11.4/14.0
33.6/15.4/19.6
38.0/20.7/24.6
40.1/23.3/26.8
23.6/9.9/12.7
Step 3.5 Flash
38.6/13.2/18.8
24.1 /13.7/17.1
37.2 /22.2/25.3
34.3/ 22.5 / 25.6
47.9 / 33.8 / 35.6
27.2/19.2/19.0
24.3/12.0/14.6
MiMo-V2-Flash
23.7/14.2/16.5
26.6 / 34.1 / 26.0
42.1 /28.1/ 30.7
35.5/20.2/21.6
44.1 /33.2/ 34.1
44.1 /24.0/28.4
13.8/16.1/10.6
Table 3: Category-wise evaluation results on WebTestBench. Each cell contains “Precision/Recall/F1”. Results in bold and underline denote the best and second-best performances.
Model
Kendall’s τ
Spearman’s ρ
Pearson r
GPT-5-Mini
68.8
75.0
78.9
GPT-5
76.6
82.7
85.7
Qwen3.5-27B
78.5
83.2
87.1
Table 4: Human judgment correlation scores for our automatic evaluation. All p -values <0.05 .
Figure 3: Comparison of perfermance between the end-to-end (E2E) setting and the oracle setting for representative models. In the oracle setting, the gold checklist is directly provided to the defect detection agent, decoupling detection performance from checklist generation quality.
Figure 4: Performance comparison across different web complexity in the oracle setting.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Instructions & Web Applications
Total Samples
100
Presentation
13
Search
9
Tool
17
Commerce
13
Appendix
Table 5: Overall statistics of WebTestBench.
GPT-5.1
Avg. Playwright Calls: 23.23
Tool
Calls
browser_click
14.69
browser_fill_form
2.14
browser_snapshot
1.81
browser_wait_for
1.41
Appendix
Table 6: Statistics of Playwright tool calls per instance for representative models.
Category
Details
Presentation
Description: Presents information in a clear, structured form, , enabling users to browse, read, and access content efficiently.
Core Functionalities: Content layout and typography, navigation structure, pagination and indexing, media rendering, basic search and filtering.
Typical Examples: Blog and article websites, personal homepages, corporate websites, documentation and wikis, portfolios, and news portals.
Search
Description: Supports search, filtering, ranking, and recommendation over heterogeneous datasets, emphasizing information discovery and matching.
Core Functionalities: Search input and query syntax, filter and sort controls, result list presentation, recommendation-based content delivery, result detail pages.
Typical Examples: Job platforms, real estate listing websites, academic literature search engines, content recommendation systems.
Appendix
Table 7: Taxonomy of web application categories used in WebTestBench.
Dimension
Details
Functionality
Description: Focuses on whether the application can successfully complete the primary tasks and business flows specified in the development instruction under normal usage conditions.
Evaluation Focus: Successful completion of core operations (create, view, edit, delete, submit); correct responses to normal user requests (search, filter, navigate, form submission); coherent cross-module flows and reachable navigation.
Typical Examples: A user can submit a search query and view the returned results; a user can create a new project and see it reflected in the list.
Constraint
Description: Focuses on whether the application is robust under abnormal or boundary usage, including input validation, error handling, and prevention of invalid or conflicting states.
Evaluation Focus: Effective validation of required fields with clear error messages; blocking of conflicting or illegal operations (e.g., double-booking, editing non-existent records); graceful handling of boundary conditions (empty lists, excessively long inputs, invalid formats); consistency between interface state and underlying data.
Typical Examples: An already-reserved meeting room cannot be booked again; submitting a form with empty required fields is blocked with a clear prompt.
Appendix
Table 8: Taxonomy of test case dimensions used in WebTestBench.
Model
Functionality
Constraint
Interaction
Content
Cov.
P
R
F1
Cov.
P
R
F1
Cov.
P
R
F1
Cov.
P
R
F1
Minimax M2.1
77.9
15.9
14.0
12.3
40.4
19.9
14.4
15.8
42.2
22.6
20.3
19.9
47.1
8.6
7.3
7.7
Qwen3 Coder Next
77.6
17.5
14.9
14.1
48.3
27.5
23.1
23.8
42.7
12.4
12.8
11.4
35.9
5.1
3.9
4.3
GLM 4.7
79.9
21.0
16.8
16.5
47.3
26.5
18.4
20.5
36.0
18.9
17.6
17.2
48.9
5.1
3.9
4.3
GLM 5
79.7
16.8
11.0
11.9
50.1
32.4
26.5
26.9
41.4
24.2
20.3
20.9
50.6
5.1
2.6
3.4
Step 3.5 Flash
79.8
28.1
18.7
20.1
53.6
32.6
27.4
27.9
48.5
24.2
21.2
21.2
60.6
2.6
2.6
2.6
Appendix
Table 9: Detailed quality dimension-wise evaluation results on WebTestBench.
Model
Presentation
Search
Tool
Commerce
Cov.
P
R
F1
Cov.
P
R
F1
Cov.
P
R
F1
Cov.
P
R
F1
Minimax M2.1
56.9
25.6
10.3
14.3
52.0
18.9
6.9
9.4
58.7
15.2
13.7
13.4
57.1
10.9
10.1
8.6
Qwen3 Coder Next
62.5
16.4
16.4
14.3
62.7
21.5
11.1
13.7
56.1
33.8
12.2
16.6
66.1
26.8
16.8
19.1
GLM 4.7
59.5
22.4
12.7
15.6
63.9
19.4
26.5
20.0
62.4
17.8
7.8
9.8
63.4
35.9
19.1
24.1
GLM 5
64.5
30.8
16.7
21.1
61.6
11.1
4.6
6.5
64.0
24.7
11.4
14.0
64.9
33.6
15.4
19.6
Step 3.5 Flash
65.7
38.6
13.2
18.8
68.4
24.1
13.7
17.1
61.4
37.2
22.2
25.3
72.8
34.3
22.5
25.6
Appendix
Table 10: Detailed category-wise evaluation results on WebTestBench.
Model
Functionality
Constraint
Interaction
Content
Overall
MiMo-V2-Flash
22.2/20.1/19.3
50.5/49.7/46.6
33.3/31.1/30.3
16.2/16.7/15.8
44.3/30.4/31.9
Step-3.5-Flash
33.5/25.6/26.4
61.4/56.9/55.6
39.2/36.9/35.9
30.3/33.3/31.2
53.4/38.7/40.9
GPT-5.1
42.8/54.8/42.0
57.0/76.5/59.2
55.9/65.7/56.0
36.5/46.2/38.2
47.0/63.4/46.8
Claude Sonnet 4.5
45.0/32.3/34.5
69.1/72.6/66.7
44.8/42.8/41.4
32.3/32.7/30.7
61.0/47.4/49.2
Appendix
Table 11: Detailed results on WebTestBench in the oracle setting.
Figure 5: An example illustrating how CUA interacts with a web application during defect detection.
Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.
Chenxu Liu, Zilu Zou, Peizhong Gao +9
Hunyuan Team, Tencent · Tsinghua University · Peking University
Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale. Existing automated proxies typically lean on reference implementations, test suites, or rigid checklists, and tend to miss the reasoned synthesis a human reviewer performs over a live session. We articulate a new evaluation regime that is simultaneously reference-free, autonomously driven, and holistically reasoned, and instantiate it through two artifacts. \dataname is an 11-domain, 54-leaf, 1,000-query WebDev benchmark spanning both static-presentation and interactive-application tasks, balanced across three difficulty tiers and three target-language groups, with briefs rewritten to resist recall from circulated prompts. \framename, grounded in Flavell's metacognitive monitoring, separates evidence accumulation from judgment across three stages: Static Perception forms a first impression from passive observation; Agent-Driven Interaction explores the application autonomously while capturing continuous screen video, audio, and per-step screenshots; Dynamic Scoring issues holistic functionality and aesthetics verdicts with structured failure attribution only after the evidence chain is complete. On \dataname, \framename aligns closely with expert human ratings while surfacing substantial headroom across 13 frontier LLMs on interactive web generation. \noindenthttps://anonymous.4open.science/r/Cookie-3CE/
Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited. To address this gap, we introduce Vision2Web, a hierarchical benchmark for visual website development, spanning from static UI-to-code generation, interactive multi-page frontend reproduction, to long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. To support flexible, thorough and reliable evaluation, we propose workflow-based agent verification paradigm based on two complementary components: a GUI agent verifier and a VLM-based judge. We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels, with state-of-the-art models still struggling on full-stack development.