WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing

Fanheng Kong; Jingyuan Zhang; Yang Yue; Chenxi Sun; Yang Tian; Shi Feng; Xiaocui Yang; Daling Wang

cs.SEMar 26, 2026

WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing

Authors: Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang,

Organizations: Northeastern University · Kuaishou Technology

Abstract

The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at https://github.com/friedrichor/WebTestBench.

Figures & tables

Benchmark	#Sample	Task	Env	Checklist	Inte.	L.H.	Func.	Cons.	Test Method
Application Quality Evaluation Methods Originate from Web Generation Tasks
Web2Code ( Yun et al., 2024 )	60K	Generation	Web	✗	✗	✗	✗	✗	Visual
Interaction2Code ( Xiao et al., 2024 )	127	Generation	Web	✗	✓	✗	✓	✗	Visual+Rule
Design2Code ( Si et al., 2025 )	484	Generation	Web	✗	✗	✗	✗	✗	Visual
DesignBench ( Xiao et al., 2025 )	900	Generation	Web	✗	✗	✗	✗	✗	Visual+Rule+MLLM
WebGen-Bench ( Lu et al., 2025b )	101	Generation	Web	✗	✓	✓	✓	✗	Agent

Table 1: Comparison with representative related benchmarks across several aspects: the number of samples ( #Sample ); the primary task ( Task ); the app environment ( Env ); whether the evaluation supports end-to-end testing without manually predefined test items ( Checklist ); whether testing requires dynamic interaction with the app ( Inte. ); whether it involves long - horizon interactions ( L.H. ); whether functional correctness is evaluated ( Func. ); whether latent logical constraints are evaluated ( Cons. ); and the approach used to execute tests ( Test Method ).

Figure 1: Overview of WebTestBench construction.

Figure 2: An illustrative example from WebTestBench. The web application is built from a given development instruction, typically comprising multiple pages with rich interactive functionality. The benchmark provides a gold checklist across four quality dimensions: Functionality, Constraint, Interaction, and Content. Each test case is annotated with a binary Pass/Fail verdict.

Model	#Turns	#Tokens	Functionality		Constraint		Interaction		Content		Overall
Model	#Turns	#Tokens	Cov.	F1	Cov.	F1	Cov.	F1	Cov.	F1	Cov.	P	R	F1
Open-Source LLMs
Minimax-M2.1	41.7	3.58M	77.9	12.3	40.4	15.8	42.2	19.9	47.1	7.7	60.1	22.3	14.6	15.2
Qwen3-Coder-Next	63.4	6.24M	77.6	14.1	48.3	23.8	42.7	11.4	35.9	4.3	60.4	27.8	15.8	17.3
GLM-4.7	41.6	3.47M	79.9	16.5	47.3	20.5	36.0	17.2	48.9	4.3	61.1	26.7	16.6	18.1
GLM-5	41.3	3.71M	79.7	11.9	50.1	26.9	41.4	20.9	50.6	3.4	63.1	30.4	15.6	19.0

Table 2: Web testing performance of representative LLM on WebTestBench under the WebTester framework. We report the Coverage metric (Cov.) for checklist generation, and the average number of iteration turns (#Turns), average context tokens (#Tokens) per instance, and Precision/Recall/F1 metrics for defect detection. Results in bold and underline denote the best and second-best performances.

Model	Presentation	Search	Tool	Commerce	DM	Workflow	UGC
Minimax M2.1	25.6/10.3/14.3	18.9/6.9/9.4	12.3/12.6/11.7	10.9/10.1/8.6	36.4/23.4/25.3	18.4/12.3/12.8	27.2 /20.3/17.8
Qwen3 Coder Next	16.4/16.4/14.3	21.5/11.1/13.7	33.8/12.2/16.6	26.8/16.8/19.1	42.6/20.1/23.2	18.4/15.8/15.0	27.0 /15.9/16.4
GLM 4.7	22.4/12.7/15.6	19.4/26.5/20.0	17.8/7.8/9.8	35.9 /19.1/24.1	41.8/22.1/24.7	22.6/15.7/16.3	21.7/16.2/16.3
GLM 5	30.8/16.7/21.1	11.1/4.6/6.5	24.7/11.4/14.0	33.6/15.4/19.6	38.0/20.7/24.6	40.1/23.3/26.8	23.6/9.9/12.7
Step 3.5 Flash	38.6/13.2/18.8	24.1 /13.7/17.1	37.2 /22.2/25.3	34.3/ 22.5 / 25.6	47.9 / 33.8 / 35.6	27.2/19.2/19.0	24.3/12.0/14.6
MiMo-V2-Flash	23.7/14.2/16.5	26.6 / 34.1 / 26.0	42.1 /28.1/ 30.7	35.5/20.2/21.6	44.1 /33.2/ 34.1	44.1 /24.0/28.4	13.8/16.1/10.6

Table 3: Category-wise evaluation results on WebTestBench. Each cell contains “Precision/Recall/F1”. Results in bold and underline denote the best and second-best performances.

Model	Kendall’s $\tau$	Spearman’s $\rho$	Pearson $r$
GPT-5-Mini	68.8	75.0	78.9
GPT-5	76.6	82.7	85.7
Qwen3.5-27B	78.5	83.2	87.1

Table 4: Human judgment correlation scores for our automatic evaluation. All

p

-values

<0.05

Figure 3: Comparison of perfermance between the end-to-end (E2E) setting and the oracle setting for representative models. In the oracle setting, the gold checklist is directly provided to the defect detection agent, decoupling detection performance from checklist generation quality.

Figure 4: Performance comparison across different web complexity in the oracle setting.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Value
Instructions & Web Applications
Total Samples	100
Presentation	13
Search	9
Tool	17
Commerce	13

Appendix

Table 5: Overall statistics of WebTestBench.

GPT-5.1
Avg. Playwright Calls: 23.23
Tool	Calls
browser_click	14.69
browser_fill_form	2.14
browser_snapshot	1.81
browser_wait_for	1.41

Appendix

Table 6: Statistics of Playwright tool calls per instance for representative models.

Category	Details
Presentation	Description: Presents information in a clear, structured form, , enabling users to browse, read, and access content efficiently.
	Core Functionalities: Content layout and typography, navigation structure, pagination and indexing, media rendering, basic search and filtering.
	Typical Examples: Blog and article websites, personal homepages, corporate websites, documentation and wikis, portfolios, and news portals.
Search	Description: Supports search, filtering, ranking, and recommendation over heterogeneous datasets, emphasizing information discovery and matching.
	Core Functionalities: Search input and query syntax, filter and sort controls, result list presentation, recommendation-based content delivery, result detail pages.
	Typical Examples: Job platforms, real estate listing websites, academic literature search engines, content recommendation systems.

Appendix

Table 7: Taxonomy of web application categories used in WebTestBench.

Dimension	Details
Functionality	Description: Focuses on whether the application can successfully complete the primary tasks and business flows specified in the development instruction under normal usage conditions.
	Evaluation Focus: Successful completion of core operations (create, view, edit, delete, submit); correct responses to normal user requests (search, filter, navigate, form submission); coherent cross-module flows and reachable navigation.
	Typical Examples: A user can submit a search query and view the returned results; a user can create a new project and see it reflected in the list.
Constraint	Description: Focuses on whether the application is robust under abnormal or boundary usage, including input validation, error handling, and prevention of invalid or conflicting states.
	Evaluation Focus: Effective validation of required fields with clear error messages; blocking of conflicting or illegal operations (e.g., double-booking, editing non-existent records); graceful handling of boundary conditions (empty lists, excessively long inputs, invalid formats); consistency between interface state and underlying data.
	Typical Examples: An already-reserved meeting room cannot be booked again; submitting a form with empty required fields is blocked with a clear prompt.

Appendix

Table 8: Taxonomy of test case dimensions used in WebTestBench.

Model	Functionality				Constraint				Interaction				Content
Model	Cov.	P	R	F1	Cov.	P	R	F1	Cov.	P	R	F1	Cov.	P	R	F1
Minimax M2.1	77.9	15.9	14.0	12.3	40.4	19.9	14.4	15.8	42.2	22.6	20.3	19.9	47.1	8.6	7.3	7.7
Qwen3 Coder Next	77.6	17.5	14.9	14.1	48.3	27.5	23.1	23.8	42.7	12.4	12.8	11.4	35.9	5.1	3.9	4.3
GLM 4.7	79.9	21.0	16.8	16.5	47.3	26.5	18.4	20.5	36.0	18.9	17.6	17.2	48.9	5.1	3.9	4.3
GLM 5	79.7	16.8	11.0	11.9	50.1	32.4	26.5	26.9	41.4	24.2	20.3	20.9	50.6	5.1	2.6	3.4
Step 3.5 Flash	79.8	28.1	18.7	20.1	53.6	32.6	27.4	27.9	48.5	24.2	21.2	21.2	60.6	2.6	2.6	2.6

Appendix

Table 9: Detailed quality dimension-wise evaluation results on WebTestBench.

Model	Presentation				Search				Tool				Commerce
Model	Cov.	P	R	F1	Cov.	P	R	F1	Cov.	P	R	F1	Cov.	P	R	F1
Minimax M2.1	56.9	25.6	10.3	14.3	52.0	18.9	6.9	9.4	58.7	15.2	13.7	13.4	57.1	10.9	10.1	8.6
Qwen3 Coder Next	62.5	16.4	16.4	14.3	62.7	21.5	11.1	13.7	56.1	33.8	12.2	16.6	66.1	26.8	16.8	19.1
GLM 4.7	59.5	22.4	12.7	15.6	63.9	19.4	26.5	20.0	62.4	17.8	7.8	9.8	63.4	35.9	19.1	24.1
GLM 5	64.5	30.8	16.7	21.1	61.6	11.1	4.6	6.5	64.0	24.7	11.4	14.0	64.9	33.6	15.4	19.6
Step 3.5 Flash	65.7	38.6	13.2	18.8	68.4	24.1	13.7	17.1	61.4	37.2	22.2	25.3	72.8	34.3	22.5	25.6

Appendix

Table 10: Detailed category-wise evaluation results on WebTestBench.

Model	Functionality	Constraint	Interaction	Content	Overall
MiMo-V2-Flash	22.2/20.1/19.3	50.5/49.7/46.6	33.3/31.1/30.3	16.2/16.7/15.8	44.3/30.4/31.9
Step-3.5-Flash	33.5/25.6/26.4	61.4/56.9/55.6	39.2/36.9/35.9	30.3/33.3/31.2	53.4/38.7/40.9
GPT-5.1	42.8/54.8/42.0	57.0/76.5/59.2	55.9/65.7/56.0	36.5/46.2/38.2	47.0/63.4/46.8
Claude Sonnet 4.5	45.0/32.3/34.5	69.1/72.6/66.7	44.8/42.8/41.4	32.3/32.7/30.7	61.0/47.4/49.2

Appendix

Table 11: Detailed results on WebTestBench in the oracle setting.

Figure 5: An example illustrating how CUA interacts with a web application during defect detection.

Automated Software Testing

Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.

See all 136 papers

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective
Sep 14, 2026Chenxu Liu, Zilu Zou, Peizhong Gao +9Automated Software Testing Web Application Generation
Hunyuan Team, Tencent · Tsinghua University · Peking University
Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation
May 28, 2026Haoyue Yang, Zhangxiao Shen, Fan Ding +8Code Generation Evaluation Automated Evaluation
Baidu Inc., Beijing, China
Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
Mar 27, 2026Zehai He, Wenyi Hong, Zhen Yang +4Web Agent Benchmarks Web Application Generation
Tsinghua University · Zhipu AI

Sep 14, 2026J/K move · Enter open · S save

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao+8

Hunyuan Team, Tencent · Tsinghua University · Peking University

Automated Software Testing Web Application Generation Automated Evaluation

Open paper AskPDF arXiv

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.

Abstract

Automated Software Testing

Explore similar work

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification