WebAnything: From the Wild Web to Scalable Agent Environments

Tianci Xie; Zikun Cai; Chaoyun Zhang; Pinxue Guo; Bingxuan Li; Rui Zheng; Hang Yan

cs.AIOct 27, 2026

WebAnything: From the Wild Web to Scalable Agent Environments

Authors: Tianci Xie, Zikun Cai, Chaoyun Zhang, Pinxue Guo, Bingxuan Li, Rui Zheng, Hang Yan

Organizations: Zhejiang University · Nanyang Technological University · Shanghai Qiji Zhifeng Co., Ltd

Abstract

Web agents require diverse, executable, and resettable website environments for reproducible training and evaluation. Human-assisted mirrors demand per-site engineering, while generative construction can diverge from specified real websites. Capture-and-replay preserves observations, with interaction coverage constrained by captured content and replay capabilities. We present WebAnything, a fully automated framework that reconstructs source-grounded, reusable Web environments from real URLs without per-site manual implementation or repair after initial setup and authorization. An Explorer-Builder collects browser-observable structures, resources, and interaction outcomes; an independent verifier checks frozen candidates and guides local repair or targeted source recapture. We construct 244 website artifacts across 14 application domains and 53 categories, supporting 12,984 executed and accepted tasks. Compared with human-assisted WebHarbor, WebAnything achieves higher aggregate scores on three fidelity metrics, with 12 wins, two ties, and no losses in a blind audit. Fine-tuning Qwen3.5-9B on 5,856 interaction trajectories improves REAL success by 9.9 percentage points, Odysseys-Easy Rubric Avg and Task Avg by 6.25 and 6.32 percentage points, and WebArena-Verified Overall success by 13.33 percentage points under the fixed harness. Code: https://github.com/china-qijizhifeng/Web-Anything

Figures & tables

Figure 1: Trade-offs in real-website environment construction. Human-assisted mirrors require per-site engineering; generative methods need source correspondence; replay coverage depends on captured content and replay capabilities. WebAnything combines source-grounded reconstruction, independent verification, and iterative refinement.

Figure 2: WebAnything reconstruction pipeline. Source evidence guides executable construction. Independent checks of frozen candidates drive local repair or targeted recapture; verified candidates are released with repeatable interaction and task-evaluation interfaces.

Figure 3: Cross-domain resource supply: 244 website artifacts across 14 application domains and 53 fine-grained categories, with the domain distribution at right.

A: Automated fidelity metrics
Comparison subset	System	Sites/states	Pixel $\uparrow$	SSIM $\uparrow$	Text F1 $\uparrow$
Five shared WebHarbor sites	WebAnything	5 / 14	0.9240	0.7904	0.8541
Same subset	WebHarbor	5 / 14	0.8083	0.6260	0.2617
VeriEnv LinkedIn	WebAnything	1 / 4	0.9574	0.8190	N/A
Same subset	VeriEnv	1 / 4	0.9436	0.7637	N/A

Table 1a: Historical source-state fidelity. Automated metrics are aggregated within sites, then equally averaged across the five-site, 14-state WebHarbor subset; the four historical LinkedIn states are reported separately. Human blind-audit wins, ties, and losses compare WebAnything against WebHarbor.

Five-site WebHarbor comparison
System	Capability coverage	Executed-test success	All-test pass
WebAnything	15/20 (75%)	19/20 (95%)	19/30 (63.3%)
WebHarbor	16/20 (80%)	23/23 (100%)	23/30 (76.7%)
Five-site VeriEnv comparison
Selected pool artifacts	22/22 (100%)	29/31 (93.5%)	29/33 (87.9%)
VeriEnv	20/22 (90.9%)	21/28 (75.0%)	21/33 (63.6%)

Table 1b: Offline interaction validity on separate five-site subsets. Capability coverage counts tested capabilities; executed-test success uses executed tests; all-test pass retains unsupported and blocked tests. Persistence and state-restoration results appear in Appendix A.7 .

Training setting	REAL	Odysseys-Easy
	Success $\uparrow$	Rubric Avg $\uparrow$	Task Avg $\uparrow$	Perfect $\uparrow$
Qwen3.5-9B	24.2	19.71	18.86	4.55
+ WebAnything SFT	34.1	25.96	25.18	9.09
$\Delta$	+9.9pp	+6.25pp	+6.32pp	+4.55pp

Table 2: Downstream SFT results. REAL and Odysseys-Easy use 91 and 44 jointly valid tasks. WebArena-Verified uses our own implementation of the fixed harness with strict scoring. Values are percentages.

Training data	Rubric Avg	Task Avg	Perfect
WebHarbor	6.93	7.79	0.00
VeriEnv	6.42	7.53	0.00
WebAnything	10.44	11.70	0.53

Table 3: Odysseys-All on 189 jointly valid tasks (%). Perfect counts: WebAnything 1/189; both baselines 0/189. Data and validity definitions: Appendix E.2 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

System	B sites	A states and reference limitations
WebHarbor	5/5	Five sites/14 states; historical Apple and ESPN references have rendering defects
VeriEnv	5/5	LinkedIn/four states; Coursera is excluded from historical paired scoring; the other three sites lack historical references

Appendix

Table 4: External artifact eligibility for RQ2. A denotes historical paired fidelity; B denotes offline interaction validity.

Figure 4: Blind-audit page comparisons between historical source references and reconstructed mirrors. Rows show Apple iPad and a Cambridge Dictionary entry. Q1/Q2 indicate overall and information-fidelity preferences; Q3 indicates discrepancy severity: 0 equivalent, 1 minor, and 2 severe. Automated Text F1 is annotated separately from human ratings. The historical Apple reference records a failed font-stylesheet request. Aggregate blind-audit results across all 14 states are reported in Table 1a .

Site	Artifact	Paired/preset states	Pixel	SSIM	Text F1
Coursera	Selected pool artifact	2/2	0.7017	0.4742	0.4236
	VeriEnv	2/2	0.7466	0.5338	0.1928
Weather	Selected pool artifact	1/2	0.6373	0.4386	0.1955
	VeriEnv	1/2	0.5188	0.4018	0.1396
Apartments	Both; source 403	0/2	N/A	N/A	N/A
Discogs	Both; source 403/access challenge	0/2	N/A	N/A	N/A

Appendix

Table 5: Current-source spot checks. Paired/preset states retain all planned states; means use the same pages that render normally on both sides. Source unavailability and mirror runtime failures are reported separately. Product scope, location, and live-content conditions follow the protocol in this section.

System	Capability coverage	Executed-test success rate	All-test pass rate	Startup	Persistence	Initial-state restoration
Five-site WebHarbor comparison
WebAnything	15/20 (75%)	19/20 (95%)	19/30 (63.3%)	5/5	1/1	1/1
WebHarbor	16/20 (80%)	23/23 (100%)	23/30 (76.7%)	5/5	2/3	3/3
Five-site VeriEnv comparison
Selected pool artifacts a	22/22 (100%)	29/31 (93.5%)	29/33 (87.9%)	5/5	6/6	5/5
VeriEnv	20/22 (90.9%)	21/28 (75.0%)	21/33 (63.6%)	5/5	4/4 b	5/5

Appendix

Table 6: Interaction validity of selected frozen environments. WebHarbor and VeriEnv are tested on their respective five-site subsets. We report capability coverage, success rate over executed tests, and pass rate over all predefined tests. External network access is disabled; explicit test adapters restore state.

Independent outcome	Pairs	Share
Passed	56	25.57%
Failed	139	63.47%
Source blocked	24	10.96%
Of these: evidence-supported discrepancies	39	17.81%

Appendix

Table 7: Independent verification of submitted candidates with passing builder self-checks. The first three outcomes are mutually exclusive; the last row is a subset of failures supported by concrete defect evidence.

Figure 5: Historical verification cases. Stashmedia reveals a same-candidate resource-check omission; Ctrip traces feedback-driven revision and issue-level independent reverification; ACCA illustrates additional control and visual defects. Evidence scopes are specified in Appendix B.8 .

Panel of Figure 6 — Figure 6: Diversity and interaction complexity of 6,172 accepted source tasks spanning 74 site labels and 14 task application domains. (a) Six automated, overlapping capability attributes derived from instructions, trajectories, and website information, each using 6,172 as the denominator. (b) Grouped distribution of executed tool-interaction steps; approximately 34.7% of tasks exceed 10 steps.

Task attribute	Tasks	Share	Sites
Information retrieval	4,802	77.8%	74
Filtering and search	2,581	41.8%	72
Comparison and aggregation	1,549	25.1%	68
Multi-page tasks	6,157	99.8%	74
State operations	1,655	26.8%	68
Cross-site tasks	2,326	37.7%	28

Appendix

Table 8: Automatically assigned multi-label capability statistics for 6,172 accepted tasks.

Metric	Mean	Median	P75	P90	P95	Maximum
Observed tool-interaction steps	10.4	8	13	21	28	89
Observed URL page states	5.5	5	7	9	12	63
Page paths excluding query parameters	4.7	4	6	8	10	62

Appendix

Table 9: Observed interaction complexity of accepted tasks. Tool operations, URL states, and page paths are counted separately from actual trajectories.

Tool-interaction steps	Tasks	Share
0	4	0.1%
1	38	0.6%
2–5	2,092	33.9%
6–10	1,899	30.8%
11–20	1,498	24.3%
21–30	399	6.5%

Appendix

Table 10: Bins of observed tool-interaction steps. Rounded shares may not sum to 100%.

Item	Recovered configuration
Training method and parameter scope	Supervised fine-tuning; trainable language model, frozen vision encoder and projection layer; BF16
Training-data counts	6,172 accepted source tasks; 5,856 training trajectories; 5,167 unique training task IDs; 60,517 step-level samples
Samples and checkpoint	60,517 step-level samples; step-473 checkpoint; approximately two nominal data passes
Optimizer	Adam with decoupled weight decay $10^{-4}$ ; $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , $\epsilon=10^{-8}$
Learning rate and schedule	$5\times 10^{-6}$ , minimum 0; cosine decay; 24 warmup steps
Batch size and randomness	Global batch size 256, microbatch size 1; random and mixing seeds both 42

Appendix

Table 11: Actual configuration associated with the step-473 training run, cross-checked against command overrides, optimizer logs, and checkpoint records.

Training data	Trajectories	Training rows
WebHarbor (Core16)	617	5,240
WebAnything (all661)	661	5,520
VeriEnv	663	9,244

Appendix

Table 12: Final training-set sizes for the data-source comparison. Row counts refer to training-file records.

Metric	WebAnything	WebHarbor	VeriEnv
Valid tasks	194	196	197
Rubric Avg	10.73	6.98	6.62
Task Avg	12.12	7.81	7.77
Perfect	0.52	0.00	0.00
Perfect count	1/194	0/196	0/197

Appendix

Table 13: Odysseys-All results using each model’s own valid-task set. Score values are percentages. The common 189-task comparison is reported separately in Table 3 .

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 179 papers

Structured Distillation of Web Agent Capabilities Enables Generalization
Apr 9, 2026Xing Han Lù, Siva ReddyLLM Agent Training Web Agents
Mila – Quebec AI Institute McGill University
WebUniverse: Scaling Verified Training Data for Agentic Web Development
Oct 23, 2026Yaorui Ding, Fengqing Jiang, Yichen Feng +8Web Agents Web Application Generation
University of Washington · Western Washington University · King Abdulaziz City for Science and Technology +1
Safe and Scalable Web Agent Learning via Recreated Websites
Mar 11, 2026Hyungjoo Chae, Jungsoo Park, Alan RitterWeb Agents LLM Agent Training
School of Interactive Computing, Georgia Institute of Technology

Apr 9, 2026J/K move · Enter open · S save

Structured Distillation of Web Agent Capabilities Enables Generalization

Xing Han Lù, Siva Reddy

Mila – Quebec AI Institute McGill University

LLM Agent Training Web Agents LLM Agent Trajectory Synthesis Language Model Distillation

Open paper AskPDF arXiv

Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

Abstract

Web Agents

Explore similar work

Structured Distillation of Web Agent Capabilities Enables Generalization

WebUniverse: Scaling Verified Training Data for Agentic Web Development

Safe and Scalable Web Agent Learning via Recreated Websites