Structured Distillation of Web Agent Capabilities Enables Generalization

Xing Han Lù; Siva Reddy

cs.LGApr 9, 2026

Structured Distillation of Web Agent Capabilities Enables Generalization

Authors: Xing Han Lù, Siva Reddy

Organizations: Mila – Quebec AI Institute McGill University

Abstract

Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

Figures & tables

Figure 1: The Agent-as-Annotators pipeline replaces three human annotation roles with LLM modules. The Task Designer is replaced by a Persona Generator and Task Generator that synthesize task intents with evaluation hints; the Annotator is replaced by an Agent ; the Supervisor is replaced by a Judge . Only successful trajectories train the student.

Proprietary
Model	WA	VWA	WoA L1	W++ L2	WoB
Gemini 3 Pro ∗	51.2	49.0	79.7	41.6	74.7
Gemini 3.1 Flash L.	42.3	35.0	58.5	21.1	74.1
Open-weight (base)
Qwen3.5-27B	41.5	37.4	57.0	18.9	70.9
Qwen3.5-9B	31.0	28.5	33.3	2.2	63.2

Table 1: Distillation gains transfer across all five benchmarks, with the largest improvement on WorkArena L1 (+18.2pp), an enterprise interface never seen during training. Success rate (%); WA = WebArena, VWA = VisualWebArena, WoA = WorkArena ( Drouin et al., 2024 ) , W++ = WorkArena++ ( Boisvert et al., 2024 ) , WoB = MiniWoB. ∗ Teacher model used to generate A3-Synth .

Pipeline	Persona	Expl.	Task	Hints	Agent	Judge	Env.
NNetNav	Static DB	✓	Retro.	–	–	LLM	WA+Web
InSTA	–	✓	Ground.	–	✓	LLM	Web
AgentTrek	–	–	Tutorial	–	✓	VLM	Web
Explorer	–	✓	Retro.	–	–	LLM	Web
Go-Browse	–	✓	Ground.	–	✓	VLM	WA
Agent-as-Annotators	✓	✓	Ground.	✓	✓	LLM	WA

Table 2: Trajectory synthesis pipelines mapped onto the Agent-as-Annotators framework. Agent-as-Annotators is the only pipeline that instantiates all six modules. Task: Ground. = tasks defined before execution; Retro. = labeled post-hoc. Shaded cells = module absent. Individual module contributions are ablated in Appendix C .

Figure 2: Base vs. fine-tuned model on a WebArena Shopping Admin task. The base Qwen3.5-9B (top, red) opens filters and browses fields for 10 actions before returning the wrong order. A3-Qwen3.5-9B (bottom, green) clicks the correct pending order from the dashboard and answers in 2 actions.

Ablation	Trajs	SR (%)	$\Delta$
Full pipeline	2,322	41.5	–
1,430 trajectories	1,430	40.2	$-$ 1.3
715 trajectories	715	37.0	$-$ 4.5
No Judge filtering	2,999	37.0	$-$ 4.5
No reasoning traces	2,322	33.6	$-$ 7.9
With personas, with hints	600	37.8	–

Table 3: Ablations on WebArena (381 tasks). Top: data scaling and module removal from the full pipeline. Bottom: module variants at matched scale (600 trajectories each).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Gemini 3 Pro
Teacher	Reddit	GitLab	Shopping	Shop Adm.	Wikipedia	Map	Examples
Pro (reduced thinking)	69.0	70.6	80.8	80.8	85.4	78.0	16,353
Pro	66.0	64.2	68.8	78.2	78.8	74.4	15,351
Gemini 3.1 Pro
3.1 Pro (reduced thinking)	58.4	67.6	61.4	63.8	51.0	45.4	14,859
Gemini 3 Flash

Appendix

Table 4: Pro (reduced thinking) achieves the highest success rates across all environments, though Flash produces more total training examples (22,707 vs. 16,353) from lower-quality trajectories. Extended teacher model comparison on A3-Synth : success rate (%) by site and total training examples produced.

Figure 3: Teacher quality on A3-Synth (x-axis) vs. student performance on WebArena (y-axis) across three teacher configurations. The student is Qwen3-VL-8B-Thinking, an earlier-generation model that was trained on all three teacher variants before Qwen3.5-9B became available. Based on these results, we selected Gemini 3 Pro (reduced thinking) as the teacher for the primary A3-Qwen3.5-9B model (41.5%).

Figure 4: Cross-benchmark success rates for Qwen3.5-9B before and after fine-tuning on A3-Synth Pro (reduced thinking) trajectories. The largest gain is on WorkArena L1 (+18.2pp), an enterprise interface never seen during training. All benchmarks except WebArena are fully out-of-distribution.

Proprietary frontier models
Model	Reddit	GitLab	Shopping	Shop Admin	Map	All
Gemini 3.1 Pro	79.2	57.4	43.3	55.1	42.2	53.8
Gemini 3 Pro	75.0	56.4	34.4	53.8	45.3	51.2
Gemini 3.1 Flash Lite	68.8	48.5	32.2	41.0	28.1	42.3
Open-weight models (base)
Qwen3.5-27B	56.2	44.6	35.6	46.2	28.1	41.5

Appendix

Table 5: Fine-tuned Qwen3.5-9B (41.5%) matches the 3

\times

larger Qwen3.5-27B on WebArena, with the largest gains on GitLab (+18.8pp) and Shopping Admin (+18.0pp). Per-site success rates (%); bold = best open-weight

\leq

9B.

Proprietary frontier models
Model	Classifieds	Shopping	Reddit	All
Gemini 3.1 Pro	47.4	51.9	39.0	47.9
Gemini 3 Pro	51.7	49.8	44.0	49.0
Gemini 3 Flash	37.9	40.8	35.0	38.8
Gemini 3.1 Flash Lite	31.9	36.9	34.0	35.0
Open-weight models (base)

Appendix

Table 6: VisualWebArena results. Qwen3.5-9B and A3-Qwen3.5-9B are evaluated on the full 910 tasks (test + train); other models use the test split only (449 tasks, marked † ). Success rate (%) by site. Cross-site tasks are grouped by starting site.

Proprietary frontier models
Model	Create	Filter	Sort	Order	Chart	Navigate	All
Gemini 3.1 Pro	70.0	33.3	86.7	98.9	90.0	100	79.4
Gemini 3 Pro	72.0	35.0	88.3	100	82.5	100	79.7
Gemini 3.1 Flash Lite	52.0	31.7	10.0	94.4	70.0	96.7	58.5
Open-weight models (base)
Qwen3.5-27B	60.0	13.3	11.7	95.6	70.0	96.7	57.0

Appendix

Table 7: A3 fine-tuning produces the largest gain on WorkArena L1 (+18.2pp), with the biggest per-category jump on Order tasks (+38.9pp). Success rate (%) by task category; Crt = Create (50), Flt = Filter (60), Srt = Sort (60), Ord = Order (90), Cht = Chart (40), Nav = Navigation (30).

Proprietary frontier models
Model	Dashboard	Filter	Navigate	Infeasible	Other	All
Gemini 3.1 Pro	0.0	76.2	53.8	53.8	29.8	40.0
Gemini 3 Pro	0.0	81.0	53.8	50.0	40.0	41.6
Gemini 3.1 Flash Lite	0.0	69.0	30.8	11.5	6.4	21.1
Open-weight models (base)
Qwen3.5-27B	0.0	31.0	15.4	50.0	10.6	18.9

Appendix

Table 8: A3 fine-tuning yields a 4.4

\times

relative improvement on WorkArena++ L2 (2.2%

\rightarrow

9.7%), with gains concentrated on Filter tasks. Dashboard tasks (n=57) are 0% for all models. Success rate (%) by category; Dsh = Dashboard (57), Flt = Filter (42), Nav = Navigate (13), Inf = Infeasible (26), Oth = Other (47).

Proprietary frontier models
Model	Click	Type	Drag	Wkfl	Widget	Compr.	Reason	All
Gemini 3.1 Pro	87.5	84.6	16.4	98.8	83.1	71.2	64.8	77.1
Gemini 3 Pro	83.8	83.2	16.4	91.8	81.7	70.9	63.5	74.7
Gemini 3.1 Flash Lite	85.0	79.7	7.3	95.3	79.7	73.1	64.8	74.1
Open-weight models (base)
Qwen3.5-9B	80.0	61.0	1.8	77.6	55.9	61.5	65.9	63.2

Appendix

Table 9: A3 fine-tuning improves atomic web skills on MiniWoB (+5.8pp), with the largest gains on Type (+11.4pp) and Workflow (+10.6pp) tasks. Success rate (%) by category; Clk = Click (160), Typ = Type (123), Drg = Drag (55), Wkf = Workflow (85), Wdg = Widget (59), Cmp = Comprehend (52), Rsn = Reason (91).

Figure 5: Base vs. A3 model on WorkArena L1 (order Standard Laptop). The base model (top, red) finds the product page but fails to configure and submit the order within 15 actions. The A3 model (bottom, green) efficiently fills in all required fields and places the order in 5 actions.

Figure 6: Base vs. A3 model on WorkArena L2 (find warranty expiration date). The base model (top, red) fails to navigate to the hardware assets module within 35 actions. The A3 model (bottom, green) navigates through the knowledge base, finds the assets list, filters by user, and reports the correct warranty date.

Figure 7: Base vs. A3 model on VisualWebArena task 378 (Reddit comment). The base model (top, red) clicks login and fails authentication for 7 actions. The A3 model (bottom, green) directly types the trading loss amount into the comment box and submits in 2 actions.

Figure 8: Base vs. A3 model on MiniWoB (enter time). The base model (top, red) repeatedly clicks the input field for 10 actions without entering the time correctly. The A3 model (bottom, green) fills in “03:42 AM” in the correct format and submits in 3 actions.

BrowserGym ecosystem (same harness)
Model	Params	WA	VWA	L1	L2	WoB
Claude 3.5 Sonnet	–	36.0	22.0	56.4	38.8 †	69.8
o1-mini	–	29.9	–	56.7	14.3 †	67.8
GPT-4o	–	31.5	26.3	45.5	7.5 †	63.8
Llama 3.1 405B	405B	22.6	–	43.3	8.9 †	64.6
Llama 3.1 70B	70B	17.1	–	27.9	3.4 †	57.6

Appendix

Table 10: Comparison with BrowserGym ecosystem results using the same agent harness and evaluation protocol. A3-Qwen3.5-9B exceeds GPT-4o and Claude 3.5 Sonnet on WebArena, approaches frontier models on WorkArena L1, and matches Claude 3.5 Sonnet on MiniWoB. Success rate (%).

\dagger

= 147/185 task overlap due to WorkArena version mismatch.

Proprietary SOTA (from official leaderboards)
Harness	Model	Params	WA	VWA	L1	L2	WoB
Custom ‡	OpAgent ( Guo et al., 2026 )	–	71.6	–	–	–	–
Custom ‡	SGV ( Andrade et al., 2025 )	–	–	54.0	–	–	–
GenericAgent	GPT-5 ( de Chezelles et al., 2024 )	–	–	–	79.1	69.4	71.5
GenericAgent	Claude 4 Sonnet ( de Chezelles et al., 2024 )	–	–	–	63.3	40.4	70.7
Open-weight ( $\geq$ 10B, from leaderboards)

Appendix

Table 11: A3-Qwen3.5-9B compared against official leaderboard results across all five benchmarks. For each benchmark, we show the top proprietary, open-weight, and sub-10B models. Our 9B SFT model is the best open-weight SFT result on full WebArena (nearly 2

\times

Go-Browse) and exceeds GPT-4o on 4/5 benchmarks under the same GenericAgent protocol. Success rate (%); ‡ = custom agent pipeline; † = test split only (leaderboard reports full benchmark). A3-Qwen3.5-9B and Qwen3.5-9B WA numbers are on the full 812-task benchmark.

Training Data	Trajs	All	Rdt	Git	Shop	ShAdm	Map
Judge-filtered (A3-Qwen3.5-9B)	2,322	41.5	60.0	53.3	34.1	44.9	26.4
Unfiltered (no Judge)	2,999	37.0	62.2	46.7	27.3	42.3	18.9
$\Delta$ (no Judge)		$-$ 4.5	+2.2	( $-$ 6.6)	( $-$ 6.8)	( $-$ 2.6)	( $-$ 7.5)

Appendix

Table 12: Judge filtering ablation on WebArena (381 tasks). Removing the Judge and training on all trajectories (including failed ones) reduces overall success rate by 4.5pp despite 40% more training data. Per-site breakdowns show the largest drops on Map (

-

7.5pp), Shopping (

-

6.8pp), and GitLab (

-

6.6pp).

Variant	Trajs	Steps	All	Rdt	Git	Shop	ShAdm	Map
285-traj	285	2,036	32.0	53.3	39.1	28.4	33.3	15.1
715-traj	715	5,060	37.0	57.8	46.7	27.3	42.3	22.6
1430-traj	1,430	9,999	40.2	62.2	46.7	35.2	46.2	20.8
Full (A3-Qwen3.5-9B)	2,322	16,353	41.5	60.0	53.3	35.2	46.2	26.4

Appendix

Table 13: Data scaling on WebArena (381 tasks). Performance increases with training data, from 32.0% at 285 trajectories to 41.5% at 2,322, with diminishing returns at larger scales.

Figure 9: (a) WebArena SR increases with training data from 32.0% (285 trajectories) to 41.5% (2,322), with diminishing returns. (b) Reasoning trace ablation: removing traces entirely (

-

7.9pp) is less harmful than truncating them (

-

10.5pp at 500 chars,

-

14.7pp at 250), suggesting incoherent mid-thought cutoffs degrade training signal more than the absence of reasoning.

Training Data	Tasks	All	Rdt	Git	Shop	ShAdm	Map
Full pipeline (A3-Qwen3.5-9B)	2,322	41.5	60.0	53.3	34.1	44.9	26.4
With hints (subsampled)	600	37.8	60.0	44.6	30.7	41.0	24.5
No hints	600	35.4	57.8	39.1	35.2	34.6	22.6
$\Delta$ (hints contribution)		$-$ 2.4	( $-$ 2.2)	( $-$ 5.5)	+4.5	( $-$ 6.4)	( $-$ 1.9)
$\Delta$ (vs. full pipeline)		$-$ 6.1	( $-$ 2.2)	( $-$ 14.2)	+1.1	( $-$ 10.3)	( $-$ 3.8)

Appendix

Table 14: Hints ablation on WebArena (381 tasks). Removing hints from the Judge reduces downstream student performance by 2.4pp at matched data size, with the largest per-site drops on Shopping Admin (

-

6.4pp) and GitLab (

-

5.5pp).

Model	SR (%)	Tasks Completed	Total Tasks
Gemini 3.1 Pro (reduced thinking)	93.1	1,396	1,500
Gemini 3 Flash	88.3	1,325	1,500
Gemini 3 Pro	87.3	1,310	1,500
Qwen3-VL-32B-Thinking	74.7	1,121	1,500
Qwen3-VL-8B-Thinking	70.4	1,056	1,500

Appendix

Table 15: Exploration success rates by model across 1,500 tasks (250 per site). Success requires the agent to explore for at least 10 steps and produce the termination string.

A3 Module	InSTA	NNetNav	Explorer	Go-Browse
Persona Gen.	–	Persona DB	–	–
Exploration	Agent	LLM Explorer	Task Proposer	NavExplorer
Task Gen.	Task Proposer	Task Labeler	Refiner + Summ.	PageExplorer
Hints	–	–	–	–
Agent	Agent	–	–	Feas.Check + Solvers
Judge	LLM Judge	ORM	Task Verifier	VLM-as-judge

Appendix

Table 16: Vocabulary mapping across trajectory synthesis pipelines. “–” indicates the module is absent.

Hyperparameter	Value
Base model	Qwen/Qwen3.5-9B
Training data	Pro (reduced thinking) (16,353 examples)
Learning rate	$1\times 10^{-5}$
LR schedule	Cosine with warmup
Warmup ratio	0.03
Weight decay	0.0

Appendix

Table 17: Training hyperparameters for A3-Qwen3.5-9B.

Benchmark	Tasks	Steps	Domain	Relation to Training
WebArena	812	30	Self-hosted web apps	In-domain (same env)
VisualWebArena	910 †	30	Visual web tasks	OOD (different tasks)
WorkArena L1	330	15	ServiceNow (basic)	OOD (different env)
WorkArena++ L2	185	50	ServiceNow (complex)	OOD (different env)
MiniWoB	625	15	Synthetic micro-tasks	OOD (different env)

Appendix

Table 18: Summary of evaluation benchmarks.

Explore similar work

Feb 16, 2026

WebWorld: A Large-Scale World Model for Web Agent Training

Web agents require massive trajectories to generalize, yet real-world training is constrained by network latency, rate limits, and safety risks. We introduce WebWorld series, the first open-web simulator trained at scale. While existing simulators are restricted to closed environments with thousands of trajectories, WebWorld leverages a scalable data pipeline to train on 1M+ open-web interactions, supporting reasoning, multi-format data, and long-horizon simulations of 30+ steps. For intrinsic evaluation, we introduce WebWorld-Bench with dual metrics spanning nine dimensions, where WebWorld achieves simulation performance comparable to Gemini-3-Pro. For extrinsic evaluation, Qwen3-14B trained on WebWorld-synthesized trajectories improves by +9.2% on WebArena, reaching performance comparable to GPT-4o. WebWorld enables effective inference-time search, outperforming GPT-5 as a world model. Beyond web simulation, WebWorld exhibits cross-domain generalization to code, GUI, and game environments, providing a replicable recipe for world model construction.

Zikai Xiao, Jianhong Tu, Chuhang Zou +7

Qwen Team, Alibaba Group · Zhejiang University

LLM World Models World Model Learning Web Agents World Model Evaluation

Ask PDF

May 19, 2026

Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection

Large language models (LLMs) have enabled web agents that follow natural language goals through multi-step browser interactions. However, agents fine-tuned on specific trajectories and domain often struggle to generalize out of domain, and offline training can be compute-inefficient due to noisy, redundant trajectories and long accessibility-tree (AXTree) states. To address both issues, we propose Weasel, a trajectory selection method for offline training of web agents. Weasel selects a fixed-budget subset of trajectory steps by optimizing an objective that balances unary importance with pairwise diversity over states, websites, and interaction patterns, solving efficiently with a greedy algorithm. We further improve efficiency with target-centered AXTree pruning that keeps only content around the ground-truth action target, and we mitigate style mismatch for reasoning-native models by replacing expert traces with model-generated, style-consistent rationales. Across AgentTrek and NNetNav training datasets, evaluations in WebArena, WorkArena, and MiniWob, and experiments with Qwen2.5-7B, Gemma3-4B, and Qwen3-8B, Weasel improves out-of-domain performance while reducing training cost, producing roughly 9.7-12.5

\times

training speedups over standard fine-tuning. We make the code available at https://github.com/fatemehpesaran310/weasel.

Fatemeh Pesaran Zadeh, Seyeon Choi, Xing Han Lù +2

Seoul National University · McGill University · Mila – Quebec AI Institute +1

OOD Generalization Training Data Selection LLM Agent Training Web Agents

Ask PDF

Mar 5, 2026

TimeWarp: Evaluating Web Agents by Revisiting the Past

As web agents close the gap with humans on benchmarks, one question arises: Do today's agents perform just as well on tomorrow's web? We introduce TimeWarp, a benchmark that emulates the evolving web. TimeWarp consists of three web environments, each with six UI versions spanning UI design, frontend code, and workflows from different eras of the internet. We pair TimeWarp with a set of complex, realistic tasks covering different forms of web navigation. Our experiments reveal that vision-based agents are vulnerable to changes, while text-based agents become brittle once fine-tuned on a single version. To address this, we propose TimeTraj, a new annotation method that uses plan distillation to collect trajectories across multiple versions. By training agents on teacher rollouts using our BC-variant, we achieve substantial performance gains: 20.4% to 37.7% for Qwen-3 4B and 0% to 27.0% for Llama-3.1 8B models. Our work helps study generalization across web designs and opens a new paradigm for collecting plans rather than trajectories to improve the robustness of web agents.

Md Farhan Ishmam, Kenneth Marino

University of Utah, USA

Computer-Use Agent Benchmarks Web Agent Benchmarks Web Agents

Ask PDF