Back to Basics: A Skill-Centric Approach to Generalist Web Agents

Minh Pham-Dinh; Md Farhan Ishmam; Hernan Yupanqui; Reaz Hassan Joarder; Kenneth Marino

cs.CVOct 26, 2026

Back to Basics: A Skill-Centric Approach to Generalist Web Agents

Authors: Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui, Reaz Hassan Joarder, Kenneth Marino

Organizations: University of Utah · Islamic University of Technology

Abstract

Browser agents do not just need more practice. They need to practice the right things. We introduce MiniWeb, a realistic miniature of the web comprising 65 interconnected websites across 15 domains, where agents navigate, search, and transact as people do. Its 369 human-annotated tasks cover everyday workflows and label 37 reusable interaction primitives grouped into 7 skill families, allowing us to diagnose which skills fail rather than only whether a task succeeds. Benchmarking ten open-weight agents, we find that drag & gesture is consistently the weakest family, with no agent solving more than 24% of its instances. To address this gap, we introduce MiniWebGen, a framework that autonomously generates verified, skill-centric training data from MiniWeb. Fine-tuning Qwen3.5-4B on just 3.1K verified trajectories improves single-primitive success by 24.9pp across 13 unseen websites, with gains across every skill family. However, these trajectories teach short-horizon interactions, which do not directly translate to full-task success. We build on this result with MiniWebAgent, which pairs the trained executor with a planner that decomposes full tasks into the skill-level steps it learned to perform. This approach improves task success by 19.4pp points on unseen MiniWeb sites and transfers to external benchmarks, with gains of 9.3pp on WebArena-Lite, 16.1pp on Online-Mind2Web, and 20.0pp on WebVoyager, without training on their trajectories. We provide a simple recipe for compact browser agents: practice reusable skills directly, then compose them to solve unfamiliar tasks.

Figures & tables

Figure 1: MiniWeb grades every primitive of a task. Top: Fara1.5-4B opens the right café but leaves the rating filter at Any. Bottom left: Overall skill profile of 10 evaluated agents. Bottom right: task success of Qwen3.5-4B before and after our training.

Scale
	WebArena	WebVoyager	OM2W	MiniWeb
Tasks	812	643	300	369
Websites	5	15	136	65
Hosting	Local	Live	Live	Local
Skill-level evaluation
Skill-labeled steps	✗	✗	✗	✓

Table 1: MiniWeb attributes task failures to skills. Top: our 36 primitives (reasoning excluded) by family; benchmark rows use our labels of 15.7K tasks, and MiniWeb uses MiniWebGen ’s trajectories (Supp. A.4 ). Bottom: MiniWeb labels, segments, and verifies every step by skill; MiniWoB++ [ 25 , 17 ] and ComponentBench [ 11 ] test interactions only outside tasks. OM2W: Online-Mind2Web [ 34 ] .

Figure 2: The 65 websites. Left: sites by domain (squares: held-out), linked to hubs at the center (colored edges) and by cross-site tasks (dashed). Right: the 65 MiniWeb sites, divided by data source.

Figure 3: Building MiniWeb bottom up. (1) Tasks of existing benchmarks are mined into 37 primitives in 7 families. (2) Sites are built around their primitives. (3) Tasks chain primitives, with a verdict per primitive. Takeaway: every step of a task maps to one skill that can be graded.

Figure 4: MiniWebGen (left) generates and gates single-primitive trajectories (real ShopHub example). MiniWebAgent (right): an untrained Qwen3.5-4B planner hands each step to one trained LoRA executor (real held-out trace).

General VLMs, browser-use harness
		Task	Skill family: instances passed							Reasoning operation
Agent	Size	success	Nav. Ag	Text Ag	Select Ag	Form Ag	I/O Ag	Drag Ag	Reason Ag	read	verify	extremum	count
Qwen3.5-4B	4B	40.1 (2.5)	55 (6)	39 (4)	42 (4)	37 (4)	32 (6)	12 (8)	60 (4)	56 (5)	43 (11)	37 (7)	31 (8)
Qwen3.5-9B	9B	54.7 (2.5)	71 (5)	55 (4)	60 (4)	62 (4)	61 (6)	24 (10)	60 (4)	65 (5)	71 (10)	46 (8)	44 (8)
Qwen3-VL-8B	8B	43.8 (2.6)	63 (6)	43 (4)	42 (4)	53 (4)	50 (7)	12 (8)	52 (5)	53 (5)	62 (10)	41 (8)	43 (9)
InternVL3.5-8B	8B	39.0 (2.5)	61 (6)	44 (4)	41 (4)	42 (4)	32 (6)	24 (10)	45 (5)	46 (5)	48 (11)	32 (7)	33 (8)

Table 2: Skill profiles of untrained agents on all 369 MiniWeb tasks. Task success, then the share of primitive instances passed (%) per skill family and reasoning operation, all judged by the VLM judge (Gemini 3.5 Flash). One run per agent; in brackets: standard error over 1,000 bootstrap resamples of the tasks. Red: each agent’s weakest skill family; green: its strongest. Bold: best per column.

Figure 5: Most drag failures happen at the gesture itself. Outcomes of the ten untrained agents on every drag & gesture Ag instance (11 slider and 6 signature tasks), from the VLM judge’s explanations; bypassed means, e.g . , typing a name instead of drawing it.

Figure 6: Failures compound along a task’s chain of primitives. Left: the ten untrained agents of Tab. 2 on all 369 tasks (grey), with

p

their mean success on one-primitive tasks. Right: Qwen3.5-4B untrained (grey) or trained (blue) on the 62 held-out tasks (Tab. 3 ).

Single agent, trained on
	MiniWeb	WA-Lite	WV	OM2W
tasks	62	154	536	280
nothing	38.7 (5.5)	27.1 (3.0)	45.5 (2.2)	17.9 (2.2)
our data	39.2 (5.7)	20.3 (3.0)	32.8 (2.0)	8.6 (1.6)
Planner, executor trained on
nothing	45.2 (5.9)	30.7 (3.3)	59.9 (2.2)	21.8 (2.4)

Table 3: Composing trained skills raises task success. Task success (%) of Qwen3.5-4B on MiniWeb ’s held-out sites, WebArena-Lite (WA-Lite), WebVoyager (WV), and Online-Mind2Web (OM2W), as a single agent or as a planner whose executor is untrained or trained at matched size on task-centric trajectories or on MiniWebGen ’s skill-centric trajectories. In brackets: standard error over 1,000 bootstrap resamples of the tasks. Green (red): significantly above (below) the untrained row of the group,

p<0.05

, by a paired test over the same tasks.

Figure 7: Practice lifts every skill family. Success per skill family on the 103 generated single-primitive tasks of the held-out sites, for a single Qwen3.5-4B agent untrained (grey) and trained on our data (blue), with the gain at the right; reasoning Ag , which is not trained, is left out.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Tasks	Labels	Labels per task	Candidates used	Only in this benchmark
WebArena	812	1,813	2.23	99	5
WebShop	12,251	42,859	3.50	36	0
WebVoyager	643	1,865	2.90	85	4
Mind2Web	2,021	5,670	2.81	124	13
All	15,727	52,207	3.32	136	22

Appendix

Table 4: The mined tasks. Tasks labeled per benchmark, their macro labels, and the candidate macros they use. Only in this benchmark : candidates that no other benchmark’s tasks use.

Verb	Labels (%)	Without WebShop (%)	Candidates	Most frequent candidate
select	31.3	9.4	27	select_to_purchase
search	26.4	16.5	5	search_by_keyword
filter	17.1	27.8	20	filter_by_attribute
add	14.6	3.1	8	add_to_cart
submit	4.4	8.4	23	submit_to_purchase
extract	3.0	16.9	14	extract_by_question_answer

Appendix

Table 5: Labels per verb. Share of the mined labels by the verb of their macro, over all four benchmarks and without WebShop; the number of candidate macros with that verb; and the most frequent of them.

Operation	Definition	Check	Instances
read	the info is present on the page — locate and read it	answer appears verbatim on a visited page	100
extremum	get the max/min among on-page items without using a sort control	chosen item is the argmax/argmin of the candidate set	41
count	count the on-page items that match a condition	count matches the true number of matching items	36
verify	compare a value on the page against a value given in the instruction	the on-page value satisfies the instruction’s stated value/condition	21

Appendix

Table 6: Reasoning operations attached to instances of the 37 primitives in the human tasks. A primitive without an operation is a pure interaction. Not listed (fewer than 15 instances): compute (7), spatial (4), compare (3).

Primitive	Description	Tasks	Sites	Gen.
Drag & gesture
filter_by_slider	Narrow results with a range slider. Span: first drag $\rightarrow$ filter applied	11	9	74
sign_by_freeformdrawing	Sign a document by DRAWING your signature freeform on a canvas (“Sign here” field). Span: start drawing in the signature field $\rightarrow$ signature adopted/submitted	6	4	100
Discrete selection
toggle_relationship	Locate a user/post/item, then toggle a relationship (follow / subscribe / save / join-group / block). Span: locate the target $\rightarrow$ toggle clicked	46	28	97
filter_by_dropdown	Narrow results by selecting a dropdown value (incl. category search). Span: open the dropdown $\rightarrow$ option selected / results update	41	28	97

Appendix

Table 7: The 37 interaction primitives , by skill family. Tasks : instances in the 369 human tasks; Sites : sites those tasks use; Gen. : trajectories generated by MiniWebGen in the training pool (2,687; the pool’s other 441 are of dropped primitives, and report_information is learned from the base model’s own successes instead). The span marks where an instance starts and ends in a demonstration. Dropped in the final refinement (fewer than 3 human-task instances): reposition_by_drag (2), save_by_form (2), search_by_pan_zoom (2), reveal_by_2fa (1), search_by_playback (1), carry_info_cross_site (0), count_entries (0), edit_by_ranking (0), edit_by_textbox (0).

Site	Source	Stated license
academic-paper-db	arXiv metadata	CC0 1.0
flights-hotels	US airline routes and fares 1993–2024 (Kaggle; US DOT data); TBO hotels (Kaggle)	Apache 2.0; MIT
health-fitness-tracking	USDA FoodData Central; Fitbit data (Furberg et al., Zenodo)	CC0 1.0; CC BY 4.0
job-sites	Job Description Dataset (Kaggle; synthetic postings)	CC0: public domain
wikis	English Wikipedia	CC BY-SA 4.0
news	Wikinews	CC BY 2.5 / 4.0

Appendix

Table 8: Public data sources of the 21 seeded sites and their stated licenses. The other 44 sites use only synthetic data.

Panel of Figure 8 — Figure 8: (a) Dashboard of the annotation tool: the three annotation steps and the coverage of sites and primitives. (b) Macro browser: one recorded span of authenticate_by_form , its place on the task’s action timeline (one row per primitive instance), and the other tasks that contain it.

Figure 9: Task editor. The embedded site (left), the instruction and the design, record, and submit steps (right), and the primitive graph with one node per instance, its subtask, and its reasoning operation (bottom).

Figure 10: Verifier builder. Each primitive instance of the task gets its own check (request, action, page visited, answer), grounded in the gold requests of the recording; the sandbox below tests the verifier against the gold recording and negative controls.

Agent	Episodes	Agree (%)	Cohen’s $\kappa$	Judge only	Verifiers only
Qwen3.5-4B	369	92.4	0.84	21	7
Qwen3.5-9B	369	90.5	0.81	30	5
Qwen3-VL-8B	368	92.4	0.84	23	5
InternVL3.5-8B	369	90.8	0.80	24	10
Fara1.5-4B	369	91.1	0.82	29	4
Fara-7B	369	93.2	0.83	15	10

Appendix

Table 9: Agreement of the two graders. Task-level agreement of the VLM judge and the task verifiers on the untrained agents’ episodes of Tab. 2 . Judge only : episodes the judge passes and the verifiers fail; verifiers only : the reverse.

Graders’ verdicts	Episodes	Audited	Judge right	Verifiers right
Both pass	1,235	30	30/30	30/30
Both fail	2,256	30	27/30	27/30
Judge only	187	36	34/36	2/36
Verifiers only	81	24	7/24	19/24
Weighted accuracy (%)	3,759	120	92.2	88.9
Cohen’s $\kappa$ with the human			0.84	0.77

Appendix

Table 10: Human audit of the graders on agent episodes, by the graders’ verdicts. Episodes : all judged episodes of the ten untrained agents in that group, on the earlier 376-task set; Audited : the labeled sample. Accuracy and

\kappa

weight each group by its share of all episodes.

Actor	Observation	Actions
Human annotator (recorder)	The live page. Saved per action: URL, title, HTML snapshot, accessibility tree, scroll position, screenshot; every HTTP request with its body, status, and response; the server’s request log.	Recorded: click , type (debounced), select , check , change , keypress (Enter, Escape, Tab), scroll , drag (from, to), clipboard_write , navigate , submit .
MiniWebGen executor (script)	Privileged, read-only DOM queries: dom.find / one (by CSS, text, or tag), refresh , focused , url , title , outline , viewport .	Pixel actions on visible elements: act.click , double_click , hover , type , press , scroll , scroll_into_view , drag , draw , upload , open_url , switch_tab ; and expect.backend() to assert the expected backend evidence.
Agent and executor (browser-use 0.13.3)	Screenshot; indexed list of interactive elements; URL, open tabs, scroll position; step counter and budget; memory of previous steps.	One per step: click (element index or 0–1000 coordinates), input , select_dropdown , dropdown_options , send_keys , scroll , find_text , navigate , go_back , switch , close , wait , screenshot , drag ∗ , draw ∗ , macro_done ∗ , done .
MiniWebAgent planner	The same state as the agent.	delegate (family group, one-step instruction), scroll , done .

Appendix

Table 11: Observation and action spaces. ∗ Actions we add to browser-use. Fara1.5-4B runs in its released agent loop with its own observation and action space.

Website	Tasks	Episodes ending at a gate
google.com/shopping	4	8 of 8
drugs.com	8	9 of 16
target.com	2	3 of 4
boardgamegeek.com	1	2 of 2
cvs.com	1	2 of 2
kaggle.com	1	2 of 2

Appendix

Table 12: Blocked Online-Mind2Web sites. The nine sites removed because they gate our network, with the share of their episodes, pooled over two full-set runs, whose final answer reports a gate (login and sign-up CAPTCHAs excluded).

Stage	Hardware	Time	API cost
Trajectory generation ( MiniWebGen )	API calls	–	$0.04 per kept trajectory
Executor training, 27.6K steps	4 $\times$ A800	5.6 h (22 GPU-h)	–
MiniWeb held-out run (62 tasks), verifiers	1 GPU	$\approx$ 1 h	–
Skill profile of one agent (369 tasks), VLM judge	1 GPU	$\approx$ 5 h	$0.045 per episode
WebArena-Lite run (154 tasks)	RTX 5090, 4 site instances	$\approx$ 3 h	$<$ $1
WebVoyager run (536 tasks, 100 steps)	24 or 40 GB GPUs	10–15 GPU-h	$26 (GPT-4o judge)

Appendix

Table 13: Compute and API cost per stage, measured on our runs.

	Single	Held-out	WebArena-	Web-	Online-
Qwen3.5-4B configuration	primitives	tasks	Lite	Voyager	Mind2Web
Single agent, untrained	55.3 $\pm$ 1.0	38.7 $\pm$ 2.5	27.1 $\pm$ 0.6	45.5	17.9
Single agent, trained	80.3 $\pm$ 0.3	39.2 $\pm$ 1.1	20.3 $\pm$ 1.3	32.8	8.6
Planner, untrained executor	54.0 $\pm$ 0.6	45.2 $\pm$ 3.4	30.7 $\pm$ 1.4	59.9	21.8
MiniWebAgent (planner + trained executor)	71.5 $\pm$ 1.3	58.1 $\pm$ 0.9	36.4 $\pm$ 1.4	65.5	33.9
Planner + task-centric executor (27.6K steps)	–	47.8 $\pm$ 2.8	34.2 $\pm$ 1.5	62.1	23.2

Appendix

Table 14: Task success (%) of every configuration. Means

\pm

standard error over runs where there are several (three for MiniWeb and WebArena-Lite; two for Fara1.5-4B ’s held-out tasks). Tab. 3 uses the first five rows.

Question	Benchmark	A $\rightarrow$ B		$\Delta$	$p$	Holm $p$
Practice alone (single agent)	Single primitives	55.3 $\rightarrow$	80.3	$+24.9$	$<10^{-4}$	$<10^{-3}$
	Held-out tasks	38.7 $\rightarrow$	39.2	$+0.5$	1.00	1.00
	WebArena-Lite	27.1 $\rightarrow$	20.3	$-6.7$	0.023	0.047
	WebVoyager	45.5 $\rightarrow$	32.8	$-12.7$	$<10^{-6}$	$<10^{-5}$
	Online-Mind2Web	17.9 $\rightarrow$	8.6	$-9.3$	$<10^{-4}$	$<10^{-3}$
Planning alone (untrained executor)	Held-out tasks	38.7 $\rightarrow$	45.2	$+6.5$	0.13	0.40

Appendix

Table 15: Every comparison of the paper. Paired permutation test over tasks on MiniWeb and WebArena-Lite (per-task means over three runs); exact McNemar test on the live web.

\Delta

comes from the unrounded means. Holm

p

: corrected within each question. The specialist and stop-early comparisons are exploratory.

Site	Tasks	Single, untr.	Single, trained	Planner, untr.	MiniWebAgent	Task-c. exec.
GitLab	30	30.0	23.3	21.1	35.6	28.9
Reddit	19	22.8	24.6	43.9	40.4	36.8
Shopping	44	35.6	25.8	38.6	46.2	44.7
Shopping admin	35	24.8	13.3	30.5	34.3	31.4
Map	26	15.4	14.1	19.2	20.5	24.4
Without Reddit, GitLab	105	27.0	18.7	31.1	35.9	35.2

Appendix

Table 16: WebArena-Lite success (%) by site , mean of three runs (benchmark evaluators). Task-c.: the executor trained on task-centric data from our training sites.

Model calls per episode (mean)	Held-out	WebArena-Lite	WebVoyager	Online-Mind2Web
Single agent, untrained	21.3	29.1	57.5	73.1
Single agent, trained	13.8	16.4	15.8	36.8
Planner, untrained executor	41.1	57.9	95.0	125.6
MiniWebAgent	36.0	57.6	79.2	108.2

Appendix

Table 17: Model calls per episode : agent steps of the single agent; planner turns plus executor steps of the planner agents. The action budget is the same for all (40 site actions on MiniWeb and WebArena-Lite, 100 on the live web).

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

Apr 14, 2026

WebXSkill: Skill Learning for Autonomous Web Agents

Autonomous web agents powered by large language models (LLMs) remain brittle on long-horizon browser workflows. A key bottleneck is a grounding gap in existing skill formulations: textual workflow skills provide natural language guidance but cannot be directly executed, while code-based skills execute without giving the agent step-level guidance for adaptation or recovery. We introduce WebXSkill, a framework that bridges this gap with executable skills, each pairing a parameterized action program with step-level natural-language guidance. WebXSkill operates in three stages: skill extraction mines reusable action subsequences from readily available synthetic agent trajectories and abstracts them into parameterized skills, skill organization indexes them into a URL-based graph for context-aware retrieval, and skill deployment exposes two complementary modes, grounded mode for fully automated execution and guided mode where skills serve as step-by-step instructions the agent follows with its native planning. WebXSkill demonstrates consistent improvements on WebArena, WebVoyager, and Online-Mind2Web. We further find that better skill deployment mode depends on a model's plan and execution capability. The code is available at https://github.com/aiming-lab/WebXSkill.

Zhaoyang Wang, Qianhui Wu, Xuchao Zhang +12

University of North Carolina at Chapel Hill · Microsoft

LLM Agent Skill Learning Long-Horizon LLM Agents Web Agents

Ask PDF

Apr 9, 2026

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.

Tanmay Gupta, Piper Wolters, Zixian Ma +13

Allen Institute for AI · University of Washington · UNC-Chapel Hill

Web Agents LLM Agent Training Tool-Using Vision-Language Agents

Ask PDF

Apr 9, 2026

Structured Distillation of Web Agent Capabilities Enables Generalization

Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

Xing Han Lù, Siva Reddy

Mila – Quebec AI Institute McGill University

LLM Agent Training Web Agents LLM Agent Trajectory Synthesis Language Model Distillation

Ask PDF

Abstract

Web Agents

Explore similar work

WebXSkill: Skill Learning for Autonomous Web Agents

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Structured Distillation of Web Agent Capabilities Enables Generalization