Why Generate a Click? Plug-in Decision Heads for Web Agents

Chengguang Gan; Yunhao Liang; Hanjun Wei; Qinghao Zhang; Zhixi Cai; Shiwen Ni

cs.CLOct 14, 2026

Why Generate a Click? Plug-in Decision Heads for Web Agents

Authors: Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Zhixi Cai, Shiwen Ni

Organizations: Independent Researcher · University of Chinese Academy of Sciences · Pusan National University · Monash University ·

Abstract

Web agents built on vision language models decide every action by generating text such as click(12), although most of that text encodes a choice among a few action types and a few dozen marked elements. We read these choices directly from the hidden states that the generator computes while it reads the goal and the screenshot. Plug-in decision heads on a frozen generator predict the action type, pool the visual tokens inside each numbered box to pick the target, and point at a listed text option or at a WRITE slot that leaves the writing of new text to the generator. A confidence cascade executes the heads' action when they are sure and otherwise lets the generator decode from the KV cache of the same forward pass. With Qwen3-VL 4B and 8B generators trained on 213k web steps, a strict threshold keeps the cascade between 0.3 points below and 1.4 points above the generator on seven offline test sets while the heads decide 12 to 64% of the steps. A looser threshold raises step success on unseen Mind2Web websites by 3.1 and 1.8 points, costs up to 2 points on GUIAct and MiniWoB++, and lifts WebArena task success from 12.5 to 13.8% (4B) and from 13.6 to 14.6% (8B) with less than half of the generated tokens. Training the heads jointly with the generator lowers the generator's own step success by up to 4.9 points; heads on a frozen generator avoid this cost.

Figures & tables

Figure 1: WebArena task 513 as solved by our 4B cascade ( $\tau=0.5$ , seed 0). On the page of step 2 the generator would spell click(6) as five generated tokens, while the heads read the type and the box from the generator’s prefill and act without generating a token. Over the episode the heads act alone twice, twice choose the action while the generator writes only the text, and hand one uncertain step to the generator (more episodes in Appendix I ).

Figure 2: Overview. The frozen generator reads the goal, the history, the marked screenshot and the listed text options in one prefill. Decision heads read a learned mix of five of its layers and return a distribution over action types, boxes and options (with a WRITE slot). When the confidence $c_{t}$ reaches $\tau$ the agent acts at once or, for WRITE, lets the generator write only the text; otherwise the generator decodes the whole action from the same KV cache.

		M2W			GUIAct web
	System	Task	Website	Domain	Single	Multi	Single (ext.)	Multi (ext.)
4B	Generator	46.2	40.9	44.3	84.1	56.1	78.9	63.5
	Heads only	39.7	37.9	39.1	81.0	54.7	75.8	60.3
	Cascade, $\tau=0.9$	46.6	42.3	44.4	84.0	55.9	78.8	63.7
	Cascade, $\tau=0.5$	47.6	44.0	44.6	82.9	54.2	77.9	61.8
	Joint, generation	42.0	36.0	42.4	83.6	52.1	78.1	60.5

Table 1: Offline step success (%). Our cascade (shaded) uses heads trained on the frozen generator of the first row. The heads decide 12 to 18% of the M2W steps and 27 to 64% of the GUIAct steps at

\tau=0.9

, and 37 to 46% and 66 to 89% at

\tau=0.5

	MiniWoB++		WebArena		Tokens/step
System	4B	8B	4B	8B	4B	8B
Generator	53.3	55.5	12.5	13.6	7.4	7.5
Cascade, $\tau=0.5$	52.5	53.6	13.8	14.6	3.1	3.3
Joint, generation	53.3	55.0	11.5	11.1	7.3	7.4
Joint, cascade	53.2	54.2	11.9	12.6	3.0	2.9
Joint, heads only	45.4	45.3	3.1	4.4	0	0

Table 2: Task success (%) online and tokens generated per WebArena step, counting every token the generator decodes, excluding the forced prefix of a WRITE step. Cascades use

\tau=0.5

System	Seen template (47)	New template (127)
Generator 4B	34.8	4.2
Cascade 4B	41.8	3.4
Generator 8B	33.3	6.3
Cascade 8B	36.9	6.3

Table 3: WebArena task success (%) on the 47 test tasks whose intent template occurs among the 115 replayed training tasks, and on the other 127.

System	s/step	Speedup	Act alone
Generator	0.234	1.00 $\times$	0%
Cascade, $\tau=0.9$	0.207	1.13 $\times$	29%
Cascade, $\tau=0.5$	0.168	1.39 $\times$	67%
Joint, generation	0.238	1.00 $\times$	0%
Joint, cascade	0.155	1.54 $\times$	67%

Table 4: Model time per step on MiniWoB++ (4B, one agent per GPU) and share of steps on which the heads act alone. Speedups are relative to the generation that hosts the heads: the generator for our cascades, joint generation for the joint cascade.

Figure 3: Step success against the share of steps the heads decide, for thresholds from 0.99 (left of each curve) to 0.5 (right). The leftmost point is the generator and the hollow marker at 100% is heads only; dashed lines give the expected success when the same share of steps goes to the heads at random. Appendix E lists all test sets.

	Generation		Heads only
Test set	4B	8B	4B	8B
M2W task	$-4.2$	$-4.3$	$+1.1$	$+1.0$
M2W website	$-4.9$	$-2.4$	$+1.3$	$+1.1$
M2W domain	$-1.9$	$-1.9$	$+1.2$	$+1.0$
GUIAct single	$-0.5$	$-0.8$	$+1.2$	$+1.0$
GUIAct multi	$-4.0$	$-3.9$	$+1.6$	$+1.7$

Table 5: Effect of joint training, in points of success: generation of the joint adapter minus that of the separately trained generator, and joint heads minus heads trained on the frozen generator.

	M2W			GUIAct
System	Task	Web.	Dom.	Single	Multi
Cascade 4B, $\tau=0.5$	47.6	44.0	44.6	82.9	54.2
Two models 4B, $\tau=0.5$	47.9	44.3	46.1	83.6	56.0
Two models 4B, $\tau=0.9$	46.7	41.9	44.5	84.3	56.0
4B heads, 8B gen.	47.8	45.6	47.2	82.9	56.5
Cascade 8B, $\tau=0.5$	50.2	46.3	48.2	82.8	56.9

Table 6: Offline step success (%) of cascades whose heads come from the joint adapter and whose generator is the separately trained one. The fourth row pairs 4B joint heads with the generation of the 8B joint adapter at

\tau=0.5

		Decided steps		Rest
Test set	Share	Heads	Gen.	Gen.
M2W task	38%	72.2	68.4	32.7
M2W website	37%	68.6	60.3	29.4
M2W domain	40%	65.1	64.3	31.1
GUIAct single	85%	87.4	88.8	56.5
GUIAct multi	66%	67.3	70.2	28.5

Table 7: The 4B cascade at

\tau=0.5

. Top: step success on the steps the heads decide, by the heads and by the generator, and on the remaining steps by the generator. Bottom: step success by operation on M2W cross-website.

	M2W			GUIAct
Target head	Task	Web.	Dom.	Single	Multi
Patch attention	35.3	34.7	37.8	54.4	11.7
Box pooling	37.3	35.2	39.2	54.2	13.5

Table 8: Heads only step success (%) with two target heads, both trained as joint heads on MiniWoB++ and M2W; GUIAct is held out from this training.

System	Success	s/step	Act alone
Generator, no thought	53.9	0.24	0%
Reasoning generator	49.9	1.65	0%
Joint, generation	51.0	1.56	0%
Joint, cascade $\tau=0.5$	51.5	0.70	66%

Table 9: MiniWoB++ success (%) of 4B generators that write a thought before the action. All rows are trained on MiniWoB++ only; the joint rows train the reasoning generator and the heads in one adapter.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Steps	Main operations
MiniWoB++	9,171	click, type, no op
Mind2Web train	7,320	click, type, select
GUIAct single	66,089	click, scroll, answer
GUIAct multi	27,700	click, scroll, answer
UGround	65,066	click
WebLINX	38,171	click, type, scroll

Appendix

Table 10: Converted training steps per source, before deduplication. MiniWoB++ excludes the held-out tasks; the WebArena replay is used only for the WebArena fine-tuning.

Setting	Value
LoRA rank, scale, dropout	16, 32, 0.05
LoRA targets	attention and MLP
LoRA parameters (4B / 8B)	33.0M / 43.6M
Head parameters (4B / 8B)	6.0M / 9.2M
Head width $d_{h}$	512
Layers read	12, 18, 24, 30, 36

Appendix

Table 11: Training settings shared by all runs. Generators that write a thought (Section 6.4 ) train for three epochs on MiniWoB++ only and decode at most 160 tokens.

System	Shop	Admin	Reddit	GitLab	Map
Generator 4B	11.8	8.5	20.7	12.1	12.5
Cascade 4B	17.6	4.3	23.0	15.2	12.5
Generator 8B	10.8	14.5	17.2	16.7	8.3
Cascade 8B	18.6	11.1	19.5	14.4	11.1

Appendix

Table 12: WebArena task success (%) per site.

	System	M2W task	M2W website	M2W domain	GUIAct single	GUIAct multi	Single, ext.	Multi, ext.
4B	Generator	45.8/46.3/46.4	40.2/40.7/41.7	44.1/44.4/44.5	84.8/84.0/83.4	54.4/56.8/57.0	79.4/79.0/78.4	62.8/64.1/63.8
	Cascade 0.9	46.1/46.3/47.2	41.6/42.2/43.1	44.3/44.5/44.5	84.8/83.8/83.4	54.4/56.8/56.5	79.4/78.8/78.2	63.0/64.3/63.7
	Cascade 0.5	46.9/47.9/48.0	43.9/44.2/43.8	44.5/44.9/44.5	84.0/82.3/82.4	53.4/54.4/54.6	78.7/77.4/77.4	61.6/62.5/61.4
	Heads only	39.3/40.0/39.7	38.1/37.8/37.7	39.0/39.3/39.0	82.6/80.0/80.4	52.7/56.5/54.9	77.0/75.0/75.5	60.0/61.7/59.4
8B	Generator	49.6/51.4/50.1	45.1/44.7/43.7	47.8/48.1/48.3	84.7/84.8/83.7	58.7/59.6/58.2	79.1/79.7/78.9	63.5/64.2/64.6
	Cascade 0.9	50.0/51.4/50.4	46.5/46.2/44.9	47.7/48.3/48.3	84.7/84.5/83.3	58.7/59.6/57.7	79.3/79.3/78.5	63.6/64.2/64.3

Appendix

Table 13: Offline step success (%) for seeds 0 / 1 / 2.

MiniWoB++
System	4B	8B
Generator	54.1 / 53.3 / 52.4	56.1 / 55.5 / 55.0
Cascade 0.9	54.1 / 53.1 / 52.1	55.6 / 55.6 / 54.8
Cascade 0.5	52.9 / 53.4 / 51.3	53.1 / 54.3 / 53.4
Heads only	44.9 / 44.9 / 43.9	44.4 / 44.4 / 45.4
WebArena

Appendix

Table 14: Online task success (%) for seeds 0 / 1 / 2.

	Test set	$\tau=0.5$	0.6	0.7	0.8	0.9	0.95	0.99	Generator
4B	M2W task	47.6 (38)	47.6 (30)	47.2 (23)	47.0 (18)	46.6 (13)	46.4 (9)	46.2 (5)	46.2
	M2W website	44.0 (37)	43.9 (29)	43.4 (23)	42.8 (18)	42.3 (12)	42.0 (8)	41.3 (3)	40.9
	M2W domain	44.6 (40)	44.7 (32)	44.6 (25)	44.5 (19)	44.4 (14)	44.3 (10)	44.3 (5)	44.3
	GUIAct single	82.9 (85)	83.2 (79)	83.6 (73)	83.8 (66)	84.0 (56)	84.1 (47)	84.1 (25)	84.1
	GUIAct multi	54.2 (66)	54.7 (57)	55.3 (48)	55.8 (39)	55.9 (27)	55.9 (19)	56.1 (8)	56.1
	Single, ext.	77.9 (87)	78.2 (82)	78.5 (75)	78.7 (69)	78.8 (60)	78.9 (51)	78.9 (33)	78.9

Appendix

Table 15: Step success (%) of the cascade across thresholds, with the share of steps the heads decide in parentheses. For every test step we store the generator’s action and, when

c_{t}\geq 0.5

, the heads’ action (with the generator’s text for WRITE), so one run yields every

\tau\geq 0.5

Split	Op.	Steps	Gen.	Heads	Cascade
Task	Click	1,037	48.5	43.4	48.6
	Type	157	36.1	19.5	45.2
	Select	59	32.2	28.2	36.2
Website	Click	781	45.8	40.8	46.0
	Type	120	24.2	17.5	33.3
	Select	72	14.8	40.3	39.8

Appendix

Table 16: Step success (%) by operation on M2W for the 4B generator, heads only, and the cascade at

\tau=0.5

	Cross-task			Cross-website			Cross-domain
System	Ele.Acc	Op.F1	Step SR	Ele.Acc	Op.F1	Step SR	Ele.Acc	Op.F1	Step SR
MindAct, Flan-T5 XL (HTML)			52.0			38.9			39.6
SeeClick 9.6B			25.5			16.4			20.8
SeeAct, GPT-4V			40.2			32.4			36.8
OmniParser + GPT-4V			39.4			36.5			42.0
Aguvis 7B			60.4			54.6			56.6

Appendix

Table 17: Multimodal-Mind2Web under the standard protocol: element accuracy, operation F1 and step success rate, averaged within tasks and then over tasks, with every test step counted and steps whose gold element lies outside our window counted as failures. Published numbers are copied from MindAct ( Deng et al., 2023 ) , SeeClick ( Cheng et al., 2024 ) , SeeAct ( Zheng et al., 2024 ) , OmniParser ( Lu et al., 2024 ) and Aguvis ( Xu et al., 2024 ) , which use their own candidate sets or full pages; our rows use the windows and candidate lists of Section 4.2 .

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 167 papers

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
Apr 9, 2026Tanmay Gupta, Piper Wolters, Zixian Ma +13Web Agents LLM Agent Training
Allen Institute for AI · University of Washington · UNC-Chapel Hill
Read More, Think More: Revisiting Observation Reduction for Web Agents
Apr 2, 2026Masafumi Enomoto, Ryoma Obara, Haochen Zhang +1Web Agents
NEC Corporation
Structured Distillation of Web Agent Capabilities Enables Generalization
Apr 9, 2026Xing Han Lù, Siva ReddyLLM Agent Training Web Agents
Mila – Quebec AI Institute McGill University

Apr 9, 2026J/K move · Enter open · S save

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko+12

Allen Institute for AI · University of Washington · UNC-Chapel Hill

Web Agents LLM Agent Training Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.

Abstract

Web Agents

Explore similar work

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Read More, Think More: Revisiting Observation Reduction for Web Agents

Structured Distillation of Web Agent Capabilities Enables Generalization