Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

Xunjian Yin; Tianchen Guan; Jinao Wang; Weili Cao; Daisy Xinlei Lin; Royce Cheng-Yue; Keagan Long; Kyle Wong

cs.CLSep 20, 2026

Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

Authors: Xunjian Yin, Tianchen Guan, Jinao Wang, Weili Cao, Daisy Xinlei Lin, Royce Cheng-Yue, Keagan Long, Kyle Wong,

Organizations: Duke University · Amazon

Abstract

As browser-use agents improve, benchmarks keep pace by adding new tasks, websites, and applications. Doing so is costly, and because many factors change at once, it is unclear what makes the new tasks harder. In this work, we instead make existing tasks harder by changing the environment in which they are performed, so that difficulty becomes a programmable property of the environment. We introduce BreakingWeb, in which each task is run twice: once on an unmodified website and once under an intervention, a controlled change to the website's content, server state, network responses, or rendered page. Each intervention is reproducible, keeps the user instruction and success criterion fixed, can be detected and worked around by a competent user, and is labeled with the behavior that recovery from it mainly requires, such as verification. BreakingWeb contains $519$ task pairs on seven self-hosted websites, and each run is scored on the website's final backend state. We evaluate six strong browser-use agents, three agents that observe only screenshots, and human participants. Interventions lower the pass rates of the six browser-use agents by $22.9$ percentage points on average, and each agent fails on nearly half of the tasks it solves on the unmodified website. Humans are affected much less: their pass rate drops by $10.0$ points on the first attempt and by $5.7$ points when they repeat the task. Most failures of the six agents ( $75\%$ ) are belief failures, in which the agent declares the run finished although a required change was never made. Our code, data, and environments are available at https://www.breakingweb.app.

Figures & tables

Figure 1: BreakingWeb constructs challenging browser-use tasks by intervening on the environment. A shopping task is run in the clean condition (top) and under matched interventions that keep the instruction and the success criterion fixed (bottom). Stage labels name the primary primitive of each intervention.

Figure 2: The four injection layers and where each acts in the request path.

Figure 3: Composition of BreakingWeb . (a) Intervention variants by environment and target primitive. (b) Base tasks by environment and difficulty tier. (c) Injection records; multi-layer variants contribute one record per layer they touch. (d) Expected step counts per task by environment.

Primitive	Recovery behavior it requires	Representative interventions
Grounding	Identify the correct UI target among noise, decoys, and adversarial content.	phishing emails, decoys & aliases, label-input misalignment
Planning	Decompose a goal into ordered sub-goals and revise the plan as new information arrives.	scrambled timestamps, missing prerequisites
State tracking	Track which steps are done and which are pending across a long trajectory, including out-of-order updates.	shuffled lists, scrambled timestamps, stale responses
Backtracking	Detect a blocked path, revert to a prior decision point, and try an alternative.	transient errors, 409 conflict, planted wrong answer
Patience	Wait or retry at a suitable rate when responses are slow, unreliable, or rate-limited.	tail latency, progressive delay, transient errors
Exploration	Find another way to act when the obvious path is blocked.	hidden target, hidden affordance, restricted affordance set

Table 1: The seven cognitive primitives, with representative interventions. The examples are illustrative; Appendix C describes the full catalog of

519

variants in

29

intervention families.

		Per primitive
Agent	Total	Grounding	Plan	State.	Back.	Patience	Expl.	Verif.
Gemini-3.1-Pro	44.5 $\downarrow$ 27.6	42.3 $\downarrow$ 34.5	59.6 $\downarrow$ 9.6	48.8 $\downarrow$ 13.1	40.2 $\downarrow$ 37.8	48.0 $\downarrow$ 24.0	37.8 $\downarrow$ 10.8	40.8 $\downarrow$ 39.4
Gemini-3-Flash	42.6 $\downarrow$ 20.6	34.5 $\downarrow$ 22.0	61.5 $\downarrow$ 3.8	47.6 $\downarrow$ 13.1	42.7 $\downarrow$ 35.4	48.0 $\downarrow$ 4.0	43.2 $\downarrow$ 8.1	39.4 $\downarrow$ 33.8
GPT-5.4	35.1 $\downarrow$ 26.2	26.8 $\downarrow$ 28.6	44.2 $\downarrow$ 28.8	36.9 $\downarrow$ 14.3	41.5 $\downarrow$ 35.4	52.0 $\downarrow$ 4.0	29.7 $\downarrow$ 13.5	35.2 $\downarrow$ 36.6
GPT-5.4-mini	15.2 $\downarrow$ 17.9	15.5 $\downarrow$ 14.3	15.4 $\downarrow$ 11.5	17.9 $\downarrow$ 8.3	8.5 $\downarrow$ 36.6	8.0 $\downarrow$ 20.0	18.9 $\downarrow$ 2.7	19.7 $\downarrow$ 28.2
Opus-4.7	31.6 $\downarrow$ 22.9	29.2 $\downarrow$ 24.4	23.1 $\downarrow$ 21.2	31.0 $\downarrow$ 19.0	37.8 $\downarrow$ 29.3	32.0 $\downarrow$ 8.0	27.0 $\downarrow$ 5.4	39.4 $\downarrow$ 32.4

Table 2: Per-primitive intervention pass rate (%). Cell background encodes the relative drop

|\Delta|/\text{pass}_{\textsc{clean}}

<\!15\%

15\text{--}30\%

30\text{--}45\%

\geq\!45\%

. Each cell shows the pass rate under intervention, followed by the change from the clean condition in points (

\downarrow

decrease,

\uparrow

increase). Table 23 adds the clean pass rates and the drops in mean score, and Figure 14 breaks the results down by environment. Rows prefixed v- are GUI-only runs of the same models. The last two rows report the cold and warm human attempts on Human-140 ( Section 5 ).

Figure 4: Distribution of intervention effects. (a) Primitive vulnerability matrix: each cell is the mean paired drop $\Delta=\mathrm{score}_{\mathrm{clean}}-\mathrm{score}_{\mathrm{intervention}}$ on variants whose primary primitive is that primitive, over base tasks with both conditions; bootstrap $95\%$ confidence intervals resample paired (model, task) units. The row maximum is highlighted. (b) Top-5 intervention families ranked by the number of failed runs ( $n\times$ failure rate), pooled across all ten evaluated models (six text-based, three GUI-only, and Qwen3-VL-235B) on intervention buckets with $n\geq 50$ ; bar color encodes the injection layer. The full ranking is in Appendix C . (c) Intervention pass rate per environment and model; Section F.1 gives the numbers.

Figure 5: Failure modes of the nine agents (six text-based agents left of the dashed divider, three GUI-only agents right). (a) Composition of failure modes; coral shades = belief, gold shades = action, navy = silent_overreach . (b) Calibration: finish rate (share of runs that end with a completion report) vs.

P(\text{failed}\mid\text{declared done})

; circles = text-based, squares = GUI-only. (c) Failure classes aggregated by harness.

Mode	Count	Identification rule
misleading success taken	1,454	Final action is a done verb; a positive check failed; final thought contains saved/success/submitted/confirmed/added/starred/completed/done/sent
premature done	433	Final action is a done verb; a positive check failed
retry loop	790	At least $5$ identical consecutive actions
plan collapse	24	No done verb; run length $\leq 5$
silent overreach	336	Final action is a done verb; no positive check failed; a negative check failed
selector hallucination	0	$\geq 3$ action errors; all such runs match an earlier rule in this evaluation

Table 3: Failure-mode taxonomy over the

3{,}037

failed intervention trajectories of the nine agents (six text-based and three GUI-only; the

432

runs labeled abandoned_run are excluded).

Model	Fail $n$	Belief	Action	Over.	FPR
Gemini-3.1-Pro	288	70.8%	3.1%	26.0%	54%
Gemini-3-Flash	298	73.5%	2.3%	24.2%	57%
GPT-5.4	337	76.9%	1.8%	21.4%	65%
GPT-5.4-mini	440	82.7%	2.3%	15.0%	85%
Opus-4.7	245	61.6%	24.5%	13.9%	55%
Sonnet-4.6	233	78.5%	16.7%	4.7%	58%

Table 4: Shares of belief failures, action failures, and overreach among failed intervention runs, per model. FPR is the false-positive rate: the share of runs ending with a completion report in which the task actually failed.

Panel of Figure 6 — Figure 6: Human-vs-agent comparison on Human-140 intervention runs. (a) Time distribution of successful intervention runs (overlapping violins; dashed lines mark medians) and intervention pass rate (legend). (b) Per-primitive human warm pass drop versus mean agent pass drop, with bootstrap $95\%$ confidence intervals over Human-140 base tasks; the dashed line marks the $1{:}1$ diagonal.

Cond.	Attempt	Pass	Time	Events
clean	cold	76.4%	75.7	37
clean	warm	80.7%	32.3	26.5
interv.	cold	66.4%	82.0	47
interv.	warm	75.0%	39.7	32

Table 5: Human-140 panel summary. Time is median wall-clock seconds; Events is median raw browser events.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

## goal: Browse the Electronics category, find the cheapest item available, and

add it to your cart.

## last_action: click('a14')

## last_action_error:

## url: http://localhost:8000/env/amazon/category/electronics?sort=price-asc

RootWebArea "Amazon -- Electronics"

[a3] navigation "Primary"

Appendix

Table 6: Example text-based harness observation on amazon_browse_category , step

3

. The header carries goal, last action, last action error, and URL; the body is the flattened accessibility tree filtered by the rules in Table 20 . Indentation marks DOM nesting; bracketed ids (e.g. [a51] ) are the addressable bid strings.

Model family	Provider docs cited	Viewport (w $\times$ h)	Aspect
Gemini-3.x, Qwen3-VL	default	$1280\times 720$	16:9
Claude (Opus, Sonnet)	Anthropic computer-use	$1024\times 768$	4:3
GPT-5.x, GPT-4o	OpenAI CUA	$1600\times 900$	16:9

Appendix

Table 7: Per-model viewport for the GUI-only harness. Each viewport follows the provider’s documented recommendation. Image scale (low, medium, or high) refers to the quality settings of each API.

Action call	Effect
click(’a51’)	Click the element with bid a51 .
dblclick(’12’)	Double-click the element.
fill(’b22’, ’hello’)	Fill a text field with the given string.
clear(’b22’)	Clear a text field.
focus(’b22’)	Focus an element without clicking.
hover(’d7’)	Hover over an element.

Appendix

Table 8: BID action grammar used by the text-based harness. Every bid argument is a quoted string that must come from the current observation; the harness rejects calls that pass a numeric bid or an out-of-scope identifier.

Action call	Effect
mouse_click(x, y)	Click at $(x,y)$ .
mouse_dblclick(x, y)	Double-click at $(x,y)$ .
mouse_move(x, y)	Move the pointer without clicking.
mouse_drag_and_drop(x1, y1, x2, y2)	Drag from $(x_{1},y_{1})$ to $(x_{2},y_{2})$ .
keyboard_type(’text’)	Type the given string into the focused element.
keyboard_press(’Enter’)	Press a single key.

Appendix

Table 9: Coord action grammar used by the GUI-only harness. Coordinates are pixels for Anthropic and OpenAI models and a

0

–

1000

normalized grid for Gemini and Qwen models ( Section A.2.2 ).

Environment	Domain	Tasks	Endpts.	Graded collections (entity type)
Gmail	Email	84	33	emails ( Email ), drafts , sent , labels , filters , contacts
Amazon	E-commerce	70	56	cart_items , orders ( Order ), returns , addresses , payment_methods , reviews , wishlist
Reddit	Social	81	45	posts ( Post ), comments , messages , notifications , subscriptions , saved_post_ids
Robinhood	Finance	71	57	orders , options_orders , positions , watchlists , transfers , recurring_investments
Booking	Travel	78	53	reservations ( Reservation ), reviews , saved_lists , messages , transactions
LMS	Education	65	46	assignments , grades , discussions , discussion_posts , peer_reviews , enrollments

Appendix

Table 10: Per-environment summary. Tasks is the count of base tasks (each paired one-to-one with a variant). Endpoints counts the public read/write routes the SPA consumes. Graded collections lists the entity collections most often referenced by canonical-diff clauses; entity types are the corresponding Pydantic classes.

Level	Tasks	Med. steps	Med. time (s)	Operational criterion
easy	79	10	180	Single-action goal on one collection (e.g. mark one email as read).
medium	118	15	240	Sequential edit on one or two collections; no set-valued obligation.
hard	132	22	360	Multi-collection goal with at least one set-valued obligation.
expert	98	30	420	Multi-collection goal that requires reading derived state (filtered list, computed total, ranking).
frontier	92	45	540	Cross-collection long-horizon goal with $\geq$ 3 obligations and at least one negative invariant on adjacent state.

Appendix

Table 11: Difficulty taxonomy. Median expected steps and median time limit are computed over all tasks at that level across the seven environments.

Class	Key	Semantics	Example
Scalar	eq	exact (or fuzzy numeric) match	quantity: {eq: 1}
	in	membership in a literal list	status: {in: [paid, pending]}
	between	numeric range, inclusive	rating: {between: [4, 5]}
	any	always true (placeholder)	added_at: {any: true}
	expr	sandboxed Python expression on x / target / state	{expr: "x == target[’id’]"}
Collection	set_eq	set equality	labels: {set_eq: [inbox, work]}

Appendix

Table 12: Canonical-diff predicate grammar. The four scalar predicates and four collection predicates compose with the four text predicates and the five logical/structural combinators to grade any field on any Pydantic state model. The matches_semantic predicate uses a

0.8

-threshold sequence-matcher ratio; eq uses a fuzzy equality that snaps numeric strings.

Environment	Tasks	Avg. inst. words	Avg. exp. steps	Avg. time limit (s)
Gmail	84	103.4	38.9	440
Reddit	81	50.2	29.4	394
Amazon	70	46.3	28.6	358
Booking	78	95.0	24.1	346
Patient Portal	70	45.8	23.4	349
LMS	65	50.2	20.3	323

Appendix

Table 13: Per-environment task statistics. Instruction words are whitespace-split tokens of instruction_template ; expected steps and time limit are task-design estimates stored in the YAML.

Environment	Ground.	Plan.	State	Backtrk.	Patience	Explore	Verify
Amazon	18	7	13	16	4	3	9
Booking	35	4	5	10	5	1	18
Gmail	25	2	13	16	5	9	14
LMS	10	17	10	2	2	10	14
Patient Portal	15	16	11	16		6	6
Reddit	55		11	9			6

Appendix

Table 14: Variant counts by (environment, target primitive). Empty cells are left blank.

Layer	Ground.	Plan.	State	Backtrk.	Patience	Explore	Verify	Total
Seed	182	18	81	17	1	36	26	361
Server	20	31	32	0	6	11	2	102
Network	68	37	47	89	38	16	65	360
Client	11	4	2	0	6	8	8	39

Appendix

Table 15: Injection records by (layer, target primitive). Variants that stack more than one injection layer contribute one record per layer, so the total over all rows exceeds

519

Layer	Families	Dispatches	Primary primitives targeted
Seed	7	14	grounding, state tracking, exploration, backtracking, verification
Server	5	10	planning, state tracking, grounding, exploration, verification
Network	7	8	patience, verification, backtracking, state tracking
Client	10	16	grounding, verification, exploration, backtracking, patience
Total	29	48	all seven primitives

Appendix

Table 16: Intervention families and dispatch branches by injection layer. Primary primitives targeted lists the primitives most often targeted by variants at that layer; every layer reaches multiple primitives.

Layer	Family	Primary primitives	Records	Representative dispatch
Seed	Decoys & aliases	ground., state	294	add_confusing_decoys
	Adversarial content	ground., verify	7	inject_adv_content
	Split information	state	0	split_information
	Contradictory update	state, plan	11	add_contradictory_update
	Content inflation	ground., explore	20	inflate_target_content
	Planted wrong answer	verify, ground.	21	plant_wrong_answer

Appendix

Table 17: The

29

intervention families of BreakingWeb , grouped by injection layer. Records counts the injection records (a multi-layer variant contributes more than one) the family accounts for in the

519

-variant catalog. Representative dispatches name the most-used branch in code; minor specializations (e.g. add_noise_orders for add_confusing_decoys ) fold into the same family.

Model	Snapshot identifier	Provider	Decoding
Claude-Opus-4.7	claude-opus-4-7	Anthropic / Bedrock	default temp.
Claude-Sonnet-4.6	claude-sonnet-4-6	Anthropic / Bedrock	default temp.
GPT-5.4	gpt-5.4	OpenAI	temp. omitted, reasoning_effort=medium
GPT-5.4-mini	gpt-5.4-mini	OpenAI	temp. omitted, reasoning_effort=medium
Gemini-3.1-Pro	gemini-3.1-pro	Google	default temp.
Gemini-3-Flash	gemini-3-flash-preview	Google	default temp.

Appendix

Table 18: Model snapshots and decoding settings used in the reported BreakingWeb sweep. Provider is the route the harness uses; Anthropic models can be served either through the Anthropic API directly or through AWS Bedrock with the same snapshot. All snapshots use max_tokens =

4096

. Temperature is left at the provider default for Anthropic and Google; the OpenAI GPT-5.x family ignores any non-default temperature, so the harness omits the kwarg entirely.

Model	Clean Pass / Score	Intervention Pass / Score	$\Delta$ Pass [95% CI]	$\Delta$ Score [95% CI]
Kimi-K2.5	32.4% / 0.463	18.1% / 0.310	14.3 [10.0, 18.5]	0.153 [0.116, 0.190]
Qwen3-VL-235B	21.2% / 0.352	9.1% / 0.211	12.1 [8.9, 15.6]	0.141 [0.106, 0.176]

Appendix

Table 19: Open-weight text-harness results on all 519 paired tasks. Differences in pass rate are in percentage points; score differences are in canonical-diff score units.

Filter	Effect
with_clickable=True	Annotates clickable nodes with the clickable role marker so the agent can prefer them over decorative elements.
with_visible=True	Annotates each node with its visible flag from the BrowserGym extra-properties dict.
filter_visible_only=True	Drops nodes whose computed visibility is false (off-screen, display:none , or covered).
filter_with_bid_only=True	Drops nodes that lack an addressable bid so the agent cannot emit an action against an unaddressable target.
Post-formatting cleanup:
ASCII-control characters ( \x00 – \x1f excluding tab and newline) are stripped before serialization.

Appendix

Table 20: Accessibility-tree filtering rules applied to the flattened tree the agent observes each step.

Model	Runs	Pass	Median s	Mean s	p90 s	Total hr
Gemini-3.1-Pro	1038	58.3%	189	240	515	69.3
Gemini-3-Flash	1038	52.9%	98	128	260	36.8
GPT-5.4	1038	48.2%	129	165	364	47.7
GPT-5.4-mini	1038	24.2%	61	78	154	22.6
Opus-4.7	1038	43.1%	498	534	999	154.0
Sonnet-4.6	1038	39.2%	692	783	1305	225.6

Appendix

Table 21: Per-model runtime summary on the full

1{,}038

task-condition sweep. Median / mean / p90 are over per-trajectory wall-clock seconds. Total wall-clock is the sum of episode wall-clock across the

1{,}038

runs; it does not include harness overhead or the time spent waiting on per-call provider rate-limits. Token-level billed counts are stored next to each trajectory but are not aggregated into the table because billed-token totals depend on provider-side accounting that we do not redistribute; we therefore do not report a USD figure here.

Asset	Contents
Code	Harness, seven environment SPAs, evaluator, intervention dispatcher, and the scripts that regenerate every figure and table from the trajectory bundle.
Task and variant YAMLs	$519$ base task YAMLs paired one-to-one with $519$ intervention variants drawn from $29$ intervention families.
Trajectory bundles	Per-trajectory raw artifacts and the aggregated agent and human tables on which the paper’s numbers are computed.

Appendix

Table 22: Asset card for the public release: code, task and variant YAMLs, and aggregated trajectory bundles.

		Per primitive
Model	Total	Grnd	Plan	State	Back	Patnce	Expl	Verif
Gemini-3.1-Pro	72.1 $\to$ 44.5 $\Delta$ p +27.6 / $\Delta$ s +0.18	76.8 $\to$ 42.3 $\Delta$ p +34.5 / $\Delta$ s +0.20	69.2 $\to$ 59.6 $\Delta$ p +9.6 / $\Delta$ s +0.08	61.9 $\to$ 48.8 $\Delta$ p +13.1 / $\Delta$ s +0.04	78.0 $\to$ 40.2 $\Delta$ p +37.8 / $\Delta$ s +0.32	72.0 $\to$ 48.0 $\Delta$ p +24.0 / $\Delta$ s +0.17	48.6 $\to$ 37.8 $\Delta$ p +10.8 / $\Delta$ s $-$ 0.02	80.3 $\to$ 40.8 $\Delta$ p +39.4 / $\Delta$ s +0.29
Gemini-3-Flash	63.2 $\to$ 42.6 $\Delta$ p +20.6 / $\Delta$ s +0.14	56.5 $\to$ 34.5 $\Delta$ p +22.0 / $\Delta$ s +0.15	65.4 $\to$ 61.5 $\Delta$ p +3.8 / $\Delta$ s +0.03	60.7 $\to$ 47.6 $\Delta$ p +13.1 / $\Delta$ s +0.09	78.0 $\to$ 42.7 $\Delta$ p +35.4 / $\Delta$ s +0.28	52.0 $\to$ 48.0 $\Delta$ p +4.0 / $\Delta$ s +0.01	51.4 $\to$ 43.2 $\Delta$ p +8.1 / $\Delta$ s $-$ 0.01	73.2 $\to$ 39.4 $\Delta$ p +33.8 / $\Delta$ s +0.24
GPT-5.4	61.3 $\to$ 35.1 $\Delta$ p +26.2 / $\Delta$ s +0.20	55.4 $\to$ 26.8 $\Delta$ p +28.6 / $\Delta$ s +0.22	73.1 $\to$ 44.2 $\Delta$ p +28.8 / $\Delta$ s +0.23	51.2 $\to$ 36.9 $\Delta$ p +14.3 / $\Delta$ s +0.09	76.8 $\to$ 41.5 $\Delta$ p +35.4 / $\Delta$ s +0.29	56.0 $\to$ 52.0 $\Delta$ p +4.0 / $\Delta$ s +0.05	43.2 $\to$ 29.7 $\Delta$ p +13.5 / $\Delta$ s +0.08	71.8 $\to$ 35.2 $\Delta$ p +36.6 / $\Delta$ s +0.30
GPT-5.4-mini	33.1 $\to$ 15.2 $\Delta$ p +17.9 / $\Delta$ s +0.18	29.8 $\to$ 15.5 $\Delta$ p +14.3 / $\Delta$ s +0.15	26.9 $\to$ 15.4 $\Delta$ p +11.5 / $\Delta$ s +0.14	26.2 $\to$ 17.9 $\Delta$ p +8.3 / $\Delta$ s +0.11	45.1 $\to$ 8.5 $\Delta$ p +36.6 / $\Delta$ s +0.36	28.0 $\to$ 8.0 $\Delta$ p +20.0 / $\Delta$ s +0.19	21.6 $\to$ 18.9 $\Delta$ p +2.7 / $\Delta$ s +0.03	47.9 $\to$ 19.7 $\Delta$ p +28.2 / $\Delta$ s +0.22
Opus-4.7	54.5 $\to$ 31.6 $\Delta$ p +22.9 / $\Delta$ s +0.19	53.6 $\to$ 29.2 $\Delta$ p +24.4 / $\Delta$ s +0.20	44.2 $\to$ 23.1 $\Delta$ p +21.2 / $\Delta$ s +0.27	50.0 $\to$ 31.0 $\Delta$ p +19.0 / $\Delta$ s +0.15	67.1 $\to$ 37.8 $\Delta$ p +29.3 / $\Delta$ s +0.24	40.0 $\to$ 32.0 $\Delta$ p +8.0 / $\Delta$ s $-$ 0.01	32.4 $\to$ 27.0 $\Delta$ p +5.4 / $\Delta$ s +0.01	71.8 $\to$ 39.4 $\Delta$ p +32.4 / $\Delta$ s +0.25

Appendix

Table 23: Per-(model, primitive) clean-and-intervention breakdown, full version of Table 2 . Each cell carries: the clean pass rate, the matched intervention pass rate (separated by

\to

), the paired

\Delta

in pass rate (

\Delta

p, in points;

\downarrow

in the main table corresponds to a positive value here), and the paired

\Delta

in mean canonical-diff score (

\Delta

s, score units). All numbers are computed only over base tasks with both conditions available.

Mode	Ground.	Plan.	State	Backtrk.	Patience	Explore	Verify
misleading_success_taken	443	91	132	216	37	66	149
premature_done	72	28	42	37	10	21	36
retry_loop	36	14	20	9	21	13	10
plan_collapse	0	0	3	2	2	0	1
silent_overreach	85	28	87	25	14	34	57

Appendix

Table 24: Failure-mode counts by target primitive (

1{,}841

failed intervention runs across the six text-based agents; the additional

1{,}196

GUI-only failures are absorbed into Figure 5 (c)).

Model	Mean steps (fail)	Mean steps (pass)	Fail $-$ Pass
Gemini-3.1-Pro	15.2	13.4	$+1.8$
Gemini-3-Flash	16.9	11.7	$+5.2$
GPT-5.4	14.6	14.6	$-0.1$
GPT-5.4-mini	10.1	9.4	$+0.7$
Opus-4.7	13.0	10.3	$+2.7$
Sonnet-4.6	13.8	9.7	$+4.1$

Appendix

Table 25: Mean steps on backtracking-targeted intervention runs, partitioned by outcome. The fail-minus-pass column is positive for every model except GPT-5.4 (

-0.1

Figure 11: Per-task human intervention tax

\tau^{\mathrm{human}}_{t}

(warm log-time ratio) versus mean agent drop

\Delta^{\mathrm{agent}}_{t}

over the

140

paired tasks of Human-140, averaged over the six text-based models. The aggregate per-primitive view appears in Figure 6 (b).

Merged group	Original primitives	Rationale
Grounding	grounding	observation / disambiguation primitive, kept distinct
Search/Planning	planning, exploration	both involve finding or sequencing paths
State reasoning	state_tracking, verification	both involve reasoning about state
Recovery	backtracking	recovery after a wrong or blocked path is its own behavior
Temporal robustness	patience	waiting / retrying under latency is distinct from recovery

Appendix

Table 26:

7\rightarrow 5

primitive merge mapping used in the robustness check.

Figure 12: Coarsening the seven BreakingWeb primitives to five groups preserves the broad vulnerability pattern, but merging planning with exploration and state tracking with verification hides nontrivial within-group contrasts. (a) Five-way merged paired $\Delta$ score heatmap on the six text-based agents; row-maximum cells are starred. (b) Within-merge diagnostic loss for the two real merges, averaged across the six text agents.

Figure 13: Intervention pass rate (%) by environment and model. Cells report the fraction of intervention runs that fully passed. “–” marks (model, environment) pairs with no runs in the intervention sweep.

Environment	Ground.	Plan.	State	Backtrk.	Patience	Explore	Verify
Amazon	0.23	0.39	0.17	0.45	-0.01	0.01	0.47
Booking	0.07	0.02	0.09	0.12	-0.06	0.10	0.23
Gmail	0.08	-0.04	0.03	-0.00	0.09	-0.01	0.27
LMS	0.08	0.12	0.07	0.21	0.00	-0.00	0.12
Patient Portal	0.18	0.20	0.12	0.34		0.02	0.26
Reddit	0.35		0.25	0.45			0.42

Appendix

Table 27: Mean paired score drop

\Delta=\mathrm{score}_{\mathrm{clean}}-\mathrm{score}_{\mathrm{intervention}}

by (environment, target primitive), averaged across the six text-based agents. Cells with no targeting variants in the catalog are left blank; cells with fewer than five base tasks per model in the sweep are italicized.

Figure 14: Per-(environment, primitive) intervention pass rate, broken down by agent. Each subplot is one model; rows are the seven environments and columns the seven primitives. Cells annotate the intervention-condition pass rate (%); cells with no targeting variants in that environment are rendered gray. Sage = high pass; coral = low pass; the same colormap is shared across subplots so any two cells are directly comparable.

Environment ( $n$ )	mislead. succ.	prem. done	retry_loop	plan_coll.	silent_over.
Amazon (237)	69.2%	15.6%	3.0%	0.0%	12.2%
Booking (229)	68.6%	8.7%	4.4%	1.3%	17.0%
Gmail (312)	70.5%	13.1%	9.6%	1.3%	5.4%
LMS (190)	65.3%	14.7%	3.2%	0.0%	16.8%
Patient Portal (226)	72.6%	16.8%	4.9%	0.4%	5.3%
Reddit (404)	57.9%	11.6%	4.7%	0.0%	25.7%

Appendix

Table 28: Failure-mode mixture by environment, as a percentage of failed intervention runs in that environment, summed across the six text-based agents. Rows sum to

100

% within rounding. The first column reports the per-environment failed-run count.

Case	Category	Target primitive	Layer	Model	Environment
A	PR / BF	grounding	seed	Opus-4.7	Amazon
B	PR / BF	backtracking, verification	network	Gemini-3.1-Pro	Robinhood

Appendix

Table 29: Index of the deep-trace cases. Category groups cases for readability: belief-failure exemplars (BF), one case per primitive (PR).

Panel of Figure 15 — Figure 15: Case A. Grounding/verification deficit, Opus-4.7 on amazon_price_comparison ( $\Delta=1.0$ ). Six representative steps from the $14$ -step trajectory. The seed/decoys_aliases intervention places three near-identical $5.0$ -star products; Opus correctly sorts by rating (step 5) but commits to the first sort-result entry at step 6 without disambiguating, then carries that decoy through cart, checkout, and the terminal send_msg_to_user that confidently restates the wrong product as “highest-rated”.

Panel of Figure 16 — Figure 16: Case B. Backtracking/verification deficit, Gemini-3.1-Pro on rh_sell_loser_buy_winner ( $\Delta=1.0$ ). Six representative steps from the $22$ -step trajectory. The network/transient_error variant first injects a real $503$ at step 4 (Gemini correctly recovers by switching to advanced trade at step 5 and a limit order at step 9), and then returns a forged $200$ with body status=pending starting at step 11. The agent treats the toast as completion and declares both legs done at step 22, even though no fill ever occurs.

Item	Description
Clarity ( $1$ – $5$ )	Was the instruction understandable as written?
Realism ( $1$ – $5$ )	Does this feel like a task a real user would attempt on the corresponding production website?
Fun / demo value ( $1$ – $5$ )	Would this task work as a demo for others?
Intervention naturalness ( $1$ – $5$ )	Intervention runs only: did the complication feel plausibly like something a real user might encounter?
Issue flags (binary):
Suspected bug	Evaluator failed on what looked like a clean success, or the intervention behaved unexpectedly.

Appendix

Table 30: Post-task rating instrument. Ordinal items use a

1

–

5

Likert scale; the four issue boxes are independent binary flags.

Statistic	Value
Duplicate task-conditions covered	35
Total duplicate attempts	70
Pairwise warm success agreement	68.6 %
Median duplicate-warm raw-event ratio	1.58
Pairs within $1.25\times$ warm raw events	28.6 %
Pairs within $1.5\times$ warm raw events	42.9 %

Appendix

Table 31: Duplicate-audit summary over the

35

duplicated task-conditions (

70

attempts), contributed by four non-designer annotators.

Annotator	Cold clean	Warm clean	Cold intv.	Warm intv.
A1	88.6%	85.7%	68.6%	71.4%
A2	82.9%	82.9%	74.3%	71.4%
A3	74.3%	80.0%	62.9%	80.0%
A4	60.0%	74.3%	60.0%	77.1%

Appendix

Table 32: Per-annotator marginal pass rates on the Human-140 primary panel. Numbers are over the cells the annotator covered; the four annotators do not necessarily cover the same (env, difficulty) cells.

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
Aug 9, 2026Zejun Xu, Taiyi Chen, Jin Li +13Computer-Use Agents Web Agent Benchmarks
Macau University of Science and Technology · FellouAI · Tsinghua University +3
WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
Apr 13, 2026Peng Yuan, Yuyang Yin, Yuxuan Cai +1Web Agent Benchmarks Benchmark Design
Tencent BAC · Tsinghua University
Back to Basics: A Skill-Centric Approach to Generalist Web Agents
Oct 26, 2026Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui +2Web Agents LLM Agent Skill Learning
University of Utah · Islamic University of Technology

Aug 9, 2026J/K move · Enter open · S save

CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

Zejun Xu, Taiyi Chen, Jin Li, Yongtong Gu+12

Macau University of Science and Technology · FellouAI · Tsinghua University+3

Computer-Use Agents Web Agent Benchmarks

Open paper AskPDF arXiv

Large language models are increasingly deployed as autonomous agents that interact with the web through browsers. While recent progress has been driven by benchmarks that evaluate end-to-end task success, these evaluations largely overlook two fundamental sources of difficulty in real web browsing: complex actions over rich user interfaces and visual perception of dynamically rendered content, especially in workflows that span multiple websites. We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks that require non-trivial UI interactions and visual understanding. Specifically, we adopt a decomposition-and-recomposition pipeline that first abstracts each website into a structured site card capturing user-facing functions, complex execution operations, and perceptual requirements, and then recomposes these components into realistic cross-site workflows. Each task is therefore grounded in multiple specific operations on each website, enabling fine-grained diagnosis. Built on this framework, we construct 420 tasks across 108 real-world websites and 24 domains under careful quality control. Experiments on state-of-the-art browser agents using our verifiable agent-as-a-judge evaluation framework show low success rates and reveal that perception-heavy interactions remain a major bottleneck, exposing substantial gaps between current agents and real-world web browsing demands.

Abstract

Web Agents

Explore similar work

CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark

Back to Basics: A Skill-Centric Approach to Generalist Web Agents