Organizations: INSA Lyon, LIRIS · Esker · Seoul National University · McGill University · Mila – Quebec AI Institute · The Ohio State University · University of Hong Kong · ServiceNow AI Research · Université Laval · Canada CIFAR AI Chair
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io
Figures & tables
Figure 1: Overview of WebArena-Pro. (A) Domain heterogeneity. WebArena-Pro spans 20 self-hosted web applications across six domain categories (productivity, communication, analytics, business operations, design and media, and travel), each modeled on a widely used service such as Notion, Slack, Spotify, Figma, or Airbnb. The full set of applications and their reference services are listed in Table 2 . (B) Multimodal observation and action space. WebArena-Pro extends the agent’s observation space beyond text and vision to include audio and video. Embedded media sources are exposed to the agent through the accessibility tree, and agents invoke transcribe() and extract_frames() actions to convert temporal media into a form usable during browser workflows. If the agents handle native audio and video processing, they invoke listen() and watch() actions.
Tasks
Modalities
Eval & Execution
Benchmark
# Envs
Templates
Instances
Vision
Audio
Video
Isolation
Human-curated
WebArena ( Zhou et al., 2023 )
6
241
812
✗
✗
✗
✗
✓
VisualWebArena ( Koh et al., 2024 )
3
233
910
✓
✗
✗
✗
✓
WebChoreArena ( Miyai et al., 2025 )
4
117
532
✓
✗
✗
✗
✓
WorkArena Suite § ( Boisvert et al., 2024 ; Drouin et al., 2024 )
1
715
19,912
✓
✗
✗
◐
✓
TheAgentCompany ( Xu et al., 2025 )
4
175
175
✓
✗
✗
✓
✓
Table 1: Comparing WebArena-Pro to existing reproducible benchmarks for web and computer-use agents. “ Isolation ” refers to per-task environment reset enabling contamination-free parallel execution. “ Human-curated ” indicates that tasks are authored or verified by humans rather than auto-generated. A half-circle (◐) indicates partial support. § WorkArena Suite combines WorkArena (33 templates instantiated to 19,912 instances) and WorkArena++ (682 compositional tasks, each sampling from thousands of valid configurations). † OSWorld targets desktop applications rather than websites, with “# Envs” showing its 9 application domains. ‡ Gym-Anything’s environments and tasks are LLM-generated at scale rather than human-curated.
Domain
Site
Reference application
Description
Productivity
MyDrive
Dropbox
Personal cloud file storage and sharing
NextCloud
Google Drive
Collaborative file storage with calendars, contacts, and chat
Notion
Notion
Notes and knowledge-management workspace
Password-Manager
1Password
Password and credential vault
Communication
Mattermost
Slack
Team messaging with channels and direct messages
Thunderbird
Gmail
Email client with folders and contacts
Table 2: The 20 self-hosted web applications in WebArena-Pro, grouped by domain. Each application is a self-hosted instance whose interface and workflow patterns are modeled on a widely-used reference application, while data and exact feature scope are controlled by the benchmark.
Coverage
Modality
Sites
Example intent
Single-site
Text/Vision
Flight-Search
Set the displayed currency to USD. Find the fastest one-way economy flight from New York (JFK) to Los Angeles (LAX) on 2026-06-15. If multiple flights are tied for the shortest total travel time, choose the first tied flight shown after sorting by Fastest.
Audio/Video
YT-Lite
In the video "Tips for winter biking", does the creator record himself from a first person point of view?
Cross-site
Text/Vision
Notion, Peppermint
Create a Notion page summarizing the VPN drops ticket and naming its assigned owner.
Audio/Video
Mattermost, Acidwave
Send Mike a direct message recommending a rock song suitable for someone going through a breakup.
Table 3: Qualitative examples of WebArena-Pro tasks, organized by site coverage (single-site or cross-site) and modality (text/vision or audio/video).
#
Type
Criterion
1
Action
The agent viewed the Project Alpha Talk history.
2
Action
The agent typed “Client Meeting Update” as the name for a new file.
3
Outcome
The content of the “Client Meeting Update” file states that the client meeting moved to next Thursday.
4
Outcome
A file named “Client Meeting Update” exists in the Nextcloud file directory.
Table 4: Qualitative rubric example from the NextCloud task configuration. The judge verifies whether the agent performed actions 1 and 2, and verifies outcomes 3 and 4 at each step.
Figure 2: Qualitative example of a Qwen3.6-27B agent trajectory in WebArena-Pro. Given a single-site audio/video task on YT-Lite, namely “ In the video, Zener Diodes: The Unsung Heroes of Electronics, was there any act of soldering shown? ”, the agent first clicks the matching video thumbnail. On the video page, it invokes transcribe() to read the spoken content and extract_frames() to sample 10 representative frames. The transcript does not mention soldering, but the extracted frames at t=128.6 s and t=144.7 s show a soldering iron, and the agent answers accordingly. The rubric verifier confirms a passing trajectory.
Domains SR (%)
Agent
Model
Prod.
Comm.
Anlt.
BusOps
Design
Travel
Overall SR (%)
GenericAgent
Qwen 3.5 9B
3.8
9.1
12.5
3.2
4.1
3.8
5.0
Qwen 3.6 27B
5.7
22.7
12.5
7.9
15.3
15.4
13.0
Qwen 3.8 27B
7.5
15.9
12.5
14.3
19.4
23.1
15.7
Gemma4 26B A4B
18.9
25.0
37.5
23.8
17.3
30.8
22.3
Gemma4 31B
18.9
38.6
37.5
30.2
30.6
34.6
30.3
Table 5: Success rates (SR) of agents on WebArena-Pro at 50-step budget, broken down by domain category. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Success requires a final reward of 1. Domain task counts are 53, 44, 16, 63, 98, and 26, respectively (300 overall). † Omni uses watch() and listen() functions.
Site Coverage
Modality
Agent
Model
Single-site (%)
Cross-site (%)
Text/Vision (%)
Audio/Video (%)
GenericAgent
Qwen 3.5 9B
5.4
4.2
5.0
5.1
Qwen 3.6 27B
17.2
4.2
10.9
19.0
Qwen 3.8 27B
20.6
5.2
12.2
25.3
Gemma4 26B A4B
27.5
11.5
24.0
17.7
Gemma4 31B
38.2
13.5
29.9
31.6
Table 6: Success rate (SR) of agents on WebArena-Pro at a 50-step budget, decomposed along the two task axes of Section 3.2 . The site coverage axis splits the 300 Test tasks into 204 single-site and 96 cross-site tasks, and the modality axis splits them into 221 text/vision and 79 audio/video tasks. Best values are bold, and second-best are underlined.
All Tasks
Media-Content-Dependent Subset
Model
Configuration
AcidWave
YT-Lite
Overall
Audio
Video
Overall
( n=35 )
( n=44 )
( n=79 )
( n=20 )
( n=23 )
( n=43 )
Gemini3.1 Pro
No media tools
11.4
20.5
16.5
0.0
4.3
2.3
T/F only
22.9
50.0
38.0
5.0
52.2
30.2
W/L only
25.7
54.5
41.8
10.0
56.5
34.9
Qwen 3.6 27B
No media tools
5.7
2.3
3.8
0.0
0.0
0.0
Table 7: Media-tool ablations on the Test split with a 50-step cap. Values report full-task SR (%; final reward equals 1). Results are shown for all evaluated tasks and a conservative, post hoc media-content-dependent subset selected by auditing task metadata, thumbnails, and grading criteria. T/F denotes transcribe() and extract_frames() ; W/L denotes watch() and listen() .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Tool / capability
Claude Code
Codex
OpenCode
Browser tools
Snapshot, screenshot, and page search
Yes
Yes
Yes
Navigate, go back, and manage tabs
Yes
Yes
Yes
Element-based click, hover, and drag
Yes
Yes
Yes
Type, fill forms, select options, and press keys
Yes
Yes
Yes
Handle dialogs, wait, and upload staged files
Yes
Yes
Yes
Appendix
Table 8: Configured action spaces of the three coding agents. Availability refers to the evaluated runs.
Figure 3: Tool-call frequencies by action category on the WebArena-Pro Test split (300 tasks per coding agent). Bars show total recorded browser, media, and native tool invocations, including observation calls. Submission calls and Codex’s exec / wait orchestration wrappers are excluded.
Figure 4: Average tool-call frequencies by action category per task. Bars show the average recorded browser, media, and native tool invocations, including observation calls. Submission calls and Codex’s exec / wait orchestration wrappers are excluded.
Domains SR (%)
Agent
Model
Prod.
Comm.
Anlt.
BusOps
Design
Travel
Overall SR (%)
GenericAgent
Qwen 3.6 27B
5.7
22.7
12.5
7.9
15.3
15.4
13.0
Gemini 3.1 Pro
28.3
27.3
43.8
20.6
33.7
30.8
29.3
Appendix
Table 9: Success rates (SR) of agents on WebArena-Pro at 50-step budget, broken down by domain category. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Success requires a final reward of 1. Domain task counts are 53, 44, 16, 63, 98, and 26, respectively (300 overall).
Domains (%)
Agent
Model
Prod.
Comm.
Anlt.
BusOps
Design
Travel
Overall (%)
GenericAgent
Qwen 3.5 9B
17.4
25.7
21.0
9.1
18.9
14.6
17.3
Qwen 3.6 27B
17.6
33.6
31.9
17.3
24.7
38.5
24.8
Qwen 3.8 27B
20.6
31.2
30.1
23.7
33.2
44.2
29.5
Gemma4 26B A4B
38.2
39.9
48.8
33.8
29.9
52.5
36.6
Gemma4 31B
42.3
48.5
61.3
43.5
45.1
49.2
46.0
Appendix
Table 10: Partial completion scores of agents on the WebArena-Pro test set (300 tasks) with a 50-step budget, broken down by domain. Scores are 100 times the mean final task reward. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel. Best and second-best overall values are bold and underlined, respectively.
Domains (%)
Model
Prod.
Comm.
Anlt.
BusOps
Design
Travel
Overall (%)
Success rate
Qwen 3.5 9B
10.0
26.7
57.1
13.3
37.5
7.1
28.0
Qwen 3.6 27B
40.0
53.3
71.4
40.0
53.1
64.3
54.0
Gemma4 26B A4B
20.0
73.3
100.0
66.7
46.9
57.1
60.0
Gemma4 31B
30.0
80.0
100.0
66.7
71.9
85.7
74.0
Appendix
Table 11: Success rates and partial completion scores of agents (variants of GenericAgent) on the WebArena-Pro dev set (100 tasks) with a 50-step budget. Domain columns abbreviate Productivity (Prod.), Communication (Comm.), Analytics (Anlt.), Business Operations (BusOps), Design & Media (Design), and Travel.
Figure 5: Trajectory length distributions of GenericAgent variants for solved and unsolved tasks on the WebArena-Pro Test split (300 tasks per model), with a 50-step budget. These variants include additional results of Table 9 .
Failure mode
Evidence
Gemini Pro omits required artifacts.
Task 225, Peppermint → Notion: The task required a Notion page summarizing a VPN ticket and naming its owner. All 50 actions remained in Peppermint; the agent never opened Notion or created the page.
Gemini Pro selects incorrect target entities.
Task 232, PipeCRM → Thunderbird: The task required emailing Leon Green at [email protected] . The agent instead sent the email to [email protected] . The recipient rubric failed, although the weekend-question content passed.
Both Gemma models exhibit empty finalizations.
Task 229, PipeCRM: Gemma 4 26B A4B and Gemma 4 31B both executed click(’739’) followed by done() . The requested deal’s close date, February 21, 2025, and status, OPEN , were visible, but neither model sent an answer.
Qwen 3.6 misses required end-state actions.
Task 22, Acidwave: The agent created VOCALS_CONFIRMED and added “STAYIN’ AT HOME - VOCALS,” then called done() after seven actions. The required “FLYING AWAY (VOCALS)” was absent.
Qwen 3.6 selects incorrect entities or values.
Task 52, Firefly → Thunderbird: The task required emailing [email protected] that Spotify Premium costs C11.99/month.Theagentusedbob@example.comand9.99, clicked Send, and reported completion.
Appendix
Table 12: Evidence for model-level failure claims in test runs for GenericAgent.
Failure pattern
Example website(s)
Illustrative evidence
Budget exhaustion
Across applications
Of Claude Opus 4.7’s 224 unsuccessful Test runs, 155 reached the 50-step limit.
Incomplete workflow / navigation failure
Nextcloud, Mattermost
An agent failed to open the budget file in Nextcloud and never navigated to Mattermost to message Jane.
Wrong answer or entity
Acidwave
An agent answered 00:00 for a folk-song duration task whose expected answer was 33:22 .
Filter, sort, or exact-constraint error
Metabase
An agent built a product-rating table but filtered Gadget instead of the required category Electronics .
Skipped verification
YT-Lite
An agent answered using a different Warhammer video without inspecting the requested Warhammer III video.
Premature termination
Peppermint
An agent declared a database-comment task infeasible without opening the target discussion.
Appendix
Table 13: Representative failure patterns and illustrative evidence of GenericAgent variants.
Figure 6: Trajectory length distributions of coding agents for solved and unsolved tasks on the WebArena-Pro Test split (300 tasks per model), with a 50-step budget.
Failure mode
Evidence
Incorrect absence claim.
Claude Code / Opus 5.5, Task 178, Notion: Asked to publish the “my summer vacation” page, the agent concluded that the page did not exist and stopped without publishing it. The other agents successfully published the requested page in their selected runs.
Local artifacts not fully delivered to the application.
Codex / GPT 6.1 Sol, Task 147, MyDrive: The agent generated all six requested algorithm files locally, but did not complete their creation in MyDrive. Its final answer acknowledged that Q1_javascript.js was still missing from the target folder.
Budget exhaustion before committing the result.
OpenCode / Kimi 3.1 Pro, Task 046, ERPNext: The agent gathered order and contact information from Metabase and PipeCRM and prepared a Sales Order. It reached 50 steps before saving; the terminal screenshot showed the order marked “Not Saved.”
Incorrect visual event counting.
OpenCode / Gemini 3.1 Pro, Task 317, YT-Lite: Asked how many times a rapper appeared holding a Sprite, the agent completed 12 frame-extraction calls but answered 11 , whereas the rubric required 9 . Media access succeeded, but the final count was incorrect under the grading criterion.
Appendix
Table 14: Examples of observed task failures across five coding-agent configurations.
Judge Agreement Rate (%)
Agent Model
Qwen 3.6 27B
Gemini 3.1 Pro
Kimi 2.6
Gemini 3.1 Pro
96.9
92.3
94.6
Gemini 3.1 Flash Lite
87.7
91.5
90.8
Claude Opus 4.7
93.8
92.2
92.2
Gemma4 31B
83.1
85.4
86.9
Qwen 3.5 9B
92.3
92.3
93.1
Appendix
Table 15: Rubric-level agreement between each judge and human annotations. Each row corresponds to the agent model that produced the trajectories, with 50 trajectories and about 130 annotated rubrics per agent. The pooled row averages the six agents.
Website
Construction basis / upstream repository
Application/upstream license
Compressed image size
acidwave
AI-assisted custom
MIT
424 MB
erp-next
Open-source ERPNext/Frappe
GPL-3.0 (ERPNext); MIT (Frappe)
627 MB
firefly
Open-source Firefly III
AGPL-3.0
347 MB
flight-search
AI-assisted edited from airline-route-data
MIT
64.0 MB
immich
Open-source Immich
AGPL-3.0
7.03 GB
jobportal
AI-assisted edited from jobportal
MIT
85.6 MB
Appendix
Table 16: Compressed Docker image sizes, upstream sources, and application licences for WebArena-Pro website environments.
Function
Claude Code Opus 5.5
Codex GPT 6.1 Sol
OpenCode Gemini 3.1 Pro
OpenCode Qwen 3.8 Max Prime
OpenCode Kimi 3.1 Pro
Browser tools
browser_snapshot
296
1 036
423
312
300
browser_take_screenshot
76
89
8
30
12
browser_find
1 307
809
357
273
305
browser_navigate
1 753
954
1 099
1 466
1 153
browser_navigate_back
7
18
53
13
40
Appendix
Table 17: Function-call frequencies across the 300 selected Test trials per coding agent. Entries are absolute invocation counts, including free observations, rather than budgeted steps.
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.
Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.
Tanmay Gupta, Piper Wolters, Zixian Ma +13
Allen Institute for AI · University of Washington · UNC-Chapel Hill
Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io
Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation that limits scalability. We present WebForge, the first fully automated framework that resolves this trilemma through a four-agent pipeline -- Plan, Generate, Refine, and Validate -- that produces interactive, self-contained web environments end-to-end without human annotation. A seven-dimensional difficulty control framework structures task design along navigation depth, visual complexity, reasoning difficulty, and more, enabling systematic capability profiling beyond single aggregate scores. Using WebForge, we construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels. Multi-model experiments show that difficulty stratification effectively differentiates model capabilities, while cross-domain analysis exposes capability biases invisible to aggregate metrics. Together, these results confirm that multi-dimensional evaluation reveals distinct capability profiles that a single aggregate score cannot capture. Code and benchmark are publicly available at https://github.com/yuandaxia2001/WebForge.