WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents
Organizations: INSA Lyon, LIRIS · Esker · Seoul National University · McGill University ·
Abstract
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io
Figures & tables
| Tasks | Modalities | Eval & Execution | ||||||
| Benchmark | # Envs | Templates | Instances | Vision | Audio | Video | Isolation | Human-curated |
| WebArena ( Zhou et al., 2023 ) | 6 | 241 | 812 | ✗ | ✗ | ✗ | ✗ | ✓ |
| VisualWebArena ( Koh et al., 2024 ) | 3 | 233 | 910 | ✓ | ✗ | ✗ | ✗ | ✓ |
| WebChoreArena ( Miyai et al., 2025 ) | 4 | 117 | 532 | ✓ | ✗ | ✗ | ✗ | ✓ |
| WorkArena Suite § ( Boisvert et al., 2024 ; Drouin et al., 2024 ) | 1 | 715 | 19,912 | ✓ | ✗ | ✗ | ◐ | ✓ |
| TheAgentCompany ( Xu et al., 2025 ) | 4 | 175 | 175 | ✓ | ✗ | ✗ | ✓ | ✓ |
| Domain | Site | Reference application | Description |
| Productivity | MyDrive | Dropbox | Personal cloud file storage and sharing |
| NextCloud | Google Drive | Collaborative file storage with calendars, contacts, and chat | |
| Notion | Notion | Notes and knowledge-management workspace | |
| Password-Manager | 1Password | Password and credential vault | |
| Communication | Mattermost | Slack | Team messaging with channels and direct messages |
| Thunderbird | Gmail | Email client with folders and contacts |
| Coverage | Modality | Sites | Example intent |
| Single-site | Text/Vision | Flight-Search | Set the displayed currency to USD. Find the fastest one-way economy flight from New York (JFK) to Los Angeles (LAX) on 2026-06-15. If multiple flights are tied for the shortest total travel time, choose the first tied flight shown after sorting by Fastest. |
| Audio/Video | YT-Lite | In the video "Tips for winter biking", does the creator record himself from a first person point of view? | |
| Cross-site | Text/Vision | Notion, Peppermint | Create a Notion page summarizing the VPN drops ticket and naming its assigned owner. |
| Audio/Video | Mattermost, Acidwave | Send Mike a direct message recommending a rock song suitable for someone going through a breakup. |
| # | Type | Criterion |
| 1 | Action | The agent viewed the Project Alpha Talk history. |
| 2 | Action | The agent typed “Client Meeting Update” as the name for a new file. |
| 3 | Outcome | The content of the “Client Meeting Update” file states that the client meeting moved to next Thursday. |
| 4 | Outcome | A file named “Client Meeting Update” exists in the Nextcloud file directory. |
| Domains SR (%) | ||||||||
| Agent | Model | Prod. | Comm. | Anlt. | BusOps | Design | Travel | Overall SR (%) |
| GenericAgent | Qwen 3.5 9B | 3.8 | 9.1 | 12.5 | 3.2 | 4.1 | 3.8 | 5.0 |
| Qwen 3.6 27B | 5.7 | 22.7 | 12.5 | 7.9 | 15.3 | 15.4 | 13.0 | |
| Qwen 3.8 27B | 7.5 | 15.9 | 12.5 | 14.3 | 19.4 | 23.1 | 15.7 | |
| Gemma4 26B A4B | 18.9 | 25.0 | 37.5 | 23.8 | 17.3 | 30.8 | 22.3 | |
| Gemma4 31B | 18.9 | 38.6 | 37.5 | 30.2 | 30.6 | 34.6 | 30.3 | |
| Site Coverage | Modality | ||||
| Agent | Model | Single-site (%) | Cross-site (%) | Text/Vision (%) | Audio/Video (%) |
| GenericAgent | Qwen 3.5 9B | 5.4 | 4.2 | 5.0 | 5.1 |
| Qwen 3.6 27B | 17.2 | 4.2 | 10.9 | 19.0 | |
| Qwen 3.8 27B | 20.6 | 5.2 | 12.2 | 25.3 | |
| Gemma4 26B A4B | 27.5 | 11.5 | 24.0 | 17.7 | |
| Gemma4 31B | 38.2 | 13.5 | 29.9 | 31.6 | |
| All Tasks | Media-Content-Dependent Subset | ||||||
| Model | Configuration | AcidWave | YT-Lite | Overall | Audio | Video | Overall |
| ( ) | ( ) | ( ) | ( ) | ( ) | ( ) | ||
| Gemini3.1 Pro | No media tools | 11.4 | 20.5 | 16.5 | 0.0 | 4.3 | 2.3 |
| T/F only | 22.9 | 50.0 | 38.0 | 5.0 | 52.2 | 30.2 | |
| W/L only | 25.7 | 54.5 | 41.8 | 10.0 | 56.5 | 34.9 | |
| Qwen 3.6 27B | No media tools | 5.7 | 2.3 | 3.8 | 0.0 | 0.0 | 0.0 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool / capability | Claude Code | Codex | OpenCode |
| Browser tools | |||
| Snapshot, screenshot, and page search | Yes | Yes | Yes |
| Navigate, go back, and manage tabs | Yes | Yes | Yes |
| Element-based click, hover, and drag | Yes | Yes | Yes |
| Type, fill forms, select options, and press keys | Yes | Yes | Yes |
| Handle dialogs, wait, and upload staged files | Yes | Yes | Yes |
| Domains SR (%) | ||||||||
| Agent | Model | Prod. | Comm. | Anlt. | BusOps | Design | Travel | Overall SR (%) |
| GenericAgent | Qwen 3.6 27B | 5.7 | 22.7 | 12.5 | 7.9 | 15.3 | 15.4 | 13.0 |
| Gemini 3.1 Pro | 28.3 | 27.3 | 43.8 | 20.6 | 33.7 | 30.8 | 29.3 | |
| Domains (%) | ||||||||
| Agent | Model | Prod. | Comm. | Anlt. | BusOps | Design | Travel | Overall (%) |
| GenericAgent | Qwen 3.5 9B | 17.4 | 25.7 | 21.0 | 9.1 | 18.9 | 14.6 | 17.3 |
| Qwen 3.6 27B | 17.6 | 33.6 | 31.9 | 17.3 | 24.7 | 38.5 | 24.8 | |
| Qwen 3.8 27B | 20.6 | 31.2 | 30.1 | 23.7 | 33.2 | 44.2 | 29.5 | |
| Gemma4 26B A4B | 38.2 | 39.9 | 48.8 | 33.8 | 29.9 | 52.5 | 36.6 | |
| Gemma4 31B | 42.3 | 48.5 | 61.3 | 43.5 | 45.1 | 49.2 | 46.0 | |
| Domains (%) | |||||||
| Model | Prod. | Comm. | Anlt. | BusOps | Design | Travel | Overall (%) |
| Success rate | |||||||
| Qwen 3.5 9B | 10.0 | 26.7 | 57.1 | 13.3 | 37.5 | 7.1 | 28.0 |
| Qwen 3.6 27B | 40.0 | 53.3 | 71.4 | 40.0 | 53.1 | 64.3 | 54.0 |
| Gemma4 26B A4B | 20.0 | 73.3 | 100.0 | 66.7 | 46.9 | 57.1 | 60.0 |
| Gemma4 31B | 30.0 | 80.0 | 100.0 | 66.7 | 71.9 | 85.7 | 74.0 |
| Failure mode | Evidence |
| Gemini Pro omits required artifacts. | Task 225, Peppermint Notion: The task required a Notion page summarizing a VPN ticket and naming its owner. All 50 actions remained in Peppermint; the agent never opened Notion or created the page. |
| Gemini Pro selects incorrect target entities. | Task 232, PipeCRM Thunderbird: The task required emailing Leon Green at [email protected] . The agent instead sent the email to [email protected] . The recipient rubric failed, although the weekend-question content passed. |
| Both Gemma models exhibit empty finalizations. | Task 229, PipeCRM: Gemma 4 26B A4B and Gemma 4 31B both executed click(’739’) followed by done() . The requested deal’s close date, February 21, 2025, and status, OPEN , were visible, but neither model sent an answer. |
| Qwen 3.6 misses required end-state actions. | Task 22, Acidwave: The agent created VOCALS_CONFIRMED and added “STAYIN’ AT HOME - VOCALS,” then called done() after seven actions. The required “FLYING AWAY (VOCALS)” was absent. |
| Qwen 3.6 selects incorrect entities or values. | Task 52, Firefly Thunderbird: The task required emailing [email protected] that Spotify Premium costs C9.99, clicked Send, and reported completion. |
| Failure pattern | Example website(s) | Illustrative evidence |
| Budget exhaustion | Across applications | Of Claude Opus 4.7’s 224 unsuccessful Test runs, 155 reached the 50-step limit. |
| Incomplete workflow / navigation failure | Nextcloud, Mattermost | An agent failed to open the budget file in Nextcloud and never navigated to Mattermost to message Jane. |
| Wrong answer or entity | Acidwave | An agent answered 00:00 for a folk-song duration task whose expected answer was 33:22 . |
| Filter, sort, or exact-constraint error | Metabase | An agent built a product-rating table but filtered Gadget instead of the required category Electronics . |
| Skipped verification | YT-Lite | An agent answered using a different Warhammer video without inspecting the requested Warhammer III video. |
| Premature termination | Peppermint | An agent declared a database-comment task infeasible without opening the target discussion. |
| Failure mode | Evidence |
| Incorrect absence claim. | Claude Code / Opus 5.5, Task 178, Notion: Asked to publish the “my summer vacation” page, the agent concluded that the page did not exist and stopped without publishing it. The other agents successfully published the requested page in their selected runs. |
| Local artifacts not fully delivered to the application. | Codex / GPT 6.1 Sol, Task 147, MyDrive: The agent generated all six requested algorithm files locally, but did not complete their creation in MyDrive. Its final answer acknowledged that Q1_javascript.js was still missing from the target folder. |
| Budget exhaustion before committing the result. | OpenCode / Kimi 3.1 Pro, Task 046, ERPNext: The agent gathered order and contact information from Metabase and PipeCRM and prepared a Sales Order. It reached 50 steps before saving; the terminal screenshot showed the order marked “Not Saved.” |
| Incorrect visual event counting. | OpenCode / Gemini 3.1 Pro, Task 317, YT-Lite: Asked how many times a rapper appeared holding a Sprite, the agent completed 12 frame-extraction calls but answered 11 , whereas the rubric required 9 . Media access succeeded, but the final count was incorrect under the grading criterion. |
| Judge Agreement Rate (%) | |||
| Agent Model | Qwen 3.6 27B | Gemini 3.1 Pro | Kimi 2.6 |
| Gemini 3.1 Pro | 96.9 | 92.3 | 94.6 |
| Gemini 3.1 Flash Lite | 87.7 | 91.5 | 90.8 |
| Claude Opus 4.7 | 93.8 | 92.2 | 92.2 |
| Gemma4 31B | 83.1 | 85.4 | 86.9 |
| Qwen 3.5 9B | 92.3 | 92.3 | 93.1 |
| Website | Construction basis / upstream repository | Application/upstream license | Compressed image size |
| acidwave | AI-assisted custom | MIT | 424 MB |
| erp-next | Open-source ERPNext/Frappe | GPL-3.0 (ERPNext); MIT (Frappe) | 627 MB |
| firefly | Open-source Firefly III | AGPL-3.0 | 347 MB |
| flight-search | AI-assisted edited from airline-route-data | MIT | 64.0 MB |
| immich | Open-source Immich | AGPL-3.0 | 7.03 GB |
| jobportal | AI-assisted edited from jobportal | MIT | 85.6 MB |
| Function | Claude Code Opus 5.5 | Codex GPT 6.1 Sol | OpenCode Gemini 3.1 Pro | OpenCode Qwen 3.8 Max Prime | OpenCode Kimi 3.1 Pro |
| Browser tools | |||||
| browser_snapshot | 296 | 1 036 | 423 | 312 | 300 |
| browser_take_screenshot | 76 | 89 | 8 | 30 | 12 |
| browser_find | 1 307 | 809 | 357 | 273 | 305 |
| browser_navigate | 1 753 | 954 | 1 099 | 1 466 | 1 153 |
| browser_navigate_back | 7 | 18 | 53 | 13 | 40 |
Web Agent Benchmarks
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.
See all 72 papers