Web Agent Benchmarks

Follow this topic Ask about this topic

Web agent benchmarks measure whether AI systems can find information and complete tasks through browser interfaces, from web research and form filling to shopping and website development. They use simulated websites for repeatable tests, live sites for realistic workflows, and controlled changes to interfaces or instructions to probe robustness. Evaluation combines task success with measures such as evidence quality, interaction traces, efficiency, and human or automated judgments. A central challenge is balancing realistic, varied settings with reproducible scoring; agents still often struggle with long tasks, changing interfaces, and verifying that their actions produced the intended result.

Latest papers

64 papers12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Do Web Agents Investigate Before They Decide?
Feb 5, 2026Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton +2Web Agent Benchmarks Hallucination in Language Models
Department of Robotics and Mechatronics Engineering, University of Dhaka · Department of Computer Science and Engineering, University of Dhaka
It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents
Dec 29, 2025Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki +7Web Agent Benchmarks Indirect Prompt Injection
University of Oxford · 2SoftServe · 3Independent +1
WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents
Oct 1, 2025Yinuo Liu, Xilong Wang, Ruohan Xu +2Web Agent Benchmarks Prompt Injection Attacks
Duke University
WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks
Jul 1, 2025Zihao Sun, Zijing Shi, Ling ChenWeb Agent Benchmarks Multimodal Agents
University of Technology Sydney, Australia

Feb 5, 2026J/K move · Enter open · S save

Do Web Agents Investigate Before They Decide?

Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton, Shahrear Bin Amin+1

Department of Robotics and Mechatronics Engineering, University of Dhaka · Department of Computer Science and Engineering, University of Dhaka

Web Agent Benchmarks Hallucination in Language Models LLM Agent Evaluation Web Agents

Open paper AskPDF arXiv

Autonomous web agents are increasingly deployed in moderation and policy enforcement, where correct decisions often depend on evidence that is not immediately visible and must be actively investigated. Yet existing benchmarks largely assume task critical information is immediately accessible. They do not measure investigative competence: recognizing when visible context is insufficient, retrieving hidden evidence, and integrating it into a final decision. We introduce MIRAGE, a benchmark of 750 multi step decision tasks across three domains: Wikipedia Forensics, Shopping Admin adjudication, and Reddit Moderation. Each task has two layers: a visible surface context that often points to the wrong action, and a hidden context, reachable only by active investigation, that contains the decisive evidence. We decompose agent performance into Investigation, Reasoning, and Decision Accuracy, complemented by an Investigative Hallucination Rate. We evaluate eight LLM agents across two model generations. Three patterns emerge. First, agents reach relevant pages but rarely extract the decisive evidence on them. Second, procedural hints improve investigation but do not consistently improve decisions on Wikipedia tasks, where decisive evidence often contradicts surface impressions. Third, 12.6% of trajectories cite fabricated facts. We call these the Navigation Discovery Gap, Collapse under Contradiction, and Investigative Hallucination. These patterns persist across model scale, generation, and reasoning architecture.

Latest papers

Do Web Agents Investigate Before They Decide?

It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents

WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks