cs.CLApr 9, 2026

ClawBench: Can AI Agents Complete Everyday Online Tasks?

Authors: Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, +20 more

Organizations: University of British Columbia · Vector Institute · Etude AI · University of Waterloo · Carnegie Mellon University · SJTU · University of Manchester · Fudan University · Lehigh University · Tsinghua University · UniPat AI · Nanjing University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI agents may be able to assist with emails and documents, but can they reliably complete everyday online workflows on real websites? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework comprising 153 everyday online tasks that people need to accomplish regularly in their lives and work, spanning 144 platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and interaction challenges of real-world web environments. An interception layer captures and blocks the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 8 frontier models show that both proprietary and open-source models complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%, which exposes gaps in current AI agents. Progress on ClawBench brings us closer to AI agents that can function as general-purpose assistants.

Figures & tables

Figure 1: ClawBench overview. Left : ClawBench contains 153 realistic online tasks spanning 15 everyday life categories and 144 live web platforms. Middle : unlike prior web-agent benchmarks that rely on offline sandbox environments with static pages and limited read-only interactions, ClawBench evaluates agents on live production websites involving dynamic, multi-step, and write-capable interactions, while preventing unintended real-world side effects and supporting fine-grained, step-level traceable evaluation. Right : despite strong performance on existing benchmarks, frontier models still achieve substantially lower success rates on ClawBench , highlighting the large gap between controlled benchmark performance and real-world online task completion.
Figure 2
Figure 4: Agent-as-Judge inference pipeline. The evaluator compares the agent trace against the human reference trajectory across five evidence layers and returns a binary pass/fail verdict with a structured justification.
ModelAgent SRHuman SRAgreement
Sonnet 4.633.33%33.33\%34.64%34.64\%93.46%93.46\%
GPT-5.46.54%6.54\%8.50%8.50\%84.97%84.97\%
Table 2: Agent-as-Judge vs. human judge on ClawBench . Agent SR and Human SR are pass rates from the Agent-as-Judge and human reviewer respectively over the same 153153 tasks; Agreement is the raw fraction of matching verdicts.
Table 3: Main results on ClawBench . Success rate (%) of 8 models on ClawBench , reported overall and for each of the 8 high-level task categories. Avg. Cost denotes the average API cost per task, and Avg. Token denotes the average token consumption per task. Bold marks the best result; underline marks the second best.
Figure 5: Operating-cost efficiency on ClawBench . Overall success rate plotted against three operating-cost axes: (a) average API cost per task in USD (log scale), (b) average tokens consumed per task in millions, and (c) average tool calls per task. The shaded region in each panel marks an efficient operating regime (SR ≥15%\geq 15\% at low cost on that axis). The bold-outlined dot highlights the most efficient capable model per panel.
Figure 6: Tool-call counts by outcome for Claude Sonnet 4.6 and Gemini 3 Flash. Dashed lines mark medians; failed trajectories make more calls and have longer tails, indicating unproductive rather than insufficient exploration.
Figure 7: Two views of last-mile hesitation. (a) Tool-use composition by model and outcome; runs with zero tool calls are excluded from this view and are analyzed separately. (b) Deepest workflow stage over the 1,224 ClawBench runs. Failures often interact substantially yet stop before the terminal step.
Figure 8: Termination reason by model. Counts of trajectory terminations across agent_idle , time_limit_exceeded , agent_exited , and unknown for the eight frontier agents.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Category leadership rotates across the eight-model panel. Per-column rank of each model on overall success rate and the eight high-level task categories. Marker size scales with success rate; gold-outlined markers mark the rank-1 model in each column.
Figure 10: Per-task solve-rate distribution across the 8-model panel. For each of the 153 tasks, the histogram bins the fraction k/8k/8 of evaluated models that pass the task.
Figure 11: Pass-vs-fail distributions across four operating-cost metrics, per model. Horizontal boxplots split by outcome over browser actions, tool/message count, wall-clock duration, and HTTP request count.
Figure 13
Figure 14
ModelProviderMax tokens
Sonnet 4.6Anthropic8192
Haiku 4.5Anthropic8192
GPT-5.4OpenAI8192
Gem 3.1 ProGoogle8192
Gem 3.1 FLGoogle8192
Gem 3 FlashGoogle8192
Appendix
Table 7: Per-model API provider and output token cap. top_p is left at the API default ( 1.01.0 ); reasoning-enabled models (Sonnet 4.6, GPT-5.4) use the provider’s high reasoning-effort setting, while the remaining models use greedy decoding at the provider default.
MetricHumanAgentGap
Key interval191–271 ms3–16 ms1515 – 90×90\times
<< 20 ms keys∼\sim 0.010.83–1.00unnatural
MousePresentNonemissing
Click interval∼\sim 6.1 s∼\sim 16.4 s2.7×2.7\times slower
ScrollFineCoarseunnatural
Appendix
Table 8: Human vs. agent interaction dynamics.

Explore similar work

CardsList

  1. StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

    Aug 18, 2026Liya Zhu, Xin Ma, Tao Liu +35AI Agent BenchmarksLLM Agent Evaluation

    ByteDance Seed · Nanjing University · M-A-P

  2. RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

    Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Computer-Use Agent BenchmarksAI Coding Agents

    1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*

  3. Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows

    Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent EvaluationAI Agent Benchmarks

    1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4