Large language model-based web agents have demonstrated strong performance on realistic web interaction tasks. However, existing evaluations are predominantly conducted under relatively stable and well-behaved interaction conditions, which may overestimate agent robustness. High task success in such idealized settings does not necessarily reflect performance under realistic web interaction. To address this limitation, we introduce a diagnostic stress-testing benchmark for web agents. We first construct realistic and controllable web environments that provide clean and stable interaction workflows as reference baselines. We then introduce structured and controlled perturbations that emulate interaction variability, including shifting layouts, altered interaction semantics, and execution disruptions. By comparing agent behavior between clean and perturbed settings, our framework enables systematic diagnosis of robustness under what-if interaction scenarios. Through extensive evaluation of state-of-the-art multimodal web agents, we show that stress-based evaluation exposes failure modes and substantial robustness gaps that remain hidden under clean benchmark conditions.
Figures & tables
Figure 1: Perturbations Across the Interaction Pipeline
Figure 2: Overview of the StressWeb benchmark. Clean reference environments provide stable interaction workflows as reference. The benchmark introduces six perturbation types spanning perception (layout chaos, DOM noise), action selection (action semantic remap with or without explicit cues), and execution (pop-up interruptions, action failures) to systematically evaluate web agent robustness under controlled interaction variability.
Model
Clean
Chaos
Noise
Failure
Pop-Up
RemapE
Remap
Avg
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
ckpt% ↑
steps ↓
GLM-4.6V
46.7
61.0
38.3
59.2
42.1
61.6
40.1
59.3
38.0
66.6
20.8
89.1
20.5
88.4
35.2
69.3
Qwen3-VL-235B
52.7
60.0
44.9
61.2
49.2
58.8
43.4
62.3
40.9
68.5
26.5
84.8
24.8
92.0
40.3
69.7
Claude Haiku 4.5
43.0
79.8
38.5
77.3
40.9
83.1
39.7
83.1
38.5
85.8
26.0
95.0
24.1
97.7
34.6
87.0
Claude Opus 4.5
55.0
66.2
41.6
71.1
51.5
67.0
47.7
71.5
42.3
80.5
39.0
85.7
28.9
94.0
43.7
76.6
Claude Sonnet 4.5
57.5
64.4
45.7
74.6
52.4
67.9
47.5
72.7
45.2
74.6
31.6
89.8
25.4
90.1
43.6
76.3
Table 1: Overall Performance Across Different Environments. We report checkpoint pass rate (ckpt%) and average interaction steps (steps).
Figure 3: Checkpoint pass rates across different perturbation modes. Lines represent individual models and the dashed horizontal lines indicate the average performance for each mode across all models. Perception perturbations (Noise and Chaos) introduce relatively mild degradation, while execution perturbations (Failure and Pop-Up) lead to moderate drops. Semantic perturbations (RemapE and Remap), which alter interaction rules, cause the most severe performance decline.
Figure 4: Performance and cost robustness under environmental perturbations. (a) Performance retention relative to the clean environment. (b) Additional interaction steps compared to the clean setting. Darker colors indicate stronger degradation or higher cost.
Figure 5: Claimed vs. actual successful queries under different modes. Environmental perturbations increase the discrepancy between perceived and actual task success, indicating reduced self-assessment reliability.
Benchmark
Actionable
Realistic
Deterministic
Diagnostic
Interventional
World of Bits
✓
✗
✓
✗
✗
WebShop
✓
✗
✓
✗
✗
Mind2Web
✓
✓
✗
✗
✗
WebArena
✓
✓
✓
✗
✗
WebVoyager
✓
✓
✗
✗
✗
BrowserArena
✓
✓
✗
✓
✗
Table 2: Comparison of representative web agent benchmarks and evaluation frameworks. A capability is marked as supported only if it is explicitly designed and systematically supported. Actionable indicates whether the benchmark requires agents to perform structured UI operations such as clicking, typing, and multi-step task workflows, rather than only browsing or reading web pages. Realistic denotes whether the environment preserves the fidelity of real-world web interfaces. Deterministic indicates whether task execution is reproducible across runs. Diagnostic denotes built-in support for attributing agent failures to specific components. Interventional indicates whether the benchmark supports controlled and reproducible environment perturbations for robustness evaluation.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Example screenshots of the generated web environments used in StressWeb. The benchmark includes ten realistic website categories covering diverse domains such as e-commerce, scheduling, email management, transportation, and cloud file systems. Each environment supports multi-step interactions, persistent state updates, and common web operations including search, filtering, navigation, and form submission.
Figure 7: Radar plots showing model performance across different websites. Each radar plot contains two curves: the performance of the specific model and the average performance across all models.
Figure 8: Radar plots showing model performance across different perturbation modes. Each plot compares the performance of an individual model against the average performance across all evaluated models.
Figure 9: Model-level self-assessment reliability across different evaluation conditions. Each plot compares the number of queries claimed as successful by the agent with the number of queries that are actually successful. The gap between the two quantities reflects the degree of self-assessment error.
Figure 10: Model performance under clean and perturbed environments. (a) Checkpoint pass rates across different perturbation modes. (b) Model ranking under the clean environment. (c) Average model performance across all perturbed environments. Colors are consistent across (b) and (c), allowing ranking differences between clean and perturbed conditions to be visually identified.
Setting
Exact Repeat (%)
Total Repeats
Max Repeat
Clean
58.4
1,599
54
Chaos
72.5
1,659
44
Failure
83.2
2,462
28
Perturbed
69.1
1,442
32
DOM
61.7
1,920
64
RemapE
96.6
4,588
65
Appendix
Table 3: Repetitive action behavior under different perturbation settings. Exact Repeat (%) denotes the percentage of trajectories containing consecutive identical actions (same action type and parameters). Semantic perturbations lead to significantly higher repetition frequency and longer repeated sequences.
Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.
We present WebPageBench, an open framework for evaluating web agents in which every task is verified from the interface's own event log. Six instrumented mock sites with brand identifiers removed (a marketplace, a bookstore, a grocery service, rail ticketing, hotel search and a document cabinet) emit typed events with parameters as a user or an agent acts. A task declares the events it requires, and success is decided by matching them, with no judge model and no scraping of rendered pages. The same instrumentation supports controlled UI variation: one configuration switch re-renders a task through a different implementation of a single control while the prompt and the success conditions stay completely identical, so sensitivity to interface form can be measured under a fixed task specification. The WebPageBench release consists of three components: 152 tasks, divided into 65 canonical scenarios and 87 control variants across light/dark UI-modes; a common runner evaluated with six browser/DOM harness configurations and five screenshot-only GUI-agent families; and a public leaderboard of 24 model-harness pairs. On the public 152-task leaderboard the gap between what agents declare finished and what the log confirms reaches 41 points (one configuration declares every task finished and satisfies the conditions on 59%).
Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2
Large language models are increasingly deployed as autonomous agents that interact with the web through browsers. While recent progress has been driven by benchmarks that evaluate end-to-end task success, these evaluations largely overlook two fundamental sources of difficulty in real web browsing: complex actions over rich user interfaces and visual perception of dynamically rendered content, especially in workflows that span multiple websites. We introduce CAP, a scalable benchmark for evaluating browser agents on cross-site, human-like web tasks that require non-trivial UI interactions and visual understanding. Specifically, we adopt a decomposition-and-recomposition pipeline that first abstracts each website into a structured site card capturing user-facing functions, complex execution operations, and perceptual requirements, and then recomposes these components into realistic cross-site workflows. Each task is therefore grounded in multiple specific operations on each website, enabling fine-grained diagnosis. Built on this framework, we construct 420 tasks across 108 real-world websites and 24 domains under careful quality control. Experiments on state-of-the-art browser agents using our verifiable agent-as-a-judge evaluation framework show low success rates and reveal that perception-heavy interactions remain a major bottleneck, exposing substantial gaps between current agents and real-world web browsing demands.
Zejun Xu, Taiyi Chen, Jin Li +13
Macau University of Science and Technology · FellouAI · Tsinghua University +3
Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep