StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability

Haoyue Bai; Dong Wang; Long Chen; Bingguang Hao; Pengyang Shao; Yonghui Yang; Yicheng He; Chenyi Zhuang

cs.SEMar 27, 2026

StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability

Authors: Haoyue Bai, Dong Wang, Long Chen, Bingguang Hao, Pengyang Shao, Yonghui Yang, Yicheng He, Chenyi Zhuang

Organizations: Inclusion AI, Ant Group · National University of Singapore · University of Illinois Urbana-Champaign

Abstract

Large language model-based web agents have demonstrated strong performance on realistic web interaction tasks. However, existing evaluations are predominantly conducted under relatively stable and well-behaved interaction conditions, which may overestimate agent robustness. High task success in such idealized settings does not necessarily reflect performance under realistic web interaction. To address this limitation, we introduce a diagnostic stress-testing benchmark for web agents. We first construct realistic and controllable web environments that provide clean and stable interaction workflows as reference baselines. We then introduce structured and controlled perturbations that emulate interaction variability, including shifting layouts, altered interaction semantics, and execution disruptions. By comparing agent behavior between clean and perturbed settings, our framework enables systematic diagnosis of robustness under what-if interaction scenarios. Through extensive evaluation of state-of-the-art multimodal web agents, we show that stress-based evaluation exposes failure modes and substantial robustness gaps that remain hidden under clean benchmark conditions.

Figures & tables

Figure 1: Perturbations Across the Interaction Pipeline

Figure 2: Overview of the StressWeb benchmark. Clean reference environments provide stable interaction workflows as reference. The benchmark introduces six perturbation types spanning perception (layout chaos, DOM noise), action selection (action semantic remap with or without explicit cues), and execution (pop-up interruptions, action failures) to systematically evaluate web agent robustness under controlled interaction variability.

Model	Clean		Chaos		Noise		Failure		Pop-Up		RemapE		Remap		Avg
Model	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$	ckpt% $\uparrow$	steps $\downarrow$
GLM-4.6V	46.7	61.0	38.3	59.2	42.1	61.6	40.1	59.3	38.0	66.6	20.8	89.1	20.5	88.4	35.2	69.3
Qwen3-VL-235B	52.7	60.0	44.9	61.2	49.2	58.8	43.4	62.3	40.9	68.5	26.5	84.8	24.8	92.0	40.3	69.7
Claude Haiku 4.5	43.0	79.8	38.5	77.3	40.9	83.1	39.7	83.1	38.5	85.8	26.0	95.0	24.1	97.7	34.6	87.0
Claude Opus 4.5	55.0	66.2	41.6	71.1	51.5	67.0	47.7	71.5	42.3	80.5	39.0	85.7	28.9	94.0	43.7	76.6
Claude Sonnet 4.5	57.5	64.4	45.7	74.6	52.4	67.9	47.5	72.7	45.2	74.6	31.6	89.8	25.4	90.1	43.6	76.3

Table 1: Overall Performance Across Different Environments. We report checkpoint pass rate (ckpt%) and average interaction steps (steps).

Figure 3: Checkpoint pass rates across different perturbation modes. Lines represent individual models and the dashed horizontal lines indicate the average performance for each mode across all models. Perception perturbations (Noise and Chaos) introduce relatively mild degradation, while execution perturbations (Failure and Pop-Up) lead to moderate drops. Semantic perturbations (RemapE and Remap), which alter interaction rules, cause the most severe performance decline.

Panel of Figure 4 — Figure 4: Performance and cost robustness under environmental perturbations. (a) Performance retention relative to the clean environment. (b) Additional interaction steps compared to the clean setting. Darker colors indicate stronger degradation or higher cost.

Figure 5: Claimed vs. actual successful queries under different modes. Environmental perturbations increase the discrepancy between perceived and actual task success, indicating reduced self-assessment reliability.

Benchmark	Actionable	Realistic	Deterministic	Diagnostic	Interventional
World of Bits	✓	✗	✓	✗	✗
WebShop	✓	✗	✓	✗	✗
Mind2Web	✓	✓	✗	✗	✗
WebArena	✓	✓	✓	✗	✗
WebVoyager	✓	✓	✗	✗	✗
BrowserArena	✓	✓	✗	✓	✗

Table 2: Comparison of representative web agent benchmarks and evaluation frameworks. A capability is marked as supported only if it is explicitly designed and systematically supported. Actionable indicates whether the benchmark requires agents to perform structured UI operations such as clicking, typing, and multi-step task workflows, rather than only browsing or reading web pages. Realistic denotes whether the environment preserves the fidelity of real-world web interfaces. Deterministic indicates whether task execution is reproducible across runs. Diagnostic denotes built-in support for attributing agent failures to specific components. Interventional indicates whether the benchmark supports controlled and reproducible environment perturbations for robustness evaluation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example screenshots of the generated web environments used in StressWeb. The benchmark includes ten realistic website categories covering diverse domains such as e-commerce, scheduling, email management, transportation, and cloud file systems. Each environment supports multi-step interactions, persistent state updates, and common web operations including search, filtering, navigation, and form submission.

Panel of Figure 7 — Figure 7: Radar plots showing model performance across different websites. Each radar plot contains two curves: the performance of the specific model and the average performance across all models.

Panel of Figure 8 — Figure 8: Radar plots showing model performance across different perturbation modes. Each plot compares the performance of an individual model against the average performance across all evaluated models.

Panel of Figure 9 — Figure 9: Model-level self-assessment reliability across different evaluation conditions. Each plot compares the number of queries claimed as successful by the agent with the number of queries that are actually successful. The gap between the two quantities reflects the degree of self-assessment error.

Panel of Figure 10 — Figure 10: Model performance under clean and perturbed environments. (a) Checkpoint pass rates across different perturbation modes. (b) Model ranking under the clean environment. (c) Average model performance across all perturbed environments. Colors are consistent across (b) and (c), allowing ranking differences between clean and perturbed conditions to be visually identified.

Setting	Exact Repeat (%)	Total Repeats	Max Repeat
Clean	58.4	1,599	54
Chaos	72.5	1,659	44
Failure	83.2	2,462	28
Perturbed	69.1	1,442	32
DOM	61.7	1,920	64
RemapE	96.6	4,588	65

Appendix

Table 3: Repetitive action behavior under different perturbation settings. Exact Repeat (%) denotes the percentage of trajectories containing consecutive identical actions (same action type and parameters). Semantic perturbations lead to significantly higher repetition frequency and longer repeated sequences.

Web Agent Benchmarks

Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.

See all 64 papers

WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents
Sep 28, 2026Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2Web Agent Benchmarks GUI Agents
DAIMLD · HSE University · NUST MISIS
CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
Aug 9, 2026Zejun Xu, Taiyi Chen, Jin Li +13Computer-Use Agents Web Agent Benchmarks
Macau University of Science and Technology · FellouAI · Tsinghua University +3
Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
Apr 8, 2026Jiwan Chung, JiHyuk Byun, Vibhav Vineet +1Process-Based Agent Evaluation Agent Evaluation
Yonsei University · Microsoft Research

Sep 28, 2026J/K move · Enter open · S save

WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

Anton Emelyanov, Maria Tikhonova, Zaven Martirosian, Sergei Averkiev+1

DAIMLD · HSE University · NUST MISIS

Web Agent Benchmarks GUI Agents Web Agents

Open paper AskPDF arXiv

We present WebPageBench, an open framework for evaluating web agents in which every task is verified from the interface's own event log. Six instrumented mock sites with brand identifiers removed (a marketplace, a bookstore, a grocery service, rail ticketing, hotel search and a document cabinet) emit typed events with parameters as a user or an agent acts. A task declares the events it requires, and success is decided by matching them, with no judge model and no scraping of rendered pages. The same instrumentation supports controlled UI variation: one configuration switch re-renders a task through a different implementation of a single control while the prompt and the success conditions stay completely identical, so sensitivity to interface form can be measured under a fixed task specification. The WebPageBench release consists of three components: 152 tasks, divided into 65 canonical scenarios and 87 control variants across light/dark UI-modes; a common runner evaluated with six browser/DOM harness configurations and five screenshot-only GUI-agent families; and a public leaderboard of 24 model-harness pairs. On the public 152-task leaderboard the gap between what agents declare finished and what the log confirms reaches 41 points (one configuration declares every task finished and satisfies the conditions on 59%).

Abstract

Web Agent Benchmarks

Explore similar work

WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking