Web agents are increasingly deployed to complete tasks autonomously on real websites, where adversarially injected content can induce them to perform unauthorized actions. Existing evaluations primarily use fixed attacks or focus on a single attack type, and evaluate attacks at the text level rather than through real state changes, thus underestimating how vulnerable agents are to diverse attacks. To address the gap, we introduce REEF, a benchmark of red-teaming executable environments with faithful verification for measuring the security vulnerabilities of web agents. REEF covers 9 verifier-grounded websites spanning travel, commerce, productivity, media, and communication, and pairs every authentic benign task with a malicious objective delivered through 5 attack types at both the user and the environment level. REEF's state-based verifiers provide the execution feedback that drives ANGLER, an adaptive near-miss-guided lure evolution framework for red-teaming. Rather than evaluating a fixed attack set, ANGLER uses outcomes from prior rollouts to guide LLM-based attack mutation and candidate selection, prioritizing near-miss seeds during refinement. Across 8 VLM-based web agents, ANGLER achieves an end-to-end attack success rate (ASR) of 20.36%, compared with 15.24% for the non-adaptive setting. Compared with attack-free execution on the same benign tasks, attacks generated by ANGLER reduce benign task success rate (BSR) from 61.9% to 44.4%.
Figures & tables
Work
#Web
User
Env.
Attack Types
Online Generation
Adaptive [-0.1ex] Search
Cross-rollout [-0.1ex] Learning
Failure-guided [-0.1ex] Refinement
Plan Validation
Execution Oracle
VWA-Adv ( Wu et al., 2025 )
3
×
✓
Text/Image
×
×
×
×
–
✓
WASP ( Evtimov et al., 2025 )
2
×
✓
IPI
×
×
×
×
–
✓
RedTeamCUA ( Liao et al., 2026 )
3
×
✓
IPI
×
×
×
×
–
✓
SecureWebArena ( Ying et al., 2026 )
6
✓
✓
6 Types
×
×
×
×
–
×
AgentVigil ( Wang et al., 2025c )
3
×
✓
IPI
✓
✓
✓
×
×
✓
MUZZLE ( Syros et al., 2026 )
4
×
✓
IPI
✓
✓
✓
✓
×
×
Table 1. Comparison of representative security benchmarks and adaptive red-teaming systems for web and computer-use agents. #Web denotes the number of web environments. User and Env. denote user-level and environment-level attacks respectively.
Figure 1. Overview of REEF. The left panel summarizes 9 web environments spanning 5 application areas and 8,316 executable cases. The center panel illustrates the compositional construction of benign tasks and attacks from structured templates and parameters. The right panel presents the state-based user, attacker, and recovery verifiers used to programmatically determine rollout outcomes.
Figure 2. Overview of ANGLER. A two-stage scheduler balances structured coverage and adaptive search, while verifier feedback guides candidate generation and contextual-bandit selection across subsequent rollouts.
Seed category
Definition
Success
The malicious objective remains satisfied at termination.
Near-miss, benign incomplete
The malicious objective is achieved transiently but not at termination, while the benign task remains incomplete.
Near-miss, benign complete
The malicious objective is achieved transiently but not at termination, while the benign task is completed.
Blocked
The malicious objective is never achieved.
Table 2. Experience-record categories for mutation-seed selection based on attack success and benign-task completion.
Website
GPT-4o
GPT-6.1 Sol
Claude Sonnet 4.5
Claude Sonnet 5.5
NA
ANGLER
CTA
NA
ANGLER
CTA
NA
ANGLER
CTA
NA
ANGLER
CTA
Coursera
22.0
58.0
84.6
7.0
10.0
10.3
22.0
17.0
70.7
17.0
25.0
0.0
Facebook
28.0
59.0
81.6
2.0
9.0
0.0
18.0
31.0
82.1
15.0
20.0
0.0
GitHub
20.0
42.0
66.7
5.0
12.0
0.0
22.0
20.0
73.4
5.0
13.0
6.7
Headspace
24.0
31.0
81.9
10.0
4.0
0.0
12.0
10.0
53.2
3.0
10.0
0.0
JD
41.0
54.0
74.7
20.0
14.0
0.0
16.0
28.0
62.7
15.0
24.0
0.0
Table 3. Matched site-level ASR (%) of the non-adaptive baseline (NA) and ANGLER for agents instantiated with 8 VLM backbones. CTA reports the engagement rate of ANGLER. Avg. denotes the unweighted mean across websites.
Figure 3. Cumulative number of verifier-confirmed successful attack rollouts as the rollout budget increases, aggregated across 8 VLM backbones and 9 websites.
Backbone Model
AF
NA
ANGLER
GPT-4o
62.2
35.0
28.9
GPT-6.1 Sol
78.4
62.2
61.1
Claude Sonnet 4.5
59.7
44.9
46.8
Claude Sonnet 5.5
69.8
56.1
52.3
Qwen3.7-Flash
43.6
31.4
27.7
Qwen3-VL
67.3
50.2
51.9
Table 4. Benign-task success rate (BSR, %) under attack-free (AF), non-adaptive (NA), and ANGLER conditions.
Figure 4. Score convergence of ANGLER across 8 VLM backbones. Each curve reports the mean of site-level rates across 9 websites.
Metric
Run 1
Run 2
Run 3
Mean ± SD
ASR
14.6
12.1
11.0
12.6 ± 1.8
CTA
41.1
44.1
41.3
42.2 ± 1.7
BSR
51.9
49.7
52.9
51.5 ± 1.6
Table 5. Run-to-run repeatability of ANGLER using Qwen3-VL across 9 websites. Values are percentages and macro-averaged across websites. The final column reports the mean and standard deviation over three independent runs.
Method / Variant
BSR
CTA
ASR
w/o Bandit Selection
50.0
39.4
10.3
w/o Experience Feedback
53.8
40.7
11.3
w/o Online Mutation
51.6
41.2
12.0
Full ANGLER
51.9
41.1
14.6
Table 6. Ablation of the adaptive components of ANGLER using Qwen3-VL across 9 websites.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Representative initial-phase injection surfaces for the four environment-level attack types in REEF. The attacks differ in how adversarial content is presented through the webpage environment. A two-by-two grid showing examples of indirect prompt injection, pop-up injection, distraction injection, and advertising injection.
Figure 6. Distribution of the 326 instantiated malicious objectives in REEF. The inner ring shows five broad task categories, while the outer ring shows their fine-grained categories.
Website
Short
Medium
Long
Total
Coursera
15
7
5
27
Facebook
16
15
0
31
GitHub
15
10
10
35
Headspace
20
20
5
45
JD.com
9
22
8
39
Skyscanner
5
0
36
41
Appendix
Table 7. Counts of instantiated malicious objectives by estimated minimum action horizon. Short, medium, and long denote ≤6 , 7 – 9 , and ≥10 actions, respectively.
Action
Description
click [id]
Click the element identified by id .
clear [id]
Clear the contents of a text-input element.
type [id] [text] [0|1]
Enter text into an element, optionally appending a newline.
hover [id]
Move the pointer over an element.
press [key_comb]
Issue a keyboard key or key combination.
scroll [up|down]
Scroll the current page in the specified direction.
Appendix
Table 8. ID-based browser actions available to the target web agents. Element IDs refer to the agent’s current observation.
Seed Category
Low-evidence
Refinement
Regime
Regime
Success
5
2
Near-miss, benign incomplete
4
5
Near-miss, benign complete
3
4
Blocked
2
3
Appendix
Table 9. Relative weights for mutation-seed sampling under the low-evidence regime and refinement regime.
Field group
Allowed output
Slot binding
slot_id , which must identify one assigned and previously unused objective slot.
Initial payload
payload_override , which revises the initial injected instruction or surface copy.
Delivery
delivery_param_overrides and delivery_parameter_id , restricted to catalog-supported delivery fields.
Display timing
show_after_agent_steps , restricted to an integer from 1 to 3.
Surface style
surface_style_overrides , restricted to environment-level attacks and whitelisted color, font-size, and font-family values.
Continuation
phase2_payload_override and the corresponding continuation-phase delivery and style fields.
Appendix
Table 10. Editable fields in the mutation-output schema.
Model
Direct PI
Indirect PI
Pop-up Injection
Distraction Injection
Ad Injection
GPT-4o
56.6/–
38.9/46.3
42.3/83.3
39.3/89.3
29.3/56.3
GPT-6.1 Sol
32.5/–
0.0/0.0
0.5/5.4
0.0/2.5
0.0/3.9
Claude Sonnet 4.5
22.5/–
9.8/29.4
14.0/75.8
24.5/80.6
8.3/58.0
Claude Sonnet 5.5
39.8/–
2.2/0.0
2.3/2.3
0.0/1.9
0.0/1.8
Qwen3.7-Flash
23.6/–
6.3/14.6
18.6/69.7
13.2/42.6
14.1/34.8
Qwen3-VL
35.5/–
0.0/0.0
8.0/69.2
9.1/40.1
1.7/15.0
Appendix
Table 11. Attack-type performance of ANGLER by target backbone model. Each cell reports ASR/CTA (%).
Website
GPT-4o
GPT-6.1 Sol
Claude Sonnet 4.5
Claude Sonnet 5.5
Coursera
51.6
557.3
123.6
85.1
Facebook
102.0
72.5
148.0
156.6
GitHub
97.0
100.6
110.1
149.4
Headspace
95.4
89.3
130.3
124.9
JD
126.3
100.8
158.5
130.6
Skyscanner
98.4
135.4
95.8
102.9
Appendix
Table 12. Average wall-clock execution time per completed valid rollout across websites and victim backbones. Values are reported in seconds.
Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actions. Existing visual red-teaming approaches use adversarial visual content to manipulate this process. However, they primarily target model inference and do not explicitly account for structured input processing or action post-processing. Consequently, model-level success does not establish control over browser execution and cannot reliably characterize end-to-end agent robustness. To address this gap, we formulate red teaming for vision-grounded web agents as an end-to-end grounding-to-execution problem, and introduce WebMirage, a framework that crafts localized visual perturbations that cause agents to select attacker-controlled content and execute the corresponding browser action across varying webpage renderings. It uses a role-slot abstraction and webpage recomposition to capture competition among webpage elements, and dataflow analysis to align optimization with action post-processing. We evaluate WebMirage across four agent configurations and six VLM backbones on 2,250 tasks covering 13 public websites and a sandbox benchmark. WebMirage achieves an average attack success rate of 91.9%, compared with 17.4% for the strongest baseline, and remains effective against three agent-level defenses.
Large language model (LLM) agents combine language-based reasoning with external tools to perform complex tasks. Adversarial inputs can exploit interactions between the agent and its environment, causing the agent to violate safety policies during execution. Yet existing evaluations often reduce agent safety to a single attack success rate (ASR), collapsing exposure, execution, observation, and adjudication and potentially conflating actual violations with evidence visibility. We introduce REDAgentBench, an executable framework for autonomous red-teaming and faithful measurement. It derives attacks from explicit safety constraints and associated agent-system vulnerabilities, runs them in isolated service sandboxes, and verifies harmful effects from service receipts and final-state changes. The benchmark contains 1,661 cases across five service surfaces. Across six models and three agent harnesses, macro-average ASR is 65.69%; reported ASR varies with harness and evidence view, while evaluation-context disclosure changes execution behavior. In a state-grounded diagnostic cohort, almost one in five confirmed violations with resolved action anchors occurs after the agent states the relevant constraint or risk, revealing a Recognition--Execution Gap. Finally, a training-free policy reminder reduces confirmed violations by more than 70 percentage points in matched replay. These findings show that executable evaluation can improve safety measurement and identify actionable intervention points.
Zixing Chen, Xingyuan Liu, Jie Zhu +6
Fudan University · The Hong Kong University of Science and Technology · Qwen DianJin Team, Alibaba Cloud Computing +1
Reusable Web-agent verification requires a rule to follow changing requirements and support subsequent execution. We introduce AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings. The resource spans 97 tasks, 79 families, and four Web services. Experiments with DeepSeek Flash/Pro establish two linked measurements. Across 48 families, complete-panel and decision-replay tests retain 82/96 Flash and 79/96 Pro source programs; paired reuse-rebuild BA intervals are [-5.47, 1.57] and [-10.16, 0.00] percentage points. Across 22 execution families, deployment-state panel correctness separates program assignments whose two saved openings are both delivered correctly: 32/33 assignments with error-free panels, versus 1/11 with panel errors. All reference-defined program-gate regressions and withheld deliveries occur in the latter group. Neutral continuation attains the highest correct delivery under the tested generic feedback, providing a concrete baseline for measuring the value added by a gate. Released instructions, states, programs, reference rules, and trajectories connect each test failure to its execution consequences. AgentProofBench makes reusable verification assessable at both the requirement and policy levels.