ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud
Organizations: Deakin University · Fudan University · The University of Melbourne · Amazon ·
Abstract
Web agents increasingly browse, shop, and manage accounts on behalf of their users. Prior work has shown that they can be misled by adversarial content, including prompt injections and deceptive interfaces. Beyond such attacks, agents operating on the open web also encounter online fraud designed to deceive humans, including fraudulent online stores, fake investment platforms, and advance-fee schemes. How web agents respond to such fraud during task execution remains underexplored, in part because scam websites are short-lived, making systematic evaluation difficult. To address this gap, we introduce ScamWorld, an interactive environment that reconstructs reported scam websites across diverse scenarios, fraud mechanisms, and languages. Each reconstructed website reproduces documented fraud mechanisms, making relevant signs of fraud observable before the agent's final action and enabling controlled, repeatable evaluation. Through an extensive study of web agents powered by six frontier LLMs, we find that baseline deception rates range from 11% to 35% across models. Agents proceed more readily with payments than with requests for credentials or wallet information. A closer analysis of agent trajectories shows that some agents acknowledge suspicious conditions yet proceed, at times treating the user's instructions as authorisation to accept additional terms imposed by the website. ScamWorld provides a reusable testbed for systematically evaluating and stress-testing web agents against real-world online fraud.
Figures & tables
| Work | Real Scam Website HTML | Executable Web Environment | Multi- modal | Multi- lingual |
| ScamFerret ( Nakano et al., 2025 ) | ✓ | ✗ | ✗ | ✓ |
| FraudSMSWalker ( Zhou et al., 2026 ) | ✗ | ✗ | ✗ | ✓ |
| Scammer4U ( Roy et al., 2026 ) | ✗ | ✓ | ✓ | ✗ |
| DECEPTICON ( Cuvin et al., 2026 ) | ✗ | ✓ | ✓ | ✗ |
| WebDecept ( Shi et al., 2026 ) | ✗ | ✓ | ✓ | ✗ |
| TrickyArena ( Ersoy et al., 2026 ) | ✗ | ✓ | ✓ | ✗ |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Deceived | ||||||||||
| Mechanism family / mechanism and definition | # Websites | # Red flag | [1pt]5.6 Luna | [1pt]Haiku 5.5 | [1pt]3.8 Flash | [1pt]K3 | [1pt]3.8 Flash | [1pt]V4.1 Flash | ||
| Payment diversion Payee is a private account | 167 | 125 | ||||||||
| M04 | Off-platform payment diversion. Payment goes to a personal account or channel outside the checkout. | 167 | 125 | 0.10 | 0.12 | 0.17 | 0.25 | 0.29 | 0.39 | |
| Non-delivery fraud Item or pet cannot be supplied | 77 | 53 | ||||||||
| M02 | Non-delivery of goods or services. Offers goods or delivery that the seller cannot or will not provide. | 65 | 37 | 0.24 | 0.26 | 0.26 | 0.34 | 0.32 | 0.47 | |
| M36 | Unlicensed or counterfeit medicine. Prescription drugs sold without a prescription or valid licence. | 10 | 9 | 0.00 | 0.10 | 0.20 | 0.30 | 0.10 | 0.60 | |
Web Agents
Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.
See all 189 papers