Graphical User Interface (GUI) agents show strong capabilities for automating web tasks, but existing interactive benchmarks primarily target benign, predictable consumer environments. Their effectiveness in high-stakes, investigative domains such as authentic e-commerce risk management remains underexplored. To bridge this gap, we present RiskWebWorld, the first highly realistic interactive benchmark for evaluating GUI agents in e-commerce risk management. RiskWebWorld features 1,513 tasks sourced from production risk-control pipelines across 8 core domains, and captures the authentic challenges of risk operations on uncooperative websites, partially environmental hijackments. To support scalable evaluation and agentic reinforcement learning (RL), we further build a Gymnasium-compliant infrastructure that decouples policy planning from environment mechanics. Our evaluation across diverse models reveals a dramatic capability gap: top-tier generalist models achieve 49.1% success, while specialized open-weights GUI models lag at near-total failure. This highlights that foundation model scale currently matters more than zero-shot interface grounding in long-horizon professional tasks. We also demonstrate the viability of our infrastructure through agentic RL, which improves open-source models by 16.2%. These results position RiskWebWorld as a practical testbed for developing robust digital workers.
Figures & tables
Figure 1: RiskWebWorld is a highly realistic, interactive benchmark for evaluating GUI agents in e-commerce risk management scenarios. It comprises 1,513 tasks across 8 business domains, with a scalable environment infrastructure that supports rigorous benchmarking and RL training.
Figure 2: RiskWebWorld benchmark infrastructure. Benchmark procedure is defined by the workflow (Sec. 3.4 ): First, remote browser devices are provisioned in the cloud and connected to the environment module through CDP. Then the environment is initialized with the task configuration (Sec. 3.2 ), and the agent (Sec. 3.1 ) starts to interact with the environment (Sec. 3.3 ) in a step-wise manner. After the loop terminates or truncates, evaluation results could serve for both benchmarking and training.
Figure 3: RiskWebWorld task statistics. (a) Task category distribution across 8 business domains. Exact values are given in parentheses. (b) Distributions of execution time and step count across task instances. (c) Comparison of step count distributions between SOP-accompanied and SOP-free tasks.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: An example of a real RiskWebWorld task. The task is to search for the origin and destination of a specific vessel, belonging to the Logistics and Supply Chain Tracking category. In case (a), the agent is given a task instruction and an SOP that provides step-by-step guidance. In case (b), the agent needs to complete the same task without the SOP, which requires stronger reasoning and exploration capabilities.
Figure 5: Environmental hijackments seen in RiskWebWorld tasks. Three representative cases are illustrated: (a) Verification Barrier (e.g., human verification CAPTCHA). (b) Unpredictable Pop-up (e.g., cookie consent forms). (c) Dynamic Content Shift (e.g., network load latency).
Figure 6: Agentic RL training curves. The blue line represents the raw mean reward across trajectories, while the orange line represents the smoothed reward using a moving average. The improvement is not strictly monotonic due to the inherent stochasticity of the environment (e.g., network uncertainty, website load latency). However, the overall upward trend indicates that the agent could learn to improve its performance based on the proposed RiskWebWorld Environment.
Scroll up/down by page units ( pages ), optionally inside a container ( index ).
Appendix
Table 5: Action space used by RiskWebWorld agents.
Figure 7: Online monitoring platform. Task execution details are recorded and managed. The private information is redacted for security purposes.
Figure 8: Three main failure modes of specialized and generalist models: no action generation (action list is empty), degenerate repetitive actions (e.g., repeatedly clicking the same button), and invalid action format (e.g., missing required parameters, truncated action schema due to token limits). The correct action format is crucial for successful execution, and failure to adhere to it leads to immediate execution failures.
Figure 9: Failure trajectory of ShowUI-2B on a Customs Declaration & Clearance Status Audit task. The model correctly identifies the next subgoal and target information in its reasoning, but fails to generate valid executable actions. It repeatedly emits input actions with hallucinated text for non-input elements, and fails to output any valid click action to interact with the page. After receiving execution errors (Step 2 to Step 5), it does not update its internal state and continues to produce similar invalid actions.
Figure 10: Failure trajectory of UI-TARS-1.5-7B on a Product Risk Profile task. The model also identifies the next subgoal and target information, but fails to generate valid executable actions. It tends to output solve_slider_captcha regardless of the page state, and generates an invalid click action with a non-existent DOM index. It also misunderstands the action definition, applying search when it has already reached the target page. From Step 5 to Step 12, we could found that the model repeatedly emits invalid actions without updating its internal state based on execution feedback, which indicates a lack of error recovery and state tracking.
Figure 11: Failure trajectory of Gemini-3-Pro on a Logistics and Supply Chain Tracking task. The model successfully identifies the next subgoal and target information, while generate valid executable actions to interact with the page. However, it does not explore a feasible path to complete the task when the provided websites block access to the required information or present a 404 error. How to enable agents to proactively explore alternative paths outside of the provided candidate websites is an important future direction to enhance their robustness in real-world applications.
Figure 12: Failure trajectory of Qwen3.5-397B-A17B on a Secure Payment Channel Validation task. The model correctly identifies the candidate webpages to complete the task, but fails to generate the true answer. This failure is attributed to the model’s limited ability to perform multi-hop reasoning and information synthesis across multiple pages, where too much states and information is presented and the model struggles to identify and aggregate the relevant pieces to arrive at the correct conclusion.
Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation that limits scalability. We present WebForge, the first fully automated framework that resolves this trilemma through a four-agent pipeline -- Plan, Generate, Refine, and Validate -- that produces interactive, self-contained web environments end-to-end without human annotation. A seven-dimensional difficulty control framework structures task design along navigation depth, visual complexity, reasoning difficulty, and more, enabling systematic capability profiling beyond single aggregate scores. Using WebForge, we construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels. Multi-model experiments show that difficulty stratification effectively differentiates model capabilities, while cross-domain analysis exposes capability biases invisible to aggregate metrics. Together, these results confirm that multi-dimensional evaluation reveals distinct capability profiles that a single aggregate score cannot capture. Code and benchmark are publicly available at https://github.com/yuandaxia2001/WebForge.
Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources. Existing benchmarks leave gaps in executable coverage of their runtime security risks, while evolving model capabilities, harnesses, tools, and threats motivate benchmark evolution. We introduce EvoRiskBench, an evolving benchmark organized around the EP-Path-EF framework, which links an initial risk entry point to a one-hop technical effect through an agent-mediated risk path. The framework defines nine entry-point categories and five effect categories; a 20-participant study supports their interpretability and classification consistency on representative cases. Guided by this framework, an automated end-to-end workflow constructs and executes risk cases in isolated environments and independently verifies outcomes using runtime traces and environment states. The benchmark provides a reproducible dataset of 450 adversarial tasks across six scenarios. We evaluate nine model-harness configurations spanning three models (GPT-5.6 Sol, DeepSeek-V4-Pro-0813, and Claude Opus 5) and three harnesses (Claude Code, Codex, and OpenClaw). Our results reveal substantial vulnerabilities across systems. The most vulnerable configuration, Codex with DeepSeek-V4-Pro-0813, reaches a 68.44% attack success rate (ASR), indicating that configuration of workspace agent is insufficient to ensure secure autonomous execution. ASR varies more across models than harnesses, and harness differences depend on the model. The benchmark cases and evaluation platform will be released after completion of artifact safety and reproducibility checks.
Shiyi Kuang, Xuemei Luo, Kun Liu +7
Novo Ordo for AI · Fudan University · Zhongguancun Laboratory
Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-Language Models (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.
Hongxin Li, Xiping Wang, Jingran Su +4
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3