cs.AIAug 25, 2026
SaveAre Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments
Organizations: Zhejiang University · Yale University · Tongji University
Abstract
GUI agents often encounter dynamic anomalies when deployed on Android devices, from unexpected pop-ups to action misuse, yet existing benchmarks lack systematic evaluation of agent robustness against runtime anomalies. We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories. We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories, and develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions. Evaluating 16 leading GUI models, we reveal universal vulnerability to dynamic anomalies, with even the strongest models suffering significant performance degradation. Furthermore, we conduct GRPO training in both original and adversarial environments to validate our benchmark, separating environment-learnable anomalies from reasoning-bottlenecked ones. Our findings show that while single-step traps at state and action layers are largely addressable through adversarial reinforcement learning, deep contextual traps, like state deadlock, expose intrinsic limitations that cannot be resolved by training in environments with traps alone.
Figures & tables
Figure 1: Four scenarios of runtime anomalies in Android agent tasks from our pilot study, including pop-ups, reasoning errors, grounding errors, and long-sequence accidents. AnTrap taxonomy organizes such anomalies into four layers (State, Thinking, Action, Round) and ten subcategories.
| Benchmark | Online Env | Stochastic Tasks | Intervention | Task Target |
| Computer Agentic Systems | ||||
| ScreenSpot Pro Li et al. (2025) | ✗ | N/A | Grounding | |
| WebArena Zhou et al. (2024) | ✗ | N/A | General Execution | |
| OSWorld Xie et al. (2024) | ✗ | N/A | General Execution | |
| It’s a TRAP! Korgul et al. (2025) | ✗ | Dynamic | Safety | |
| AutoElicit Jones et al. (2026) | ✗ | Dynamic | Safety | |
Table 1: Comparison of recent graphical user interface benchmarks. Unlike prior evaluation suites that predominantly focus on general execution or passive robustness, AnTrap injects runtime anomalies during execution to assess agent robustness in mobile environments with stochastic instructions.
Figure 2: Overview of AnTrap . (a) Original Environment: a base emulator that supports parallel evaluation over base tasks and environments. (b) AnTrap Construction: during dynamic execution, we intercept the agent’s loop and inject four-layer anomalies through the trap toolbox. (c) Diversity Settings: trap type, injection timing, frequency, and content source can be flexibly configured to create diverse and realistic dynamic traps.
| Subcategory | Definition |
| Category 1: State Layer | |
| (1) External Interruption | Unexpected windows, system alerts or application jumps. |
| (2) Visual Obscuration | Rendering errors or image loading failures obscuring critical interface. |
| Category 2: Thinking Layer | |
| (3) Temporal Conflict | False or outdated observations that lead the agent to reason about a misaligned state. |
| (4) Visual Hallucination | Hallucinate text, icons, buttons, relationships, or details that do not exist on the screenshot. |
Table 2: Systematic taxonomy of dynamic traps in the AnTrap benchmark. Comprehensive programmatic implementation details are provided in the Appendix A .
| Model | Orig | S-Layer | T-Layer | A-Layer | R-Layer | Avg | |||||||||||
| Ext.I. | Vis.O. | Tmp.C. | Vis.H. | Grd.E. | Typ.M. | Int.D. | St.DL. | Ctx.D. | Loop | ||||||||
| Human Baseline | |||||||||||||||||
| Human Annotator | 94.1 | 94.1 | 92.4 | - | - | 94.1 | 93.6 | - | 93.2 | 92.8 | - | 93.4 | |||||
| Proprietary Models | |||||||||||||||||
| Claude-Sonnet-4.6 | 74.2 | 64.8 | 66.1 | 66.9 | 72.0 | 69.9 | 67.8 | 69.5 | 63.1 | 63.1 | 61.9 | 66.5 | |||||
| Gemini-3-Pro | 72.9 | 61.9 | 68.6 | 60.2 | 62.7 | 68.6 | 60.6 | 66.1 | 63.6 | 61.4 | 62.3 | 63.6 | |||||
Table 3: Success rates (%) of GUI agents on original tasks and under each trap subcategory of AnTrap . Cell color intensity indicates performance drop from the original baseline. Abbreviations: Ext.I. = External Interruption, Vis.O. = Visual Obscuration, Tmp.C. = Temporal Conflict, Vis.H. = Visual Hallucination, Grd.E. = Grounding Error, Typ.M. = Type Mismatch, Int.D. = Intent Deviation, St.DL. = State Deadlock, Ctx.D. = Context Disruption.
Figure 3: Layer-level performance comparison across training regimes. Grouped bars show the results of baseline, original-env GRPO, and AnTrap -env GRPO for each trap layer.
| GUI-Owl-7B | UI-TARS-7B | ||||||
| Layer | Subcategory | Base | +Orig | +Trap | Base | +Orig | +Trap |
| – | Original Task | 63.1 | 69.9 | 69.5 | 29.7 | 36.0 | 35.6 |
| S | Ext. Interruption | 43.2 | 45.3 | 54.2 | 28.0 | 29.2 | 36.4 |
| Vis. Obscuration | 57.6 | 58.9 | 66.5 | 27.1 | 28.4 | 35.2 | |
| T | Tmp. Conflict | 40.7 | 41.9 | 45.8 | 21.6 | 22.5 | 25.8 |
| Vis. Hallucination | 58.9 | 59.7 | 64.0 | 22.9 | 23.7 | 27.1 | |
Table 4: Success rates (%) under different subcategories of GRPO training regimes. +Orig : trained in original environment; +Trap : trained in AnTrap environment. denotes 5% absolute improvement over baseline.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Subcategory | Implementation |
| Category 1: State Layer | |
| (1) External Interruption | Pop-ups are triggered through a dedicated TrapOverlay APK installed on the emulator, which registers a SYSTEM_ALERT_WINDOW overlay and reacts to Android Debug Bridge (ADB) broadcasts. Four pop-up types are supported: runtime permission dialogs, modal center dialogs, top notification banners, and fullscreen advertisements. Content is drawn from a curated bank of realistic Android prompts ( permission, center-dialog, banner, and fullscreen-ad templates). Alternatively, the agent’s own Vision-Language Model (VLM) generates content online, conditioned on the foreground-app package name; this mode falls back to the template bank on JSON-parse failure or when the foreground process is a system launcher. |
| (2) Visual Obscuration | Pixel-level masks are applied to the captured frame through Pillow. The target region is one of three: a random clickable or editable UI element drawn from the accessibility tree, a uniformly sampled element regardless of affordance, or a random rectangle covering – of the screen. The selected region is replaced by either a Gaussian blur with configurable kernel radius (default ) or a solid color patch when a color is specified. |
| Category 2: Thinking Layer | |
| (3) Temporal Conflict | A rolling buffer of past frames is maintained throughout the episode. At trigger time the current observation passed to the model is replaced with the frame captured steps earlier (default ), while the unperturbed frame is still saved into the agent’s persistent history. As an alternative, a random screenshot can be sampled from a pre-collected pool of unrelated app states and resized to the current resolution. |
| (4) Visual Hallucination | For each text-bearing UI element identified via its accessibility-tree bounding box, the background color is estimated by median sampling along the bbox edges, the original region is overwritten with that color, and a misleading replacement is rendered in DejaVu Sans with the contrast-inverted color. Replacements are drawn from a curated -pair antonym dictionary covering common UI verbs such as Save/Delete, Allow/Deny, and Confirm/Cancel; unrecognized labels fall back to a character-level permutation. |
Table 5: Detailed implementation of the ten dynamic trap operators in the AnTrap benchmark. Each row specifies the concrete mechanism, content sources, and parameters used to realize the corresponding trap during evaluation.
| Component | Hyperparameter | Value |
| Data | Max Prompt Len. | 32768 |
| Data | Max Response Len. | 512 |
| Data | Train Batch Size | 8 |
| Actor/Policy | Parallelism | FSDP2 |
| Actor/Policy | Micro Batch/GPU | 1 |
| Actor/Policy | LR | 1e-6 |
Table 6: Main hyperparameters in GRPO.
| GUI-Owl-7B | UI-TARS-7B | ||||
| Layer | Subcategory | Base | +Mixed | Base | +Mixed |
| – | Original Task | 63.1 | 69.1 | 29.7 | 35.6 |
| S | Ext. Interruption | 43.2 | 46.2 | 28.0 | 30.1 |
| Vis. Obscuration | 57.6 | 59.7 | 27.1 | 29.2 | |
| T | Tmp. Conflict | 40.7 | 42.4 | 21.6 | 22.9 |
| Vis. Hallucination | 58.9 | 60.2 | 22.9 | 24.2 | |
Table 7: Results of mixed-trap GRPO training, where all ten trap subcategories are randomly injected.
Figure 4: Prompts used by the agent’s own VLM to generate External Interruption pop-up content, contextualized on the foreground-app package name { current_app }. The four templates correspond to the four pop-up types rendered by the TrapOverlay APK; on JSON-parse failure or when { current_app } resolves to a system launcher, the framework falls back to the curated -entry template bank described in Section 3.2 .
Figure 5: System prompt used for Claude-Sonnet in AnTrap .
Figure 7: System prompt used for MAI-UI in AnTrap .
Figure 8: System prompt used for UI-TARS in AnTrap .
Figure 9: System prompt used for GUI-Owl-1.5 / Qwen3-VL / Gemini3 in AnTrap .
Figure 10: The web interface for preliminary work, through which we use this annotation website to collect performance data of GUI agents conducting trial and error in real-world scenarios, as well as the error cases they encounter.
Figure 11: For the ExpenseDeleteMultiple task in the original set with no trap injection, GUI-Owl-7B successfully completes the task.
Figure 14: For the ExpenseDeleteMultiple task in the adversarial set with an S-Layer Trap ( Visual Obscuration ) injected at step 5 after GRPO training, GUI-Owl-7B successfully overcomes the perturbation and completes the task.
More in this topic
Mobile GUI Agents
Mobile GUI agents use visual and structured interface information to carry out tasks across smartphone apps.
See all 84 papers