MobileProbe: Adaptive Security Evaluation of Mobile Agents via Trajectory-Aware Injection
Authors: Yingbin Jin, Ji'an Wang, HaoYang Chai, Hanjun Luo, Li Hu, Xinfeng Li, Huadi Zheng, PeiZhao Hu, Qingqing Ye, Haibo Hu
Organizations: The Hong Kong Polytechnic University · Zhejiang University · Huawei · Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU
Mobile GUI agents read the screen of a phone and operate apps on a user's behalf to complete multi-step tasks such as online shopping or messaging. Because the agent acts on whatever the screen shows, an attacker can embed malicious instructions into ordinary UI content and redirect its actions. We refer to this threat as environmental injection attack (EIA). Existing benchmarks measure this risk by pinning hand-crafted payloads to a fixed step of a fixed task. This approach faces two limitations. First, each payload is bound to one task in one app and does not transfer without manual redesign. Second, agents given the same task may follow different routes, so an agent that never reaches the injected page is recorded as safe without being tested. This insufficient test coverage can cause fixed-point benchmarks to underestimate the security risk of deployed mobile agents. In this work, we propose MobileProbe, a trajectory-aware red-teaming framework that evaluates mobile agents adaptively. MobileProbe discovers attacker-controllable content surfaces through automated pentesting, localizes vulnerable decision points along the agent's trajectory, and iteratively refines context-aware payloads through a surrogate-filtered evolutionary loop. We construct a benchmark of 83 long-horizon attack instances spanning 12 mobile applications. Each is annotated with a Confidentiality, Integrity, or Availability (CIA) attack goal to cover the principal categories of security threats facing mobile agents. On three representative agent architectures, MobileProbe attains higher attack success rates than human-written, zero-shot LLM, and prior automated red-teaming baselines adapted to the mobile domain. MobileProbe can serve as a pre-deployment auditing tool to systematically uncover vulnerabilities and inform security hardening of mobile agents.
Figures & tables
Fig. 1: Illustration of Insufficient Test Coverage . Two agents given the same task follow divergent execution paths. Agent A reaches the planted injection point and is exposed to the attack; Agent B follows a different route, never encounters the payload, and is recorded as safe even though it was never tested. Fixed-point benchmarks therefore cannot guarantee comparable vulnerability exposure across agents.
Attack Vector
Privilege
Reliability
Capacity
Alternative Candidates (Excluded)
Overlay Window
Privileged †
High
Unconstrained
System Notification
Unprivileged
Low
Severely Limited
UI Identity Spoof
Unprivileged
Low
Misidentify Only
Selected Vectors
Malicious Text
Unprivileged
High
Unconstrained
TABLE I: Comparison of candidate visual injection vectors.
Fig. 2: Overview of the MobileProbe framework.
Fig. 3: Red Team Agentic System.
Strategy
Payload Miss Rate (PMR) ↓
Gemini
Agentic
GUI-Owl
Expert-Fixed
13.3%
12.0%
22.9%
Cross-Agent
12.0%
10.8%
15.7%
MobileProbe
4.8%
3.6%
6.0%
TABLE II: Test Coverage Across Injection Strategies
Method
Attack Success Rate (ASR) ↑
Gemini
Agentic
GUI-Owl
Overall Performance
Static Template
6.8%
8.0%
34.5%
Zero-shot LLM
8.0%
9.2%
15.7%
Adapted PAIR
56.6%
54.6%
77.5%
Adapted Genesis
40.6%
39.0%
61.4%
TABLE III: Attack Effectiveness Comparison with Baselines
Dimension
Category
Attack Success Rate (ASR) ↑
Gemini
Agentic
GUI-Owl
Task Scope
Single-App
77.8%
82.4%
75.9%
Cross-App
90.8%
89.4%
92.2%
Attack Goal
Confidentiality
88.9%
88.9%
85.2%
Integrity
91.7%
89.3%
89.3%
Availability
75.0%
81.0%
81.0%
TABLE IV: Attack Effectiveness Across Scopes and Goals
Method
Gemini (%)
Agentic (%)
GUI-Owl (%)
SHR ↑
ASR-F ↑
SHR ↑
ASR-F ↑
SHR ↑
ASR-F ↑
w/o Attack
BU = 54.6
BU = 56.6
BU = 26.5
Static Template
0.0
13.3
0.0
15.7
21.4
31.7
Zero-shot LLM
0.0
9.7
33.3
9.3
0.0
16.4
Adapted PAIR
47.3
62.8
51.8
57.4
46.3
74.3
Adapted Genesis
37.5
15.9
40.2
9.3
42.9
47.5
TABLE V: Attack Quality Comparison with Baselines
Fig. 4: Human evaluation of MobileProbe payload quality.
Variant
ASR ↑
Avg. Rounds ↓
MobileProbe
85.1%
7.9
w/o Multi-signal Feedback
76.3%
11.4
w/o Evolutionary Refinement
28.5%
N/A
w/o Surrogate Pre-filtering
63.9%
12.8
TABLE VI: Component Analysis on Gemini-3.1-Pro.
Score
Context Relevance (Rel)
Decision Criticality (Crit)
Reasoning Exploitability (Expl)
1
Unrelated to the attack objective.
Routine browsing; no substantive decision.
Independent task focus; resistant to distraction.
2
Same app; no specific target-related data.
Exploratory, fully reversible interaction.
Environment-aware, but core intent misaligned.
3
Relevant context provides an injection basis.
Preparation for a key action.
Seeking clues or input; open to suggestions.
4
Abundant target-related information supports injection.
State-changing decision alters task flow.
Reasoning aligned; only execution parameters missing.
5
Direct access to attack-relevant data or operations.
Consequential final or irreversible action.
Chain-of-thought fully aligned with triggering the attack.
TABLE VII: Vulnerability Scoring Rubric for Candidate Pages.
Type
Description
Strategy Space ( Σ )
Operational Suggestion
Frames the instruction as a workflow optimization recommendation.
Urgency Appeal
Manufactures time pressure to compel immediate response.
Authority Impersonation
Attributes instructions to an entity with perceived legitimate authority.
Contextual Deception
Supplies false premises rather than direct instructions.
Social Context
Leverages peer behavior or community norms to motivate compliance.
TABLE VIII: Description of Strategy ( Σ ) and Style ( Ω ) Dimensions.
App
Page Type
Trigger Signals
Exclusion Conditions
Mail
Email Detail
Subject, sender, recipient chip, body text
Compose, Search in mail
Messages
Conversation Detail
Conversation title, received body, input field
Start chat, Create group, Set up Messages
Mattermost
Conversation Detail
Channel title, message list, input field
Find channels, Browse channels, Create channel
Taodian
Product Detail
Product title, action buttons, merchant info
Home, category, cart, profile, search pages
Mastodon
Post Detail
Post title, author, body/media, reply field
Composer, Edit profile, Search
Mastodon
Hashtag Feed
Hashtag title, feed subtitle, author, body
TABLE IX: Injectable Page Templates for Malicious Text.
App
Page Types
Count
Maps
Home, Search Suggestions, Place Detail, Place Actions, Route Result, Fastest Route
6
Mastodon
New Post, Edit Post, Edit Profile, Profile, Profile Posts, Post Detail, Hashtag Feed, Manage Hashtags, Account Settings
LLM-powered GUI agents that autonomously operate smartphones are rapidly transitioning from research prototypes to early real-world deployment. However, because these agents routinely process untrusted environmental content, they are highly vulnerable to environmental injection attacks, which include indirect prompt injections and adversarial instructions. Such attacks can manipulate the behavior of agents without user awareness through diverse channels encountered in everyday mobile use. Despite these risks, existing benchmarks often fail to capture everyday user scenarios, lacking a systematic evaluation of GUI agents under environmental injection attacks on mobile devices. To address this gap, we introduce MobileWorldSafety, a benchmark of 142 risk tasks built on real Android applications. For each task, we define a programmatically verifiable risk indicator over the final system state and evaluate outcomes with a two-stage pipeline: rule-based verification handles unambiguous cases, while an LLM judge adjudicates ambiguous ones. This distinguishes safety failures from capability failures and enables objective and reproducible assessment. Evaluations on six agents, including both general agents and specialized GUI agents, demonstrate that all agents remain highly vulnerable, with attack success rates ranging from 40.4% to 66.9%. These findings indicate that current agents often fail to maintain safety alignment when adversarial content is presented as ordinary mobile context. MobileWorldSafety provides a foundation for quantifying these vulnerabilities and advancing research on robust mobile GUI agents.
As GUI-based mobile agents rapidly progress, rigorous safety evaluation of their autonomous decision-making in realistic app interfaces becomes increasingly critical. Existing benchmarks mainly focus on execution-level anomalies using task success or hijack rates, but fail to capture the in-task goal deviation risk in multi-candidate selection tasks, where the decision may be steered toward an attacker-specified target, even in violation of instruction-implied constraints (e.g., cheapest/highest-rated), without any overt execution anomalies. We present DIBench, a decision integrity benchmark for measuring this risk in mobile agents. DIBench covers 7 commercial and 3 simulated apps with 5 task types. Under a threat model restricted to non-privileged UI content, we construct 8 deceptive injection probe instantiations that can steer critical selections without overt anomalies. The benchmark includes 1,000 clean and 36,672 injected instances, with a unified protocol and integrity metrics for comparison. Experiments spanning 4 agent frameworks and 7 base models show that completion-based evaluation can overestimate agent trustworthiness and miss decision-integrity risks: deceptive injections steer selections and shift early action policies, inflating completion rates and creating a misleading illusion of safety. Common defenses, including detection, image preprocessing, and prompt reminders, yield inconsistent integrity gains. Overall, DIBench provides a unified, reproducible benchmark to quantify the risk of in-task goal deviation in mobile agents and enable comparable evaluations of safety defenses.
Li Hu, Kanghua Mo, Yingbin Jin +2
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University
The rise of autonomous AI agents represents a major paradigm shift in how users interact with mobile devices. Frameworks such as MobileRun and Mobile-Use can autonomously navigate Android applications and execute complex multi-step tasks. To interpret user interfaces, these frameworks rely primarily on Android accessibility (A11y) trees and secondarily on visual screenshots. In this paper, we demonstrate that this architectural dependence on unsanitized accessibility metadata, together with visual input, introduces a systemic vulnerability to indirect prompt injection. We show that adversarial prompts can cause autonomous agents to abandon their original objectives, violate context boundaries, and perform unauthorized device actions. Our empirical evaluation demonstrates goal hijacking, context drift, and unauthorized actions across visually hidden and fully exposed attack scenarios. In aggregate, MobileRun reaches an attack success rate of 0.822 with Gemma4:31B, while Mobile-Use with Qwen3.6:35B reduces this to 0.150 but does not eliminate context drift or unauthorized actions. These findings reveal that current mobile agent frameworks fail to enforce semantic context boundaries, treating passive environmental text as trusted instructions. Finally, we present a taxonomy of these attacks and discuss the need for zero-trust input validation, dedicated security agents, and strict context isolation within mobile agent architectures.
Rahul Deivasigamani, Sayeda Faatin Alvi, Derqui Andrea +2