DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections

Li Hu; Kanghua Mo; Yingbin Jin; Qingqing Ye; Haibo Hu

cs.CRSep 29, 2026

DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections

Authors: Li Hu, Kanghua Mo, Yingbin Jin, Qingqing Ye, Haibo Hu

Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University

Abstract

As GUI-based mobile agents rapidly progress, rigorous safety evaluation of their autonomous decision-making in realistic app interfaces becomes increasingly critical. Existing benchmarks mainly focus on execution-level anomalies using task success or hijack rates, but fail to capture the in-task goal deviation risk in multi-candidate selection tasks, where the decision may be steered toward an attacker-specified target, even in violation of instruction-implied constraints (e.g., cheapest/highest-rated), without any overt execution anomalies. We present DIBench, a decision integrity benchmark for measuring this risk in mobile agents. DIBench covers 7 commercial and 3 simulated apps with 5 task types. Under a threat model restricted to non-privileged UI content, we construct 8 deceptive injection probe instantiations that can steer critical selections without overt anomalies. The benchmark includes 1,000 clean and 36,672 injected instances, with a unified protocol and integrity metrics for comparison. Experiments spanning 4 agent frameworks and 7 base models show that completion-based evaluation can overestimate agent trustworthiness and miss decision-integrity risks: deceptive injections steer selections and shift early action policies, inflating completion rates and creating a misleading illusion of safety. Common defenses, including detection, image preprocessing, and prompt reminders, yield inconsistent integrity gains. Overall, DIBench provides a unified, reproducible benchmark to quantify the risk of in-task goal deviation in mobile agents and enable comparable evaluations of safety defenses.

Figures & tables

Benchmark	Decision-Level	Multi-Candidate	Paired Eval.
MSB Lee et al. (2026)	✗	✗	✗
MLA-Trust Yang et al. (2025)	✗	✗	✗
GhostEI-Bench Chen et al. (2025c)	✗	✗	✗
D-GARA Chen et al. (2025d)	✗	✗	✓
SecMoba Yang et al. (2024)	✓	✗	✗
AgentHazard Liu et al. (2025b)	✗	✗	✓

Table 1: Comparison of mobile-agent security benchmarks.

Figure 1: Overview of DIBench: Components, Construction Pipeline, and Extensible Paired Evaluation Suite.

Statistic	Value
#Apps	10 (7 real + 3 simulated)
#Task types	5
#Inst./(app, task type)	20
#Benign instances	$10\times 5\times 20=1{,}000$
Candidate-set size	$n_{i}=\|C_{i}\|,\;n_{i}\in[3,6]$
#Probe variants	$\|\mathcal{P}\|=8$

Table 2: Dataset statistics.

Model	Attack Method	AppAgent			Mobile-Agent-V2			Mobile-Agent-E			Mobile-Agent-V3
Model	Attack Method	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	42.0%	28.4%	-	71.1%	53.8%	-	38.6%	24.9%	-	51.3%	35.5%	-
	Naive Attack	48.0%	30.1%	15.4%	75.9%	51.9%	19.4%	38.3%	24.3%	12.1%	50.4%	35.1%	7.9%
	Escape Attack	42.6%	25.7%	13.2%	73.8%	56.0%	14.7%	40.1%	26.7%	11.7%	56.7%	39.7%	7.2%
	Context Ignore Attack	67.2%	43.3%	43.3%	85.0%	59.6%	44.6%	51.2%	31.9%	32.7%	65.8%	50.9%	35.3%
	Fake Completion Attack	49.2%	32.1%	19.3%	78.9%	58.7%	30.1%	39.1%	26.0%	13.4%	51.7%	34.1%	8.8%

Table 3: Commerce results under deceptive injection. Summary row: for TCR/COR, we report

\Delta=\max(\cdot)-\text{No Attack}

per column; for ASR, we report

\max(\text{ASR})

per column. Within the summary row, max and min are highlighted for each metric.

Figure 2: Action-type distribution on Commerce under No Attack (N.) and Combined Attack (A.) across agent frameworks and model families. Stacked bars show the proportion of each action type.

Attack Method	AppAgent (Q2.5 / Q3)	MobileAgent-E (Q2.5 / Q3)	MobileAgent-V2 (Q2.5 / Q3)	MobileAgent-V3 (Q2.5 / Q3)	Image-Only
No Attack	0.0% / 0.0%	0.0% / 0.0%	0.0% / 0.0%	0.0% / 0.0%	0.0%
Naive Attack	2.7% / 9.3%	0.0% / 2.7%	2.7% / 2.7%	0.0% / 1.3%	18.2%
Escape Attack	1.3% / 4.0%	5.3% / 1.3%	1.3% / 2.7%	1.3% / 0.0%	21.2%
Context Ignore Attack	65.3% / 53.3%	65.3% / 68.0%	61.3% / 69.3%	66.7% / 66.7%	97.0%
Fake Completion Attack	9.3% / 20.0%	9.3% / 13.3%	13.3% / 17.3%	10.7% / 8.0%	79.3%
Combined Attack	72.0% / 66.7%	68.0% / 60.0%	68.0% / 65.3%	77.3% / 66.7%	99.7%

Table 4: Injection-detection recall on Commerce across attack methods with Qwen3-VL-32B . Columns report Planning+Image detection under four agent frameworks with Qwen2.5-VL and Qwen3-VL (Q2.5/Q3), and Image-Only detection. Best and second-best per column are highlighted in dark green and light green.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Cross-channel carrier variants under combined-semantics deceptive injection.

Model	Attack Method	AppAgent			Mobile-Agent-V2			Mobile-Agent-E			Mobile-Agent-V3
Model	Attack Method	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	55.1%	48.7%	-	67.7%	60.5%	-	50.1%	45.9%	-	47.6%	41.2%	-
	Naive Attack	66.4%	55.6%	19.8%	66.0%	55.0%	9.5%	52.9%	46.0%	6.5%	57.2%	47.9%	8.5%
	Escape Attack	62.2%	51.0%	16.1%	65.2%	54.1%	8.7%	49.7%	42.9%	5.5%	51.3%	42.7%	6.0%
	Context Ignore Attack	87.0%	62.3%	42.1%	70.7%	56.6%	18.7%	78.7%	56.6%	39.0%	82.8%	65.0%	31.8%
	Fake Completion Attack	76.2%	62.5%	24.2%	68.0%	55.5%	12.5%	59.3%	49.6%	12.6%	63.4%	50.8%	10.9%

Appendix

Table 5: Simulator results under deceptive injection. Summary row: for TCR/COR, we report

\Delta=\max(\cdot)-\text{No Attack}

per column; for ASR, we report

\max(\text{ASR})

per column. Within the summary row, max and min are highlighted for each metric.

Figure 4: Selection probability over candidate indices in Simulator under No Attack (black) and Combined Attack (colored) across agent frameworks. The x-axis is the item index (Others denotes non-selection). Colored curves indicate the target index; line style denotes the base model (solid: Qwen2.5-VL, dashed: Qwen3-VL).

Framework	Agent Model	Attack Method	No Defense	Noise		JPEG		Resizing
				$\sigma=2$	$\sigma=5$	$Q=95$	$Q=75$	$0.9\times$	$0.75\times$
AppAgent	Qwen2.5-VL	Combined	45.8%	50.7%	60.0%	45.3%	45.3%	50.7%	46.7%
		Context Ignore	43.3%	46.7%	40.0%	45.3%	45.3%	42.7%	36.0%
	Qwen3-VL	Combined	37.9%	29.3%	34.7%	33.3%	29.3%	40.0%	28.0%
		Context Ignore	43.7%	32.0%	34.7%	34.7%	33.3%	41.3%	32.0%
MobileAgent-V2	Qwen2.5-VL	Combined	42.9%	50.7%	49.3%	45.3%	46.7%	53.3%	44.0%

Appendix

Table 6: Attack success rate (ASR) under image-preprocessing defenses on Commerce across agent frameworks, base models, and attack methods. We report the no-defense baseline and three preprocessing operations (Gaussian noise, JPEG compression, and resizing) with different parameter settings. Row-wise min values are highlighted, and entries higher than No Defense are marked as worse than baseline .

Framework	Agent Model	Attack Method	ASR			COR
			No Def $\downarrow$	Prompt $\downarrow$	$\Delta$ $\uparrow$	No Def $\uparrow$	Prompt $\uparrow$	$\Delta$ $\uparrow$
AppAgent	Qwen2.5-VL	Combined	45.8%	44.0%	1.8	46.9%	41.3%	-5.6
		Context Ignore	43.3%	33.3%	10.0	43.3%	45.3%	2.0
	Qwen3-VL	Combined	37.9%	18.7%	19.3	50.8%	58.7%	7.9
		Context Ignore	43.7%	20.0%	23.7	44.4%	57.3%	12.9
MobileAgent-V2	Qwen2.5-VL	Combined	42.9%	41.3%	1.5	50.4%	57.3%	6.9

Appendix

Table 7: Prompt-based defense results on Commerce apps across agent frameworks, base models, and attack methods. We report attack success rate (ASR) and correctness rate (COR) for No Defense and Prompt, together with

\Delta

relative to the no-defense baseline: for ASR,

\Delta=\text{No Def}-\text{Prompt}

; for COR,

\Delta=\text{Prompt}-\text{No Def}

Attack Method	Open-source (Shared)						Open-source (Specialized)						Closed-source
	Qwen2.5-VL			Qwen3-VL			GUI-Owl			Mai-UI			DoubaoSeed-1.6			Gemini-3-Flash			GPT-5.2
	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
No Attack	47.6%	41.2%	-	79.2%	71.5%	-	52.2%	35.1%	-	44.6%	27.4%	-	37.8%	24.7%	-	39.5%	25.8%	-	38.4%	32.6%	-
Naive	57.2%	47.9%	8.5%	85.6%	75.9%	18.7%	52.3%	34.2%	10.6%	40.3%	23.2%	14.9%	35.3%	23.9%	7.9%	37.1%	20.9%	13.8%	42.6%	31.5%	14.1%
Escape	51.3%	42.7%	6.0%	85.6%	75.8%	19.3%	54.3%	37.1%	10.1%	42.9%	25.9%	9.7%	34.3%	22.5%	10.6%	40.4%	25.1%	11.9%	43.4%	30.8%	12.3%
Context Ignore	82.8%	65.0%	31.8%	91.1%	67.7%	43.3%	47.1%	29.1%	19.9%	50.1%	24.1%	26.7%	31.8%	21.6%	5.4%	37.6%	25.5%	5.0%	40.8%	35.1%	5.9%

Appendix

Table 8: Simulator results with Mobile-Agent-V3 under deceptive injection: base-model comparison by model category. Summary row: for TCR/COR, we report

\Delta=\max_{m}(\cdot)-(\cdot)_{\text{No Attack}}

per column; for ASR, we report

\max_{m}(\text{ASR})

per column (max over attack methods

m

). Within the summary row, max and min are highlighted for each metric.

Model	Attack Method	Carrier			Appagent			Mobile_Agent_V2			Mobile_Agent_E			Mobile_Agent_V3			Summary # Max
Model	Attack Method	Title	Cover	Pert.	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	$\times$	$\times$	$\times$	55.1%	48.7%	-	67.7%	60.5%	-	50.1%	45.9%	-	47.6%	41.2%	-	-	-	-
	Combined	$\checkmark$	$\times$	$\times$	90.7%	61.1%	45.0%	75.4%	54.5%	26.0%	79.1%	57.2%	37.4%	83.6%	64.6%	30.7%	2	0	3
	Combined	$\times$	$\checkmark$	$\times$	80.5%	63.7%	26.0%	94.9%	77.8%	20.5%	71.1%	60.6%	19.9%	81.9%	68.3%	19.5%	1	3	0
	Combined	$\checkmark$	$\checkmark$	$\times$	90.2%	61.8%	42.1%	94.1%	74.4%	26.9%	76.0%	57.1%	34.1%	87.1%	69.1%	28.6%	1	1	1
	Combined	$\times$	$\times$	$\checkmark$	51.2%	45.6%	6.5%	84.3%	73.4%	8.1%	43.1%	39.1%	3.4%	57.7%	48.1%	6.2%	0	0	0

Appendix

Table 9: Injection-carrier placement analysis in Simulator across four agent frameworks. We report TCR/COR/ASR for different carrier placements (Title, Cover, Pert.). Summary # Max (last three columns): for each row, we count how many framework columns attain the row-wise maximum in TCR, COR, and ASR, respectively.

Figure 5: Cross-app variation in attack success rate (ASR) under deceptive injection.

Figure 6: Selection distribution by task type on Simulator under No Attack (N.) vs. Combined Attack (A.) across agent frameworks and model families, showing shifts among Correct(Target), Wrong(Target), and Wrong(Non-Target) (stacked).

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 521 papers

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps
Aug 18, 2026Sujin Chen, Lijun Li, Tianyi Du +1AI Agent Security Mobile GUI Agents
Shanghai Artificial Intelligence Laboratory
ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
Aug 25, 2026Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro +3Computer-Use Agent Benchmarks Computer-Use Agent Evaluation
FAIR at Meta
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Mobile GUI Automation GUI Agents
Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

Aug 18, 2026J/K move · Enter open · S save

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

Sujin Chen, Lijun Li, Tianyi Du, Jing Shao

Shanghai Artificial Intelligence Laboratory

AI Agent Security Mobile GUI Agents Prompt Injection Attacks on AI Agents AI Agent Safety Benchmarks

Open paper AskPDF arXiv

LLM-powered GUI agents that autonomously operate smartphones are rapidly transitioning from research prototypes to early real-world deployment. However, because these agents routinely process untrusted environmental content, they are highly vulnerable to environmental injection attacks, which include indirect prompt injections and adversarial instructions. Such attacks can manipulate the behavior of agents without user awareness through diverse channels encountered in everyday mobile use. Despite these risks, existing benchmarks often fail to capture everyday user scenarios, lacking a systematic evaluation of GUI agents under environmental injection attacks on mobile devices. To address this gap, we introduce MobileWorldSafety, a benchmark of 142 risk tasks built on real Android applications. For each task, we define a programmatically verifiable risk indicator over the final system state and evaluate outcomes with a two-stage pipeline: rule-based verification handles unambiguous cases, while an LLM judge adjudicates ambiguous ones. This distinguishes safety failures from capability failures and enables objective and reproducible assessment. Evaluations on six agents, including both general agents and specialized GUI agents, demonstrate that all agents remain highly vulnerable, with attack success rates ranging from 40.4% to 66.9%. These findings indicate that current agents often fail to maintain safety alignment when adversarial content is presented as ordinary mobile context. MobileWorldSafety provides a foundation for quantifying these vulnerabilities and advancing research on robust mobile GUI agents.

Abstract

AI Agent Evaluation

Explore similar work

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents