DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections

Li Hu; Kanghua Mo; Yingbin Jin; Qingqing Ye; Haibo Hu

cs.CRSep 29, 2026

DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections

Authors: Li Hu, Kanghua Mo, Yingbin Jin, Qingqing Ye, Haibo Hu

Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University

Abstract

As GUI-based mobile agents rapidly progress, rigorous safety evaluation of their autonomous decision-making in realistic app interfaces becomes increasingly critical. Existing benchmarks mainly focus on execution-level anomalies using task success or hijack rates, but fail to capture the in-task goal deviation risk in multi-candidate selection tasks, where the decision may be steered toward an attacker-specified target, even in violation of instruction-implied constraints (e.g., cheapest/highest-rated), without any overt execution anomalies. We present DIBench, a decision integrity benchmark for measuring this risk in mobile agents. DIBench covers 7 commercial and 3 simulated apps with 5 task types. Under a threat model restricted to non-privileged UI content, we construct 8 deceptive injection probe instantiations that can steer critical selections without overt anomalies. The benchmark includes 1,000 clean and 36,672 injected instances, with a unified protocol and integrity metrics for comparison. Experiments spanning 4 agent frameworks and 7 base models show that completion-based evaluation can overestimate agent trustworthiness and miss decision-integrity risks: deceptive injections steer selections and shift early action policies, inflating completion rates and creating a misleading illusion of safety. Common defenses, including detection, image preprocessing, and prompt reminders, yield inconsistent integrity gains. Overall, DIBench provides a unified, reproducible benchmark to quantify the risk of in-task goal deviation in mobile agents and enable comparable evaluations of safety defenses.

Figures & tables

Benchmark	Decision-Level	Multi-Candidate	Paired Eval.
MSB Lee et al. (2026)	✗	✗	✗
MLA-Trust Yang et al. (2025)	✗	✗	✗
GhostEI-Bench Chen et al. (2025c)	✗	✗	✗
D-GARA Chen et al. (2025d)	✗	✗	✓
SecMoba Yang et al. (2024)	✓	✗	✗
AgentHazard Liu et al. (2025b)	✗	✗	✓

Table 1: Comparison of mobile-agent security benchmarks.

Figure 1: Overview of DIBench: Components, Construction Pipeline, and Extensible Paired Evaluation Suite.

Statistic	Value
#Apps	10 (7 real + 3 simulated)
#Task types	5
#Inst./(app, task type)	20
#Benign instances	$10\times 5\times 20=1{,}000$
Candidate-set size	$n_{i}=\|C_{i}\|,\;n_{i}\in[3,6]$
#Probe variants	$\|\mathcal{P}\|=8$

Table 2: Dataset statistics.

Model	Attack Method	AppAgent			Mobile-Agent-V2			Mobile-Agent-E			Mobile-Agent-V3
Model	Attack Method	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	42.0%	28.4%	-	71.1%	53.8%	-	38.6%	24.9%	-	51.3%	35.5%	-
	Naive Attack	48.0%	30.1%	15.4%	75.9%	51.9%	19.4%	38.3%	24.3%	12.1%	50.4%	35.1%	7.9%
	Escape Attack	42.6%	25.7%	13.2%	73.8%	56.0%	14.7%	40.1%	26.7%	11.7%	56.7%	39.7%	7.2%
	Context Ignore Attack	67.2%	43.3%	43.3%	85.0%	59.6%	44.6%	51.2%	31.9%	32.7%	65.8%	50.9%	35.3%
	Fake Completion Attack	49.2%	32.1%	19.3%	78.9%	58.7%	30.1%	39.1%	26.0%	13.4%	51.7%	34.1%	8.8%

Table 3: Commerce results under deceptive injection. Summary row: for TCR/COR, we report

\Delta=\max(\cdot)-\text{No Attack}

per column; for ASR, we report

\max(\text{ASR})

per column. Within the summary row, max and min are highlighted for each metric.

Figure 2: Action-type distribution on Commerce under No Attack (N.) and Combined Attack (A.) across agent frameworks and model families. Stacked bars show the proportion of each action type.

Attack Method	AppAgent (Q2.5 / Q3)	MobileAgent-E (Q2.5 / Q3)	MobileAgent-V2 (Q2.5 / Q3)	MobileAgent-V3 (Q2.5 / Q3)	Image-Only
No Attack	0.0% / 0.0%	0.0% / 0.0%	0.0% / 0.0%	0.0% / 0.0%	0.0%
Naive Attack	2.7% / 9.3%	0.0% / 2.7%	2.7% / 2.7%	0.0% / 1.3%	18.2%
Escape Attack	1.3% / 4.0%	5.3% / 1.3%	1.3% / 2.7%	1.3% / 0.0%	21.2%
Context Ignore Attack	65.3% / 53.3%	65.3% / 68.0%	61.3% / 69.3%	66.7% / 66.7%	97.0%
Fake Completion Attack	9.3% / 20.0%	9.3% / 13.3%	13.3% / 17.3%	10.7% / 8.0%	79.3%
Combined Attack	72.0% / 66.7%	68.0% / 60.0%	68.0% / 65.3%	77.3% / 66.7%	99.7%

Table 4: Injection-detection recall on Commerce across attack methods with Qwen3-VL-32B . Columns report Planning+Image detection under four agent frameworks with Qwen2.5-VL and Qwen3-VL (Q2.5/Q3), and Image-Only detection. Best and second-best per column are highlighted in dark green and light green.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Cross-channel carrier variants under combined-semantics deceptive injection.

Model	Attack Method	AppAgent			Mobile-Agent-V2			Mobile-Agent-E			Mobile-Agent-V3
Model	Attack Method	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	55.1%	48.7%	-	67.7%	60.5%	-	50.1%	45.9%	-	47.6%	41.2%	-
	Naive Attack	66.4%	55.6%	19.8%	66.0%	55.0%	9.5%	52.9%	46.0%	6.5%	57.2%	47.9%	8.5%
	Escape Attack	62.2%	51.0%	16.1%	65.2%	54.1%	8.7%	49.7%	42.9%	5.5%	51.3%	42.7%	6.0%
	Context Ignore Attack	87.0%	62.3%	42.1%	70.7%	56.6%	18.7%	78.7%	56.6%	39.0%	82.8%	65.0%	31.8%
	Fake Completion Attack	76.2%	62.5%	24.2%	68.0%	55.5%	12.5%	59.3%	49.6%	12.6%	63.4%	50.8%	10.9%

Appendix

Table 5: Simulator results under deceptive injection. Summary row: for TCR/COR, we report

\Delta=\max(\cdot)-\text{No Attack}

per column; for ASR, we report

\max(\text{ASR})

per column. Within the summary row, max and min are highlighted for each metric.

Figure 4: Selection probability over candidate indices in Simulator under No Attack (black) and Combined Attack (colored) across agent frameworks. The x-axis is the item index (Others denotes non-selection). Colored curves indicate the target index; line style denotes the base model (solid: Qwen2.5-VL, dashed: Qwen3-VL).

Framework	Agent Model	Attack Method	No Defense	Noise		JPEG		Resizing
				$\sigma=2$	$\sigma=5$	$Q=95$	$Q=75$	$0.9\times$	$0.75\times$
AppAgent	Qwen2.5-VL	Combined	45.8%	50.7%	60.0%	45.3%	45.3%	50.7%	46.7%
		Context Ignore	43.3%	46.7%	40.0%	45.3%	45.3%	42.7%	36.0%
	Qwen3-VL	Combined	37.9%	29.3%	34.7%	33.3%	29.3%	40.0%	28.0%
		Context Ignore	43.7%	32.0%	34.7%	34.7%	33.3%	41.3%	32.0%
MobileAgent-V2	Qwen2.5-VL	Combined	42.9%	50.7%	49.3%	45.3%	46.7%	53.3%	44.0%

Appendix

Table 6: Attack success rate (ASR) under image-preprocessing defenses on Commerce across agent frameworks, base models, and attack methods. We report the no-defense baseline and three preprocessing operations (Gaussian noise, JPEG compression, and resizing) with different parameter settings. Row-wise min values are highlighted, and entries higher than No Defense are marked as worse than baseline .

Framework	Agent Model	Attack Method	ASR			COR
			No Def $\downarrow$	Prompt $\downarrow$	$\Delta$ $\uparrow$	No Def $\uparrow$	Prompt $\uparrow$	$\Delta$ $\uparrow$
AppAgent	Qwen2.5-VL	Combined	45.8%	44.0%	1.8	46.9%	41.3%	-5.6
		Context Ignore	43.3%	33.3%	10.0	43.3%	45.3%	2.0
	Qwen3-VL	Combined	37.9%	18.7%	19.3	50.8%	58.7%	7.9
		Context Ignore	43.7%	20.0%	23.7	44.4%	57.3%	12.9
MobileAgent-V2	Qwen2.5-VL	Combined	42.9%	41.3%	1.5	50.4%	57.3%	6.9

Appendix

Table 7: Prompt-based defense results on Commerce apps across agent frameworks, base models, and attack methods. We report attack success rate (ASR) and correctness rate (COR) for No Defense and Prompt, together with

\Delta

relative to the no-defense baseline: for ASR,

\Delta=\text{No Def}-\text{Prompt}

; for COR,

\Delta=\text{Prompt}-\text{No Def}

Attack Method	Open-source (Shared)						Open-source (Specialized)						Closed-source
	Qwen2.5-VL			Qwen3-VL			GUI-Owl			Mai-UI			DoubaoSeed-1.6			Gemini-3-Flash			GPT-5.2
	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
No Attack	47.6%	41.2%	-	79.2%	71.5%	-	52.2%	35.1%	-	44.6%	27.4%	-	37.8%	24.7%	-	39.5%	25.8%	-	38.4%	32.6%	-
Naive	57.2%	47.9%	8.5%	85.6%	75.9%	18.7%	52.3%	34.2%	10.6%	40.3%	23.2%	14.9%	35.3%	23.9%	7.9%	37.1%	20.9%	13.8%	42.6%	31.5%	14.1%
Escape	51.3%	42.7%	6.0%	85.6%	75.8%	19.3%	54.3%	37.1%	10.1%	42.9%	25.9%	9.7%	34.3%	22.5%	10.6%	40.4%	25.1%	11.9%	43.4%	30.8%	12.3%
Context Ignore	82.8%	65.0%	31.8%	91.1%	67.7%	43.3%	47.1%	29.1%	19.9%	50.1%	24.1%	26.7%	31.8%	21.6%	5.4%	37.6%	25.5%	5.0%	40.8%	35.1%	5.9%

Appendix

Table 8: Simulator results with Mobile-Agent-V3 under deceptive injection: base-model comparison by model category. Summary row: for TCR/COR, we report

\Delta=\max_{m}(\cdot)-(\cdot)_{\text{No Attack}}

per column; for ASR, we report

\max_{m}(\text{ASR})

per column (max over attack methods

m

). Within the summary row, max and min are highlighted for each metric.

Model	Attack Method	Carrier			Appagent			Mobile_Agent_V2			Mobile_Agent_E			Mobile_Agent_V3			Summary # Max
Model	Attack Method	Title	Cover	Pert.	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR	TCR	COR	ASR
Qwen2.5-VL	No Attack	$\times$	$\times$	$\times$	55.1%	48.7%	-	67.7%	60.5%	-	50.1%	45.9%	-	47.6%	41.2%	-	-	-	-
	Combined	$\checkmark$	$\times$	$\times$	90.7%	61.1%	45.0%	75.4%	54.5%	26.0%	79.1%	57.2%	37.4%	83.6%	64.6%	30.7%	2	0	3
	Combined	$\times$	$\checkmark$	$\times$	80.5%	63.7%	26.0%	94.9%	77.8%	20.5%	71.1%	60.6%	19.9%	81.9%	68.3%	19.5%	1	3	0
	Combined	$\checkmark$	$\checkmark$	$\times$	90.2%	61.8%	42.1%	94.1%	74.4%	26.9%	76.0%	57.1%	34.1%	87.1%	69.1%	28.6%	1	1	1
	Combined	$\times$	$\times$	$\checkmark$	51.2%	45.6%	6.5%	84.3%	73.4%	8.1%	43.1%	39.1%	3.4%	57.7%	48.1%	6.2%	0	0	0

Appendix

Table 9: Injection-carrier placement analysis in Simulator across four agent frameworks. We report TCR/COR/ASR for different carrier placements (Title, Cover, Pert.). Summary # Max (last three columns): for each row, we count how many framework columns attain the row-wise maximum in TCR, COR, and ASR, respectively.

Figure 5: Cross-app variation in attack success rate (ASR) under deceptive injection.

Figure 6: Selection distribution by task type on Simulator under No Attack (N.) vs. Combined Attack (A.) across agent frameworks and model families, showing shifts among Correct(Target), Wrong(Target), and Wrong(Non-Target) (stacked).

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 521 papers

Aug 18, 2026

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

LLM-powered GUI agents that autonomously operate smartphones are rapidly transitioning from research prototypes to early real-world deployment. However, because these agents routinely process untrusted environmental content, they are highly vulnerable to environmental injection attacks, which include indirect prompt injections and adversarial instructions. Such attacks can manipulate the behavior of agents without user awareness through diverse channels encountered in everyday mobile use. Despite these risks, existing benchmarks often fail to capture everyday user scenarios, lacking a systematic evaluation of GUI agents under environmental injection attacks on mobile devices. To address this gap, we introduce MobileWorldSafety, a benchmark of 142 risk tasks built on real Android applications. For each task, we define a programmatically verifiable risk indicator over the final system state and evaluate outcomes with a two-stage pipeline: rule-based verification handles unambiguous cases, while an LLM judge adjudicates ambiguous ones. This distinguishes safety failures from capability failures and enables objective and reproducible assessment. Evaluations on six agents, including both general agents and specialized GUI agents, demonstrate that all agents remain highly vulnerable, with attack success rates ranging from 40.4% to 66.9%. These findings indicate that current agents often fail to maintain safety alignment when adversarial content is presented as ordinary mobile context. MobileWorldSafety provides a foundation for quantifying these vulnerabilities and advancing research on robust mobile GUI agents.

Sujin Chen, Lijun Li, Tianyi Du +1

Shanghai Artificial Intelligence Laboratory

AI Agent Security Mobile GUI Agents Prompt Injection Attacks on AI Agents AI Agent Safety Benchmarks

Ask PDF

Aug 25, 2026

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguous instructions. We introduce ADeptS-Bench, a dual-stream trustworthiness benchmark, grounded in the ADEPTS capability framework and general population user studies. The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface. The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous. Evaluating seven models reveals that no model consistently exceeds 80% task success while staying below 30% attack success; every model clicks "Checkout" on a $25K order without hesitation, and none detects that a "factory reset" button is mislabeled as "Optimize." An ablation reveals three distinct safety architectures: tool-dependent (ASR +21-23pp without refusal tool), partially tool-dependent (+10-11pp), and no mechanism (unchanged). In disambiguation, all models overestimate consequence severity, mirroring the over-refusal bias observed in safety. We release all data, evaluation code, and analysis tools upon publication.

Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro +3

FAIR at Meta

Computer-Use Agent Benchmarks Computer-Use Agent Evaluation Computer-Use Agents AI Agent Safety Benchmarks AI Agent Security Benchmarks Computer-Use Agent Safety+1

Ask PDF

Sep 7, 2026

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.

Jintian Feng, Long Chen, Xiao Yu +7

Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

Mobile GUI Automation GUI Agents Agent Evaluation Mobile GUI Agents Mobile Agent Benchmarks

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents