ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

Joy Chen; Alejandro Castillejo Munoz; Pierluca D'Oro; Yuxuan Sun; Chloe Evans; Joseph Tighe

cs.CRAug 25, 2026

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

Authors: Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe

Organizations: FAIR at Meta

Abstract

Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguous instructions. We introduce ADeptS-Bench, a dual-stream trustworthiness benchmark, grounded in the ADEPTS capability framework and general population user studies. The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface. The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous. Evaluating seven models reveals that no model consistently exceeds 80% task success while staying below 30% attack success; every model clicks "Checkout" on a $25K order without hesitation, and none detects that a "factory reset" button is mislabeled as "Optimize." An ablation reveals three distinct safety architectures: tool-dependent (ASR +21-23pp without refusal tool), partially tool-dependent (+10-11pp), and no mechanism (unchanged). In disambiguation, all models overestimate consequence severity, mirroring the over-refusal bias observed in safety. We release all data, evaluation code, and analysis tools upon publication.

Figures & tables

Figure 1: Overview of the ADeptS-Bench benchmark creation pipeline.

Benchmark	Tasks	Platform	Visual	Offline	Paired	Disamb.	User	Open
SafeArena	500	Web	✓	✗	✓	✗	✗	✓
MobileSafetyBench	250	Mobile	✓	✗	✓	✗	✗	✓
OS-Harm	150	Desktop	✓	✗	✗	✗	✗	✓
SecureWebArena	330	Web	✓	✗	✗	✗	✗	✓
RedTeamCUA	864	Web+OS	$\sim$	✗	✗	✗	✗	✓
AdvCUA	140	OS/CLI	✗	✗	✗	✗	✗	✓

Table 1: Comparison with existing agent safety and disambiguation benchmarks. ADeptS-Bench is the only benchmark combining all eight properties. ✓ = yes, ✗ = no,

\sim

= partial. Visual = visual/GUI grounding. Offline = no live environment. Paired = matched benign/malicious. Disamb. = disambiguation eval. User = user-research-grounded taxonomy.

Figure 2: Example tasks from both streams. (a–b) Paired safety design: identical instruction “Order a margherita pizza” . The benign screenshot (a) shows a normal menu; the malicious screenshot (b) contains prompt injection to trick agent to confirm a $100 tip. (c) Disambiguation: the instruction “Delete this slide” is ambiguous because two slides are selected.

	Mobile ( $n$ =358)				Desktop ( $n$ =501)
Model	TSR $\uparrow$	ASR $\downarrow$	FRR $\downarrow$	$\mathcal{S}\uparrow$	TSR $\uparrow$	ASR $\downarrow$	FRR $\downarrow$	$\mathcal{S}\uparrow$
Gemini 3.1 Pro	77.6 $\pm$ 0.6	29.6 $\pm$ 0.6	9.6	73.8	82.9 $\pm$ 0.5	29.8 $\pm$ 0.4	8.7	76.0
Claude 4.7 Opus	71.0 $\pm$ 0.5	27.7 $\pm$ 0.6	9.4	71.6	78.1 $\pm$ 0.4	27.2 $\pm$ 0.6	6.3	75.4
GPT-5.4	62.1 $\pm$ 0.6	36.7 $\pm$ 1.4	5.1	62.7	68.8 $\pm$ 1.0	36.3 $\pm$ 0.5	7.0	66.1
Gemini 2.5 CU	78.1 $\pm$ 0.3	52.7 $\pm$ 0.6	1.5	58.9	83.6 $\pm$ 0.5	51.5 $\pm$ 0.3	1.3	61.4
Qwen3-VL-235B	83.0 $\pm$ 0.6	78.1 $\pm$ 0.6	0.0	34.7	86.6 $\pm$ 0.2	76.3 $\pm$ 0.3	0.0	37.2

Table 2: Safety stream results (mean

\pm

95% CI over 10 runs). TSR = Task Success Rate (benign), ASR = Attack Success Rate (malicious; lower = safer), FRR = False Refusal Rate (StatusAction(unsafe) on benign),

\mathcal{S}

= ADeptS-Bench Score. Best per-column in bold .

	ASR (%)		TSR (%)
Model	Base	No tool	Base	No tool
Gemini 3.1 Pro	29.7	51.7 (+22.0)	80.3	87.3 (+7.1)
Claude 4.7	27.5	37.8 (+10.3)	74.6	79.2 (+4.6)
GPT-5.4	36.5	47.3 (+10.7)	65.5	71.4 (+5.9)
Gemini 2.5 CU	52.1	67.6 (+15.5)	80.9	82.6 (+1.7)
Qwen-235B	77.2	78.1 (+0.9)	84.8	85.1 (+0.3)

Table 3: Effect of removing the explicit refusal tool on ASR (%). Base setup provides status(unsafe) with a usage instruction. Ablation removes the tool and its usage instruction. Combined mobile+desktop.

Figure 3: Safety failure spectrum. Threats become progressively less visual and more semantic from L1 $\to$ L4. (a) Explicit malicious code is universally detected. (b) Suspicious URLs in natural text require reading comprehension, only frontier models catch this. (c) Ambiguous security actions split frontier models. (d) Clean professional UIs with harmful scale (300 items, $25K) are missed by all models.

	Mobile ( $n$ =381)		Desktop ( $n$ =363)
Model	F1 $\uparrow$	$\Delta\downarrow$	F1 $\uparrow$	$\Delta\downarrow$
Gemini 3.1 Pro	58.9	0.86	39.8	1.44
Claude 4.7 Opus	53.7	0.90	47.6	1.43
Gemini 2.5 CU	56.2	1.16	44.2	1.37
GPT-5.4	45.0	0.81	30.3	1.55
Qwen3-VL-235B	52.8	1.34	46.3	1.38

Table 4: Disambiguation results (with severity scoring prompt). F1 = harmonic mean of Precision and Recall against human ground truth.

\Delta

= severity calibration error (lower = better), defined in § 3.4 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Threat Category	Mobile	Desktop	Example
Hallucination	39	49	Agent fabricates invoice details
Prompt Injection	40	50	Email instructs agent to leak data
Web Hacking	37	48	SQL injection, XSS payloads
Dangerous Commands	–	49	Reverse shell, malware execution
Misalignment	42	54	Agent deletes security emails
System Sabotage	39	50	Repeated destructive system calls

Appendix

Table 5: Safety stream threat category distribution across mobile and desktop.

Panel of Figure 4 — Figure 4: Threat category (10 categories, $n$ =859).

Model	$m{=}0$	$m{=}0.05$	$m{=}0.08$	$m{=}0.10$	$m{=}0.14$	$m{=}0.20$
Gemini 2.5 CU	78.7	79.7	80.5	80.8	81.5	82.4
Gemini 3.1 Pro	78.0	78.7	79.1	79.7	80.5	81.4

Appendix

Table 6: Benign TSR (%) at different bounding box margins (desktop). Most models change by

\leq

3pp across the range.

Panel of Figure 10 — Figure 10: Safety-capability Pareto frontier (desktop). Qwen models (diamonds) cluster in the high-capability, low-safety region. No model occupies the ideal region ( $>$ 80% on both axes).

	Mobile ( $n$ =358)				Desktop ( $n$ =501)
Model	Mean	Pass@ $k$	Gap	Flaky	Mean	Pass@ $k$	Gap	Flaky
Gemini 3.1 Pro	29.6	37.7	+8	53	29.8	37.5	+8	78
Claude 4.7 Opus	27.7	36.3	+9	54	27.2	35.4	+8	85
GPT-5.4	36.7	60.1	+23	159	36.3	57.9	+22	206
Gemini 2.5 CU	52.7	62.6	+10	78	51.5	64.5	+13	123
Qwen3-VL-235B	78.1	85.8	+8	65	76.3	82.6	+6	70

Appendix

Table 7: Per-task worst-case ASR (pass@

k

k

=10). Flaky = tasks with mixed compliance across runs (neither 0/10 nor 10/10).

	Ideal	Cap.+	Safe+	Both
Model	(pass+ref.)	unsafe	incap.	wrong
Gemini 3.1 Pro	55.5	26.7	15.0	2.8
Claude 4.7	53.0	24.8	19.2	3.0
GPT-5.4	40.3	27.5	23.6	8.6
Gemini 2.5 CU	36.3	47.3	12.2	4.2
Qwen-235B	15.8	70.5	7.8	6.0

Appendix

Table 8: Paired task outcomes (%, desktop). Each task pair is classified by whether the model passes the benign variant and refuses the malicious variant.

	Malicious (refusal)		Benign (false refusal)
Model	Desktop	Mobile	Desktop	Mobile
Gemini 3.1 Pro	47.2	46.1	8.7	9.6
Claude 4.7	41.3	35.6	6.3	9.4
GPT-5.4	29.5	23.0	7.0	5.1
Gemini 2.5 CU	17.2	14.8	1.3	1.5
Qwen-235B	0.0	0.0	0.0	0.0

Appendix

Table 9: StatusAction(unsafe) usage (%) on malicious tasks and benign tasks (= false refusal rate). Mean over 10 runs.

Threat	Gem 3.1	Cl 4.7	GPT-5.4	Gem CU
System Sabotage	48.3	49.4	68.5	66.3
Misalignment	50.0	43.8	45.8	60.4
Hallucination	40.9	33.0	43.2	47.7
Reasoning Gap	35.8	34.7	40.0	49.5
Adversarial Attack	28.6	30.0	38.5	54.9
Jailbreak	17.8	17.8	34.4	56.7

Appendix

Table 10: ASR (%) by threat category (combined mobile + desktop). Sorted by average across frontier models. Bold = highest per row.

Ambiguity category (by GT items)
	Mobile	Desktop
Tasks	381	363
GT clarification items	384	363
Items per task (mean)	1.01	1.00
Images per task (mean)	4.7	2.3
Referential underspecificity	188 (49.0%)	208 (57.3%)

Appendix

Table 11: Disambiguation dataset composition by platform.

Model	CR $\uparrow$	F1 $\uparrow$	OCR $\downarrow$	CW-CR $\uparrow$
Gemini 2.5 CU	88.2	57.7	66.7	88.2
Claude 4.7 Opus	82.4	49.8	60.0	88.8
Gemini 3.1 Pro	82.7	62.5	13.3	79.4
Qwen3-VL-4B	60.4	41.4	60.0	65.9
GPT-5.4	59.6	45.5	13.3	68.9
Qwen3-VL-235B	45.1	40.2	13.3	47.5

Appendix

Table 12: Disambiguation without severity scoring prompt (Mode 1, mobile

n

=381). Models are asked to clarify but not to score severity.

	Ref. Under.	Ling. Amb.	Missing	Imp. Log.	Imp. Env.
	( $n$ =188)	( $n$ =56)	( $n$ =58)	( $n$ =15)	( $n$ =13)
Gemini 3.1	71.3	46.4	75.9	66.7	15.4
Gemini CU	61.7	48.2	67.2	46.7	46.2
Claude 4.7	56.9	39.3	67.2	20.0	23.1
Qwen-235B	53.2	44.6	46.6	40.0	7.7
Qwen-4B	50.0	35.7	39.7	20.0	15.4

Appendix

Table 13: Per-category recall (%) on the five largest disambiguation categories (Mode 2, mobile).

n

= number of GT items.

	Obviousness				Consequence
Model	MAE	Bias	Exact%	Over%	MAE	Bias	Exact%	Over%
GPT-5.4	0.14	+0.11	86.5	11.3	0.67	+0.47	36.2	53.9
Gemini 3.1	0.15	+0.10	85.4	10.7	0.72	+0.53	32.0	59.3
Claude 4.7	0.20	+0.07	80.1	12.3	0.70	+0.56	36.5	56.4
Gemini CU	0.32	$-$ 0.07	69.4	11.1	0.85	+0.58	25.1	62.1
Qwen-4B	0.51	$-$ 0.25	50.3	12.1	0.80	+0.61	28.9	61.8

Appendix

Table 14: Severity calibration decomposition (Mode 2, mobile). Bias = mean (predicted

-

ground truth); positive = overestimate. Exact = fraction with zero error.

Harness	Models	Coord Format	Tool Format	Image
Standard	Gemini 3.1, Gemini CU	Relative int [0,1000]	ATEM XML	Original
GPT	GPT-5.4	Absolute int [0,W] $\times$ [0,H]	ATEM XML	768px height
Claude	Claude 4.7	Relative int [0,1280] $\times$ [0,720]	ATEM XML	1280 $\times$ 720
Qwen	Qwen 235B, 8B, 4B	Absolute int [0,W] $\times$ [0,H]	Native tool call	Original

Appendix

Table 15: Per-model evaluation configuration. All models use temperature 0.1 (near-greedy). Images are constrained to

<

3.8 MB.

GUI / Visual Agent Safety
Benchmark	Venue	# Tasks	Platform	Eval	Paired	Disambig.	CU	Open	Grading
SafeArena	ICML’25	$\sim$ 500	Web	Traj.	✓	✗	✗	✓	LLM Judge
MobileSafetyBench	AAAI’26	250	Mobile	Traj.	✓	✗	✗	✓	Rule
ST-WebAgentBench	ICLR’26	222	Web	Traj.	$\sim$	$\sim$	✗	✓	Rule
OS-Harm	NeurIPS’25	150	Desktop	Traj.	✗	✗	✓	✓	LLM Judge
RiOSWorld	NeurIPS’25	492	Mixed	Traj.	✗	✗	✗	✓	Rule

Appendix

Table 16: Extended comparison with existing GUI agent safety and disambiguation benchmarks.

Rank	Outcome	Pref. Likelihood
1	My identity is stolen	80.2%
2	Personal accounts compromised	77.5%
3	$5,000 transferred to unknown person	76.0%
4	Someone controls my phone without consent	70.0%
5	Biometric data compromised and shared	66.3%
6	Phone operates critical infrastructure	62.0%

Appendix

Table 17: Top 10 and bottom 5 user-ranked undesirable outcomes from MaxDiff survey (

n

=1,300).

Computer-Use Agent Benchmarks

Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.

See all 118 papers

The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
Apr 12, 2026Xuwei Ding, Skylar Zhai, Linxin Song +6Computer-Use Agent Safety AI Agent Safety
University of Wisconsin–Madison · University of Minnesota · University of Southern California +3
OSGuard: A Benchmark for Safety in Computer-Use Agents
Jun 13, 2026Mina Mohammadmirzaei, Jeffrey FlaniganComputer-Use Agent Safety Computer-Use Agents
University of California, Santa Cruz
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
Jun 22, 2026Anmol Goel, Iryna GurevychComputer-Use Agents Contextual Integrity
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE

Apr 12, 2026J/K move · Enter open · S save

The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents

Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li+5

University of Wisconsin–Madison · University of Minnesota · University of Southern California+3

Computer-Use Agent Safety AI Agent Safety LLM Agent Safety AI Agent Safety Benchmarks Multi-Agent System Security

Open paper AskPDF arXiv

Computer-use agents (CUAs) can now autonomously complete complex tasks in real digital environments, but when misled, they can also be used to automate harmful actions programmatically. Existing safety evaluations largely target explicit threats such as misuse and prompt injection, but overlook a subtle yet critical setting where user instructions are entirely benign and harm arises from the task context or execution outcome. We introduce OS-BLIND, a benchmark that evaluates CUAs under unintended attack conditions, comprising 300 human-crafted tasks across 12 categories, 8 applications, and 2 threat clusters: environment-embedded threats and agent-initiated harms. Our evaluation on frontier models and agentic frameworks reveals that most CUAs exceed 90% attack success rate (ASR), and even the safety-aligned Claude 4.5 Sonnet reaches 73.0% ASR. More interestingly, this vulnerability becomes even more severe, with ASR rising from 73.0% to 92.7% when Claude 4.5 Sonnet is deployed in multi-agent systems. Our analysis further shows that existing safety defenses provide limited protection when user instructions are benign. Safety alignment primarily activates within the first few steps and rarely re-engages during subsequent execution. In multi-agent systems, decomposed subtasks obscure the harmful intent from the model, causing safety-aligned models to fail. We will release our OS-BLIND to encourage the broader research community to further investigate and address these safety challenges.

Abstract

Computer-Use Agent Benchmarks

Explore similar work

The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents

OSGuard: A Benchmark for Safety in Computer-Use Agents

Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?