Computer-using agents (CUAs) act directly on graphical user interfaces, yet their perception of the screen is often unreliable. Existing work largely treats these failures as performance limitations, asking whether an action succeeds, rather than whether the agent is acting on the correct object at all. We argue that this is fundamentally a security problem. We formalize the visual confused deputy: a failure mode in which an agent authorizes an action based on a misperceived screen state, due to grounding errors, adversarial screenshot manipulation, or time-of-check-to-time-of-use (TOCTOU) races. This gap is practically exploitable: even simple screen-level manipulations can redirect routine clicks into privileged actions while remaining indistinguishable from ordinary agent mistakes. To mitigate this threat, we propose the first guardrail that operates outside the agent's perceptual loop. Our method, dual-channel contrastive classification, independently evaluates (1) the visual click target and (2) the agent's reasoning about the action against deployment-specific knowledge bases, and blocks execution if either channel indicates risk. The key insight is that these two channels capture complementary failure modes: visual evidence detects target-level mismatches, while textual reasoning reveals dangerous intent behind visually innocuous controls. Across controlled attacks, real GUI screenshots, and agent traces, the combined guardrail consistently outperforms either channel alone. Our results suggest that CUA safety requires not only better action generation, but independent verification of what the agent believes it is clicking and why. Materials are provided\footnote{Model, benchmark, and code: https://github.com/vllm-project/semantic-router}.
Figures & tables
Figure 1: The ScreenSwap attack. The compromised agent crops and swaps button pixel regions before forwarding the screenshot to the LLM. The LLM clicks what it perceives as “Acknowledge Alert,” but the actual click hits “Admin: Reset Credentials.”
Figure 2: Dual-channel contrastive classification. The image channel classifies the click-target crop; the text channel classifies the LLM’s reasoning. Block if either flags danger.
Configuration
Recall
Specif.
F1
Acc.
Image + text veto
87.4%
88.8%
.915
87.8%
Image cross-app (max)
83.1%
88.8%
.889
84.5%
Image per-app (max)
79.7%
81.9%
.859
80.3%
Dual agree (both must match)
69.9%
71.5%
.779
70.3%
Table 1: ScreenSpot-Pro evaluation (260 screenshots, 1040 test cases/config). Image+text veto is the optimal fusion. Details in Appendix E .
Channel
Recall
Specif.
F1
Acc.
Image-only
0%
100%
.000
50%
Text-only
100%
100%
1.00
100%
Dual (OR)
100%
100%
1.00
100%
Dual (AND)
0%
100%
.000
50%
Table 2: Neutral-button evaluation (20 scenarios). Buttons are visually identical; only reasoning text distinguishes danger.
Figure 3: Complementary threat surfaces. OR-logic covers both danger-styled buttons (image channel) and neutral buttons with dangerous intent (text channel).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
KB Strategy
Acc.
Min ∣Δ∣
Avg ∣Δ∣
SigLIP (768d)
Image-only
4/4
0.1126
0.2088
Ours (384d)
Image-only
4/4
0.1779
0.3035
Ours (384d)
Text-only
2/4
0.0012
0.0354
Ours (384d)
Cross-modal
3/4
0.0852
0.1340
Appendix
Table 3: Single-modality contrastive KB results. ∣Δ∣ = margin between restricted and permitted KB scores.
Scenario
Text
Image
Dual (OR)
Img-only
Correct click (Ack Alert)
NORMAL
NORMAL
ALLOWED
ALLOWED
Correct click (Escalate)
NORMAL
NORMAL
ALLOWED
ALLOWED
Correct click (Isolate)
NORMAL
NORMAL
ALLOWED
ALLOWED
Says Ack, hits Admin
NORMAL
ADMIN
BLOCKED
BLOCKED
Says Escalate, hits Admin
NORMAL
ADMIN
BLOCKED
BLOCKED
Explicit admin intent
ADMIN
ADMIN
BLOCKED
BLOCKED
Appendix
Table 4: Channel comparison on 8 SOC dashboard scenarios. All buttons are danger-styled; image channel catches all threats.
Config
Text KB
Img
Agg
FP
TN
Spec.
F1
Original
10+10
5 apps
mean
3
3
50%
.824
Expanded text
20+20
5 apps
mean
5
1
17%
.737
Expanded text + image
20+20
7 apps
mean
5
1
17%
.737
Expanded + margin
20+20
7 apps
mean
3
3
50%
.824
Original + max
10+10
5
max
2
4
67%
.875
Expanded + max
20+20
5 apps
max
3
3
50%
.824
Appendix
Table 5: KB ablation study on 13 OS-Harm scenarios. Recall is 100% (7/7) for all configurations; the key variable is specificity (safe actions correctly allowed).
Figure 4: Representative samples of synthetically generated 100×100 button crops used in the visual knowledge base ( Kimg± ). Safe actions (top row) use success, accent, or neutral styling. Dangerous actions (bottom row) use explicit danger or warning styling.
Model
Type
Params
Acc.
Generalist VLMs
GPT-4o
generalist
∼ 1.8T?
0.8%
Qwen2-VL-72B
generalist
72B
1.0%
Qwen2-VL-7B
generalist
7B
1.6%
GUI grounding specialists
ShowUI
specialist
2B
7.7%
Appendix
Table 7: GUI grounding accuracy on ScreenSpot-Pro by model size. Specialist models are fine-tuned for GUI grounding; generalists are not. Data from Li et al. (2025b) ; Bai et al. (2025) .
Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguous instructions. We introduce ADeptS-Bench, a dual-stream trustworthiness benchmark, grounded in the ADEPTS capability framework and general population user studies. The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface. The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous. Evaluating seven models reveals that no model consistently exceeds 80% task success while staying below 30% attack success; every model clicks "Checkout" on a $25K order without hesitation, and none detects that a "factory reset" button is mislabeled as "Optimize." An ablation reveals three distinct safety architectures: tool-dependent (ASR +21-23pp without refusal tool), partially tool-dependent (+10-11pp), and no mechanism (unchanged). In disambiguation, all models overestimate consequence severity, mirroring the over-refusal bias observed in safety. We release all data, evaluation code, and analysis tools upon publication.
Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro +3
Computer-use agents (CUAs) can now autonomously complete complex tasks in real digital environments, but when misled, they can also be used to automate harmful actions programmatically. Existing safety evaluations largely target explicit threats such as misuse and prompt injection, but overlook a subtle yet critical setting where user instructions are entirely benign and harm arises from the task context or execution outcome. We introduce OS-BLIND, a benchmark that evaluates CUAs under unintended attack conditions, comprising 300 human-crafted tasks across 12 categories, 8 applications, and 2 threat clusters: environment-embedded threats and agent-initiated harms. Our evaluation on frontier models and agentic frameworks reveals that most CUAs exceed 90% attack success rate (ASR), and even the safety-aligned Claude 4.5 Sonnet reaches 73.0% ASR. More interestingly, this vulnerability becomes even more severe, with ASR rising from 73.0% to 92.7% when Claude 4.5 Sonnet is deployed in multi-agent systems. Our analysis further shows that existing safety defenses provide limited protection when user instructions are benign. Safety alignment primarily activates within the first few steps and rarely re-engages during subsequent execution. In multi-agent systems, decomposed subtasks obscure the harmful intent from the model, causing safety-aligned models to fail. We will release our OS-BLIND to encourage the broader research community to further investigate and address these safety challenges.
Xuwei Ding, Skylar Zhai, Linxin Song +6
University of Wisconsin–Madison · University of Minnesota · University of Southern California +3
Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all existing baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under adversarial settings while preserving or even improving task success rate in benign environments.