Computer-Use Agents

Follow this topic Ask about this topic

Computer-use agents study how AI systems can interpret screens or structured interface data and act through clicks, typing, gestures, and software tools. Common approaches combine vision-language models with action planning, feedback and verification, reusable skills or memory, and training through demonstrations or reinforcement learning in real or synthetic environments. Benchmarks increasingly test long-horizon, cross-application work and measure task outcomes alongside efficiency, reliability, and safety. Agents can handle many routine interactions, but complex workflows remain difficult, and performance depends strongly on the interface, task design, and safeguards against errors or misuse.

Latest papers

160 papers31 papers in the last four weeks, up 121% on the four weeks before. 0.2% of all new papers.

Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents
Mar 16, 2026Xunzhuo Liu, Bowei He, Xue Liu +3Computer-Use Agent Safety AI Agent Security
vLLM Semantic Router Project · MBZUAI · McGill University +2
Adaptive Vision-Language Model Routing for Computer Use Agents
Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Agent Routing Computer-Use Agents
vLLM Semantic Router Project · MBZUAI · McGill University +2
CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
Mar 11, 2026Marta Sumyk, Oleksandr KosovanComputer-Use Agent Evaluation AI Agent Auditing
Ukrainian Catholic University Lviv, Ukraine
Video-Based Reward Modeling for Computer-Use Agents
Mar 10, 2026Linxin Song, Jieyu Zhang, Huanxin Sheng +6Computer-Use Agent Evaluation Reward Modeling
University of Southern California · University of Washington · MBZUAI +1
OSExpert: Computer-Use Agents Learning Professional Skills via Exploration
Mar 9, 2026Jiateng Liu, Zhenhailong Wang, Rushi Wang +6Computer-Use Agents Agent Skill Learning
University of Illinois Urbana-Champaign · Stevens Institute of Technology
Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding
Feb 25, 2026Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad +2Computer-Use Agents Web Agents
University College London London, United Kingdom Work done at UCL Nexus Labs.
IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents
Feb 19, 2026Seoyoung Lee, Seobin Yoon, Seongbeen Lee +4Computer-Use Agents Long-Horizon Agent Tasks
Sookmyung Women’s University, Seoul, Republic of Korea
When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
Feb 9, 2026Yuting Ning, Jaylen Jones, Zhehao Zhang +5LLM Self-Correction Computer-Use Agents
The Ohio State University · Amazon AGI
Measuring Harmfulness of Computer-Using Agents
Jul 31, 2025Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang +3LLM Agent Safety Computer-Use Agent Benchmarks
Independent Researcher · Arizona State University · Gradient +1
AppAgent: Multimodal Agents as Smartphone Users
Dec 21, 2023Chi Zhang, Zhao Yang, Jiaxuan Liu +6Computer-Use Agents LLM Agent Skill Learning
Tencent

Mar 16, 2026J/K move · Enter open · S save

Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents

Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo+2

vLLM Semantic Router Project · MBZUAI · McGill University+2

Computer-Use Agent Safety AI Agent Security Computer-Use Agents Software Security Runtime Enforcement for AI Agents Adversarial Attacks

Open paper AskPDF arXiv

Computer-using agents (CUAs) act directly on graphical user interfaces, yet their perception of the screen is often unreliable. Existing work largely treats these failures as performance limitations, asking whether an action succeeds, rather than whether the agent is acting on the correct object at all. We argue that this is fundamentally a security problem. We formalize the visual confused deputy: a failure mode in which an agent authorizes an action based on a misperceived screen state, due to grounding errors, adversarial screenshot manipulation, or time-of-check-to-time-of-use (TOCTOU) races. This gap is practically exploitable: even simple screen-level manipulations can redirect routine clicks into privileged actions while remaining indistinguishable from ordinary agent mistakes. To mitigate this threat, we propose the first guardrail that operates outside the agent's perceptual loop. Our method, dual-channel contrastive classification, independently evaluates (1) the visual click target and (2) the agent's reasoning about the action against deployment-specific knowledge bases, and blocks execution if either channel indicates risk. The key insight is that these two channels capture complementary failure modes: visual evidence detects target-level mismatches, while textual reasoning reveals dangerous intent behind visually innocuous controls. Across controlled attacks, real GUI screenshots, and agent traces, the combined guardrail consistently outperforms either channel alone. Our results suggest that CUA safety requires not only better action generation, but independent verification of what the agent believes it is clicking and why. Materials are provided\footnote{Model, benchmark, and code: https://github.com/vllm-project/semantic-router}.