Computer-Use Agents

Follow this topic Ask about this topic

Computer-use agents study how AI systems can interpret screens or structured interface data and act through clicks, typing, gestures, and software tools. Common approaches combine vision-language models with action planning, feedback and verification, reusable skills or memory, and training through demonstrations or reinforcement learning in real or synthetic environments. Benchmarks increasingly test long-horizon, cross-application work and measure task outcomes alongside efficiency, reliability, and safety. Agents can handle many routine interactions, but complex workflows remain difficult, and performance depends strongly on the interface, task design, and safeguards against errors or misuse.

Latest papers

161 papers32 papers in the last four weeks, up 129% on the four weeks before. 0.2% of all new papers.

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Jul 30, 2026Qiushi Sun, Kanzhi Cheng, Yian Wang +20VLM Evaluation Computer-Use Agent Benchmarks
The University of Hong Kong · Nanjing University · National University of Singapore +4
Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
Jul 30, 2026Woongkyu Lee, Jungwook ChoiComputer-Use Agents Inference-Time Scaling
Hanyang University
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
Jul 30, 2026Zihan Dong, Rui Qian, Qishi Zhan +3Computer-Use Agents GUI Agents
Georgia Institute of Technology · Fudan University · Marquette University +3
How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
Jul 30, 2026Hanzhang Zhou, Panrong Tong, Xu Zhang +13Mobile GUI Automation Computer-Use Agents
MAI-UI Team Alibaba Token Hub, Alibaba Group
Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale
Jul 30, 2026Yash Pandya, Sahil Gupta, Sarthak Harne +10Computer-Use Agent Benchmarks Computer-Use Agents
Microsoft Research · Work done while at Microsoft Research.
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
Jul 29, 2026Jingbo Zhou, Yusai Zhao, Qi Bao +12Computer-Use Agent Benchmarks Computer-Use Agents
Agent Frontier Team, Large Model Frontier Research Department, Baidu Inc., China.
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
Jul 28, 2026Abhishek Pillai, Samir Kumar Nayak, Yuan ChenComputer-Use Agent Benchmarks Computer-Use Agents
NVIDIA Austin, TX, USA · NVIDIA Bangalore, Karnataka, IND
SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
Jul 25, 2026Yang Wan, Zhenhao Zhang, Jierui Wang +1Reward Modeling LLM-as-a-Judge
1Zhejiang University · University of Rochester
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Jul 24, 2026Yan Yang, Xiangru Jian, Ziyang Luo +7Computer-Use Agents Tool-Using Agents
Salesforce AI Research
OpenForgeRL: Train Harness-native Agents in Any Environment
Jul 23, 2026Xiao Yu, Baolin Peng, Ruize Xu +7Computer-Use Agents LLM Agent Training
Columbia University · Microsoft Research · Dartmouth College
RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control
Jul 20, 2026Di Lu, Bo Zhang, Xiyuan Li +5Aerial Robotics Safety-Critical Control
School of Cyber Engineering, Shaanxi Key Lab of Network and System Security, Xidian University, Xi’an, Shaanxi 710071, China. · College of Cyber Security, Jinan University, Guangzhou 510632, China.
Teach it to stop, not just to click
Jul 19, 2026Barada Sahu, Shivesh PandeyAI Agent Reliability Agent Reliability
Cabal AI · Para AI
Tactile: Giving Computer-Using Agents Hands and Feet
Jul 16, 2026Yong Liu, Zhenyi Zhong, Zhanpeng ShiComputer-Use Agents GUI Agents
Shanghai Jiao Tong University
StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure
Jul 13, 2026Wenyi Wu, Sibo Zhu, Kun Zhou +3Computer-Use Agents Long-Horizon LLM Agents
University of California, San Diego · 2Aether AI Lab · §Work done during an internship at Aether AI.
SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL
Jul 13, 2026Bowen Lv, Xiao Liu, Yanyu Ren +7Computer-Use Agents Reinforcement Learning with Verifiable Rewards
Tsinghua University
MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
Jul 11, 2026Chengguang Gan, Hanjun Wei, Yunhao Liang +3Computer-Use Agents Web Agent Benchmarks
Independent Researcher · University of Chinese Academy of Sciences · Monash University +2
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Jul 7, 2026Mianqiu Huang, Taofeng Xue, Chong Peng +12RL for GUI Agents Partially Observable RL
1Meituan · 2Fudan University · 3Shanghai Jiao Tong University +1
DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching
Jun 30, 2026Meng Chen, Anya Ji, Tsung-Han Wu +4Computer-Use Agents AI in Education
University of California, Berkeley · 2Technical University of Munich
Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents
Jun 30, 2026Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt +2Computer-Use Agents Self-Improving Agents
1Stanford University, Stanford, CA 94305, USA
PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks
Jun 30, 2026Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar +6Computer-Use Agents Rubric-Based Evaluation
Carnegie Mellon University · Microsoft · UMass Amherst +3
LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents
Jun 29, 2026Yogeswar Reddy ThotaComputer-Use Agents GUI Automation
Department of Electrical and Computer Engineering University of Texas at Dallas Richardson, TX, USA
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Jun 28, 2026Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33Computer-Use Agent Benchmarks Computer-Use Agents
XLANG Lab
Agent-Computer Observation Interfaces Enable Dynamic Computer Use
Jun 28, 2026Bojie Li, Noah ShiAudio-Visual Understanding Computer-Use Agents
Pine AI · University of Washington
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
Jun 26, 2026Shoufa Chen, Luyuan Wang, Xuan Yang +7Computer-Use Agent Benchmarks Computer-Use Agents
Meta AI · Duke University · Stanford University
It Lied to a Doctor to Buy Poison Ingredients: Quantifying Real-World Misuse of Phone-use Agents
Jun 26, 2026Yiming Sun, Chen Chen, Zifan Zhou +1Computer-Use Agents Mobile GUI Agents
Fudan University, JADE (Whitzard AI) Team
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Jun 24, 2026Divake Kumar, Sina Tayebati, Devashri Naik +5Computer-Use Agent Benchmarks Uncertainty Quantification
University of Illinois Chicago · Intel Labs · Capital One AI Labs
GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
Jun 22, 2026Xiao Zhou, Siyue Zhang, Yilun Zhao +4Computer-Use Agent Benchmarks Computer-Use Agents
NNYU Shanghai · Y Yale NLP Lab
AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction
Jun 22, 2026Shanhui Zhao, Jiacheng Liu, Guohong Liu +13Computer-Use Agents AI Agent Security
Tsinghua University · Peking University · The University of Hong Kong
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
Jun 22, 2026Anmol Goel, Iryna GurevychComputer-Use Agents Contextual Integrity
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE

Jul 30, 2026J/K move · Enter open · S save

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang+19

The University of Hong Kong · Nanjing University · National University of Singapore+4

VLM Evaluation Computer-Use Agent Benchmarks Reward Modeling Computer-Use Agents Multimodal Reward Modeling Computer-Use Agent Evaluation+1

Open paper AskPDF arXiv

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60x lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.