Computer-Use Agents

Follow this topic Ask about this topic

Computer-use agents study how AI systems can interpret screens or structured interface data and act through clicks, typing, gestures, and software tools. Common approaches combine vision-language models with action planning, feedback and verification, reusable skills or memory, and training through demonstrations or reinforcement learning in real or synthetic environments. Benchmarks increasingly test long-horizon, cross-application work and measure task outcomes alongside efficiency, reliability, and safety. Agents can handle many routine interactions, but complex workflows remain difficult, and performance depends strongly on the interface, task design, and safeguards against errors or misuse.

Latest papers

161 papers32 papers in the last four weeks, up 129% on the four weeks before. 0.2% of all new papers.

PhoneBuddy: Training Open Models for Agentic Phone Use
Jun 22, 2026Zhengyang Tang, Xin Lai, Pengyuan Lyu +23Computer-Use Agents Mobile GUI Agents
Tencent Hunyuan · Gaoling School of Artificial Intelligence, Renmin University of China · The Chinese University of Hong Kong, Shenzhen +1
Agent MechSuits: Mechanistic Subspace Safety Steering for Multi-Turn CLI Agents
Jun 21, 2026Weidi Luo, Qiming Zhang, Yihao Quan +5Computer-Use Agents LLM Interpretability
University of Georgia · University of South Florida · Rutgers University +3
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Jun 21, 2026Yikun Fu, Bowen Fu, Zhenyu Wu +10AI Agent Benchmarks Computer-Use Agents
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Xi’an Jiaotong University +5
ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Jun 19, 2026Vincent Siu, Manasi Sharma, Dawn Song +2Computer-Use Agent Benchmarks Computer-Use Agents
1Scale AI · University of California, Santa Cruz · *Work done during an internship at Scale AI. +1
Fara-1.5: Scalable Learning Environments for Computer Use Agents
Jun 18, 2026Ahmed Awadallah, Sahil Gupta, Yash Lara +12Computer-Use Agents LLM Agent Training
Beyond Global Replanning: Hierarchical Recovery for Cross-Device Agent Systems
Jun 18, 2026Shu Yao, Yuhua Luo, Qian Long +7Hierarchical Planning Multi-Agent Orchestration
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Shanghai Innovation Institute · 3Southeast University +1
Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Jun 18, 2026Yuexing Hao, Xiaomin LiAgentic RL Computer-Use Agents
Massachusetts Institute of Technology · Harvard University
VISUALSKILL: Multimodal Skills for Computer-Use Agents
Jun 16, 2026Ziyan Jiang, Li An, Yujian Liu +5Computer-Use Agents LLM Agent Skill Learning
1UC Santa Barbara · 2MIT CSAIL · 3MIT-IBM Watson AI Lab
PreAct: Computer-Using Agents that Get Faster on Repeated Tasks
Jun 16, 2026Bojie LiComputer-Use Agents LLM Agent Skill Learning
Pine AI
ProCUA-SFT Technical Report
Jun 15, 2026Jaehun Jung, Ximing Lu, Brandon Cui +11Supervised Fine-Tuning Computer-Use Agents
1NVIDIA · University of Washington
LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
Jun 15, 2026Anqi Zou, Han Deng, Chengyu Zhang +9Computer-Use Agent Benchmarks Computer-Use Agents
Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong +1
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
Jun 15, 2026Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh +1Personal AI Agents Tool-Augmented Language Model Agents
Carnegie Mellon University
OSGuard: A Benchmark for Safety in Computer-Use Agents
Jun 13, 2026Mina Mohammadmirzaei, Jeffrey FlaniganComputer-Use Agent Safety Computer-Use Agents
University of California, Santa Cruz
ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm
Jun 11, 2026Jiaxin Ai, Tao Hu, Xuemeng Yang +11Computer-Use Agents Program Synthesis
Wuhan University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +4
A History-Aware Visually Grounded Critic for Computer Use Agents
Jun 9, 2026Jaewoo Lee, Zaid Khan, Archiki Prasad +7Computer-Use Agents Long-Horizon Agent Tasks
University of North Carolina at Chapel Hill1 · Capital One2 · University of Texas at Austin3
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Jun 8, 2026Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom +3Personal AI Agents Mobile GUI Agents
Carnegie Mellon University
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Jun 8, 2026Wanli Li, Bowen Zhou, Yunyao Yu +4Agent Benchmarks Computer-Use Agents
Microsoft Research Asia · Tsinghua University
How AI Agents Reshape Knowledge Work: Autonomy, Efficiency, and Scope
Jun 5, 2026Jeremy Yang, Kate Zyskowski, Noah Yonack +1Computer-Use Agents Web Search Agents
Harvard University · Perplexity
MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
Jun 4, 2026Victor Muryn, Maksym Shamrai, Sofiia Mazepa +1Computer-Use Agent Benchmarks Computer-Use Agents
1MacPaw Research, Kyiv, Ukraine
Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming
Jun 3, 2026Nicholas SabanComputer-Use Agents AI Agent Safety
Patronus AI · University of California, Berkeley
SkillHarness: Harnessing Safe Skills for Computer-Use Agents
Jun 2, 2026Yurun Chen, Biao Yi, Keting Yin +1Computer-Use Agents Continual Learning
Zhejiang University
MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
Jun 2, 2026Jia Yu, Zilong Wang, Xinyang Jiang +2Computer-Use Agent Benchmarks LLM Agent Evaluation
Microsoft Research Asia, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China
Multi-Agent Computer Use
Jun 1, 2026Jing Yu Koh, Ruslan Salakhutdinov, Daniel FriedMulti-Agent Coordination Computer-Use Agents
Carnegie Mellon University
ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents
May 29, 2026Jeremy Tien, Abishek Anand, Yu-Rou Tuan +3Agent Reliability Computer-Use Agents
Carnegie Mellon University
PRO-CUA: Process-Reward Optimization for Computer Use Agents
May 27, 2026Yifei He, Rui Yang, Hao Bai +2Reinforcement Learning Process Reward Models
University of Illinois Urbana-Champaign
Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents
May 27, 2026Suji Kim, Kangsan Kim, Sung Ju HwangComputer-Use Agents Synthetic Data Generation
KAIST · Samsung Electronics
AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
May 25, 2026Jingwei Sun, Jianing Zhu, Yuanyi Li +3Computer-Use Agent Benchmarks Computer-Use Agents
TMLR Group, Hong Kong Baptist University · The University of Texas at Austin · Sydney AI Centre, The University of Sydney +1
CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents
May 25, 2026Bowen Wang, Dunjie Lu, Junli Wang +11Computer-Use Agents Synthetic Data Generation
The University of Hong Kong · Qwen Team, Alibaba Inc. · University of California, San Diego +1
ECHO: Terminal Agents Learn World Models for Free
May 23, 2026Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah +1World Model Learning Computer-Use Agents
Microsoft Research
Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling
May 20, 2026Caleb Winston, Ron Yifeng Wang, Azalia Mirhoseini +1Computer-Use Agents LLM Agent Orchestration
Stanford University, Stanford, CA, USA.

Jun 22, 2026J/K move · Enter open · S save

PhoneBuddy: Training Open Models for Agentic Phone Use

Zhengyang Tang, Xin Lai, Pengyuan Lyu, Xinyuan Wang+22

Tencent Hunyuan · Gaoling School of Artificial Intelligence, Renmin University of China · The Chinese University of Hong Kong, Shenzhen+1

Computer-Use Agents Mobile GUI Agents LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67% after supervised fine-tuning to 40.67% after real-app RL and 45.33% after mixed RL. On AndroidWorld, the same progression rises from 60.3% to 77.2% to 83.2%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.