Computer-Use Agent Evaluation

Latest papers

41 papers13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Jun 8, 2026Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom +3Personal AI Agents Mobile GUI Agents
Carnegie Mellon University
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Jun 8, 2026Wanli Li, Bowen Zhou, Yunyao Yu +4Agent Benchmarks Computer-Use Agents
Microsoft Research Asia · Tsinghua University
RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Agent Benchmarks AI Coding Agents
1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University College London · Nanjing University · Tencent
Computer Use at the Edge of the Statistical Precipice
May 7, 2026Pierluca D'Oro, Sneha Silwal, William Wong +6Computer-Use Agent Benchmarks Computer-Use Agents
Meta Superintelligence Labs
CocoaBench: Evaluating Unified Digital Agents in the Wild
Apr 13, 2026CocoaBench Team, Shibo Hao, Zhining Zhang +29AI Agent Benchmarks Computer-Use Agent Evaluation
Gym-Anything: Turn any Software into an Agent Environment
Apr 7, 2026Pranjal Aggarwal, Graham Neubig, Sean WelleckComputer-Use Agents Computer-Use Agent Benchmarks
CMU
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Apr 6, 2026Rongqian Chen, Yu Li, Zeyu Fang +3Computer-Use Agent Evaluation Reward Modeling
George Washington University, Washington D.C., USA
CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
Mar 11, 2026Marta Sumyk, Oleksandr KosovanComputer-Use Agent Evaluation AI Agent Auditing
Ukrainian Catholic University Lviv, Ukraine
Video-Based Reward Modeling for Computer-Use Agents
Mar 10, 2026Linxin Song, Jieyu Zhang, Huanxin Sheng +6Computer-Use Agent Evaluation Reward Modeling
University of Southern California · University of Washington · MBZUAI +1
General Agent Evaluation
Feb 26, 2026Elron Bandel, Asaf Yehudai, Lilach Eden +12Agent Evaluation Automated Agent Evaluation
IBM Research · MIT

Jun 8, 2026J/K move · Enter open · S save

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang+2

Carnegie Mellon University

Personal AI Agents Mobile GUI Agents User Modeling Computer-Use Agent Benchmarks Computer-Use Agents AI Agent Benchmarks+3

Open paper AskPDF arXiv

A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existing mobile agent benchmarks lack this kind of personalization. We introduce iOSWorld, the first interactive native iOS simulator benchmark built around a persistent user identity spanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-source computer-use models in both vision-only and privileged vision+XML settings. The best configuration reaches 52% overall but only 37% on multi-app tasks. Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code.