Computer-Use Agent Evaluation

Latest papers

41 papers13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials
Nov 2, 2026Anh Nguyen, Tianyi Xu, Phat Tran +3Computer-Use Agents World Model-Based Planning
Oregon State University
Performance, Scalability and Maintenance: An Experimental Study of RPA vs. Autonomous Agents in the Healthcare Domain
Nov 2, 2026Daniel Schulz, Pedro Cabral Buchaim, Bernardo Soares +4Business Process Automation Computer-Use Agent Evaluation
Department of Engineering, Computer Science and Architecture, University of Santa Cruz do Sul (UNISC), Santa Cruz do Sul, Rio Grande do Sul, Brazil. · Graduate Program in Industrial Systems and Processes (PPGSPI), University of Santa Cruz do Sul (UNISC), Santa Cruz do Sul, Rio Grande do Sul, Brazil. · Clínica de Mastologia Dra. Ana Paula Muller Ltda., Porto Alegre, Rio Grande do Sul, Brazil.
Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents
Oct 30, 2026Haoqing Wang, Xiaoyi Zhang, Sicheng Fan +5Computer-Use Agents Human-Computer Interaction
Microsoft Research · Fudan University · Tsinghua University
VeriCUA: Verification as Agentic Exploration for Computer-Using Agents
Oct 29, 2026Qiushi Sun, Da Yu, Alexander Knop +5Computer-Use Agent Evaluation Automated Agent Evaluation
Google · The University of Hong Kong
EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation
Oct 29, 2026Ankush Kadu, Athira Gopal, Niharika Singh +2Computer-Use Agent Evaluation AI Agent Auditing
QpiAI, Bangalore, India
What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click
Oct 29, 2026Vasundra SrinivasanComputer-Use Agents Computer-Use Agent Evaluation
Principal AI Engineer, Author—Data Engineering for Multimodal AI (O’Reilly), Stanford School of Engineering
CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments
Oct 27, 2026Kianté Fernandez, Caitlin Chen, Jialu Zhou +3Multimodal Agent Evaluation Human Behavior Simulation
Department of Psychology, University of California, Los Angeles
iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents
Oct 26, 2026Yuting Ning, Ruijia Cheng, Sushobhan Ghosh +7Personal AI Agents Mobile Agent Benchmarks
The Ohio State University · Apple
Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing
Oct 24, 2026Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei +4LLM Agent Harnesses Automated Software Testing
Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden · Volvo Group, Gothenburg, Sweden
AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack
Oct 22, 2026Justin Tan, Miao Li, Ding ZhaoLLM Agent Security AI Agent Security Benchmarks
Carnegie Mellon University
Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
Oct 20, 2026Byeonghun Bang, Sangwoo Park, Jinheon Baek +3Computer-Use Agents Computer-Use Agent Evaluation
KAIST · ETRI
CUA-Lite: An Open Platform for Computer Use Agents
Oct 19, 2026Zhanhui Zhou, Weichen Zhang, Haoran Liu +6Computer-Use Agents Computer-Use Agent Benchmarks
TOMORI: A Tool-Orchestrated Mixing Operator for REAPER Interaction
Oct 18, 2026Yining Wang, Tianjiao LiTool-Using Agents Audio Editing
Beyond Next-Screen Prediction: The Decision Value of GUI World Models
Oct 17, 2026Sizhe Tang, Hanhan Zhou, Jeffrey Svajlenko +1World Model Evaluation Computer-Use Agent Evaluation
George Washington University · Amazon Web Services
PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent Evaluation Computer-Use Agent Benchmarks
The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14
DelegationBench: Measuring When AI Agents Should Ask Before Acting
Oct 4, 2026Shiva PochampallyLLM Agent Evaluation AI Agent Benchmarks
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
Sep 24, 2026Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen +6Computer-Use Agent Evaluation Computer-Use Agents
University of Utah
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Sep 24, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Tool-Augmented Language Model Agents LLM Agent Evaluation
Eon
ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
Aug 25, 2026Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro +3Computer-Use Agent Benchmarks Computer-Use Agent Evaluation
FAIR at Meta
Qwen-CUA: Native Computer Use for (almost) Everything
Aug 3, 2026Dunjie Lu, Shuai Bai, Tianyi Bai +43Computer-Use Agent Benchmarks Computer-Use Agents
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Jul 30, 2026Qiushi Sun, Kanzhi Cheng, Yian Wang +20VLM Evaluation Computer-Use Agent Benchmarks
The University of Hong Kong · Nanjing University · National University of Singapore +4
How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
Jul 25, 2026Yang Wan, Zhenhao Zhang, Jierui Wang +1Reward Modeling LLM-as-a-Judge
1Zhejiang University · University of Rochester
Copy-on-Write Scoring: Application-Specific Agent Evaluations
Jul 15, 2026Joanna Roy, Sven HoelzelTool-Use Evaluation LLM Agent Evaluation
trail-ml, Munich, Germany
PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks
Jun 30, 2026Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar +6Computer-Use Agents Rubric-Based Evaluation
Carnegie Mellon University · Microsoft · UMass Amherst +3
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Jun 28, 2026Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33Computer-Use Agent Benchmarks Computer-Use Agents
XLANG Lab
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5LLM Agent Evaluation Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Jun 21, 2026Yikun Fu, Bowen Fu, Zhenyu Wu +10AI Agent Benchmarks Computer-Use Agents
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Xi’an Jiaotong University +5
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
Jun 15, 2026Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh +1Personal AI Agents Tool-Augmented Language Model Agents
Carnegie Mellon University

Nov 2, 2026J/K move · Enter open · S save

Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials

Anh Nguyen, Tianyi Xu, Phat Tran, Harry Yu+2

Oregon State University

Computer-Use Agents World Model-Based Planning Computer-Use Agent Evaluation World Model Evaluation Computer-Use Agent Benchmarks

Open paper AskPDF arXiv

Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges. However, the contribution of each component -- tutorials, the judge, and the world model -- to overall improvement has not been evaluated in isolation. To fill this gap, we first construct OSWorld-Tutorial and ScienceBoard-Tutorial: two datasets for CUAs with curated multimodal tutorials, covering everyday and scientific domains. We then propose RAG-PolicyJudge (RAG-PJ), a policy-judge framework for CUAs conditioned on task-specific tutorials: the policy proposes candidate actions, and the judge selects among them. To isolate the effect of world models given tutorials, we compare RAG-PJ with and without common world models: using either zero-shot text-based descriptions of next states or an oracle that returns ground-truth visual observations. We find that RAG-PJ gains little benefit from these rollouts when tutorials are present, even with perfect visual futures. This suggests prior world-model gains in tutorial-conditioned methods come mainly from tutorial understanding and the judge's ability to select the proposed actions, rather than from predicted rollouts. Error analysis reveals that the bottleneck is not rollout fidelity, but the judge's ability to reason over future states and recognize task completion. This highlights the need to improve not just next-state prediction in world models, but also reasoning over their predicted rollouts. Datasets and code are available at https://github.com/realjoenguyen/RAGPJ-official.