cs.LGMar 25, 2026

CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

Authors: Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, Kaixin Li, Patrice Bechard, Spandana Gella, Sai Rajeswar

Organizations: University of Waterloo · ServiceNow · Mila · Universit´e de Montr´eal · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Computer-use agents (CUAs) hold great promise for automating complex desktop workflows, yet progress toward general-purpose agents is bottlenecked by the scarcity of continuous, high-quality human demonstration videos. Recent work emphasizes that continuous video, not sparse screenshots, is the critical missing ingredient for scaling these agents. However, the largest existing open dataset, ScaleCUA, contains only 2 million screenshots, equating to less than 20 hours of video. To address this bottleneck, we introduce CUA-Suite, a large-scale ecosystem of expert video demonstrations and dense annotations for professional desktop computer-use agents. At its core is VideoCUA, which provides approximately 10,000 human-demonstrated tasks across 87 diverse applications with continuous 30 fps screen recordings, kinematic cursor traces, and multi-layerfed reasoning annotations, totaling approximately 55 hours and 6 million frames of expert video. Unlike sparse datasets that capture only final click coordinates, these continuous video streams preserve the full temporal dynamics of human interaction, forming a superset of information that can be losslessly transformed into the formats required by existing agent frameworks. CUA-Suite further provides two complementary resources: UI-Vision, a rigorous benchmark for evaluating grounding and planning capabilities in CUAs, and GroundCUA, a large-scale grounding dataset with 56K annotated screenshots and over 3.6 million UI element annotations. Preliminary evaluation reveals that current foundation action models struggle substantially with professional desktop applications (~60% task failure rate). Beyond evaluation, CUA-Suite's rich multimodal corpus supports emerging research directions including generalist screen parsing, continuous spatial control, video-based reward modeling, and visual world models. All data and models are publicly released.

Figures & tables

Figure 1: CUA-Suite Overview. Human GUI trajectories are recorded across desktop platforms, expert-verified, and annotated with keyframes, bounding boxes, and interaction logs. The resulting CUA-Suite comprises UI-Vision , a comprehensive benchmark to evaluate grounding and planning capabilities of computer-use agents; GroundCUA , which consists of densely labeled UI screenshots with 3.6 million annotations; VideoCUA , which comprises 55 hours of video of humans performing computer-use tasks with detailed action trajectories.
ModelBasicFunctionalSpatialAvg.
MAI-UI-32B ( Zhou et al., 2025b )59.157.126.947.7
MAI-UI-8B ( Zhou et al., 2025b )51.749.622.541.3
OpenCUA-72B ( Wang et al., 2025 )–––37.3
UI-Venus-Ground-72B ( Gu et al., 2025 )45.642.323.737.2
PhiGround-7B + o3 ( Zhang et al., 2025c )44.243.820.536.2
OpenCUA-32B ( Wang et al., 2025 )–––33.3
Table 1: Element Grounding Performance on UI-Vision . Models are ranked in descending order by their average score. Results are drawn from their respective original publications, with the exception of Qwen3-VL, which is reported as evaluated in Zhou et al. (2025b) .
DatasetPlat.Tasks#EnvsVideoDesk.HumanCoT#Act.Scale
Web
Mind2Web ( Deng et al., 2024 )Web2,350137✗✗✓✗3∼{\sim} 17K SS
AgentTrek ( Xu et al., 2025 )Web10,398127✗✗✗ss10∼{\sim} 126K SS
Mobile
AITW ( Rawles et al., 2023 )Mob.715K357✗✗Mix.✗4∼{\sim} 4.6M SS
GUI-Odyssey ( Lu et al., 2024a )Mob.8,334212✗✗Mix.✗9∼{\sim} 128K SS
Table 2: Comparison of VideoCUA with existing GUI trajectory and agent datasets. VideoCUA is the only large-scale, human-curated dataset providing continuous 30 fps expert video for professional desktop applications with long multi-layered CoT annotations. Video : continuous video recordings released with the dataset (not per-step screenshots). CoT : chain-of-thought annotations ( ℓ\ell = long multi-layered, ss = short). Though not released directly, VideoCUA has annotations dense enough for CoT generation with the code released. Scale : approximate data volume (SS = screenshots; h = video hours; fr. = video frames). OmniACT provides single-step tasks. OSWorld is an evaluation benchmark without training trajectories. “–” indicates not reported.
ModelPredsMean Px ↓\downarrowMed. Px ↓\downarrow@20px ↑\uparrow@50px ↑\uparrow
OpenCUA-7B1,946387.5236.07.9%16.5%
OpenCUA-32B1,999274.297.022.0%37.7%
Table 3: Action Prediction Results. OpenCUA models evaluated on 256 sampled VideoCUA tasks (87 apps) under task-level instruction with 5-step action history context.
Figure 2: Representative Prediction Failures. Green circles indicate ground truth and red circles show model predictions. These examples illustrate desktop-specific challenges for CUAs.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

CategoryPlatforms
EducationAnki, Zotero, Calibre, OpenBoard, Mendeley
BrowsersBrave, Chromium, Mozilla Firefox, DuckDuckGo
DevelopmentVS Code, Atom, Eclipse, NetBeans, PyCharm, IntelliJ IDEA, Brackets, Geany, Bluefish, KDevelop, Komodo Edit, Code::Blocks, Qt Creator, Arduino IDE
ProductivityLibreOffice Calc, LibreOffice Draw, LibreOffice Impress, LibreOffice Writer, draw.io, Joplin, OpenProject, Affine, PDFedit, OnlyOffice Calendar, OnlyOffice Document Editor, OnlyOffice Forms, OnlyOffice PDF Forms, OnlyOffice Presentation, OnlyOffice Spreadsheet, Nextcloud, Gnumeric, Simplenote, WeKan
Graphics and DesignBlender, GIMP, Inkscape, Krita, darktable, FontForge, Scribus, WordPress
Video and Audio ProductionOpenShot, OBS Studio, Lightworks, Shotcut, Natron, OpenToonz, Audacity, MuseScore
Appendix
Table 4: Categories of desktop applications and their corresponding applications.
CategoryDescription and Common UI Elements
Input ElementInteractive fields where users enter or modify data, like text boxes, checkboxes, radio buttons, etc.
SidebarVertical or horizontal panels that provide quick access to tools or navigation. Examples include tool palettes, folder trees, settings sidebars.
Information DisplayRegions that primarily present textual or numerical information. Examples include labels, console outputs, document text, and code blocks.
ButtonClickable controls that trigger an action like submit button, “OK/Cancel” buttons, play/pause buttons
NavigationElements that help users move within or across applications. Examples: tabs, back/forward arrows etc.
Visual ElementsNon-textual graphical elements that convey information or functionality. Examples include icons, thumbnails, images, charts, and progress bars.
Appendix
Table 5: UI element categories in CUA-Suite with descriptions and representative examples.
ApplicationDomainPredsMean Px@50px
Lowest @50px
DarktablePhoto Editing28259.03.6%
OnlyOffice FormsForms17232.611.8%
KritaDigital Art31417.812.9%
KodiMedia Center41591.914.6%
GnuCashAccounting52359.221.2%
Appendix
Table 6: Per-Application Performance (OpenCUA-32B). Applications exhibiting the lowest and highest @50px success rates (minimum 3 coordinate predictions). Performance varies widely, from under 5% to over 70%, reflecting the diversity of desktop application interfaces.
 OpenCUA-32BOpenCUA-7B
Distance RangeCount%Count%
50–100px25220.2%22213.7%
100–200px25220.2%32520.0%
200–300px15012.0%24615.1%
300–500px20516.5%29418.1%
>> 500px38631.0%53733.1%
Appendix
Table 7: Distribution of Prediction Distances >> 50px. Both models show a substantial proportion of high-distance predictions, with roughly one-third exceeding 500 pixels.
More in this topic

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 161 papers

Explore similar work

CardsList
Jun 15, 2026

ProCUA-SFT Technical Report

Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs -- and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.

Jaehun Jung, Ximing Lu, Brandon Cui +11

1NVIDIA · University of Washington

Supervised Fine-TuningComputer-Use AgentsLLM Agent Trajectory SynthesisLLM Agent Training
AskPDF
Aug 3, 2026

Qwen-CUA: Native Computer Use for (almost) Everything

Native computer use offers a general interface for agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive experience, and learning from sparse yet verifiable outcomes. We introduce Qwen-CUA, a native computer-use agent with a 397B-A17B Qwen mixture-of-experts backbone. It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs. Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence while preserving reusable prompt prefixes. For training, we build a cloud rollout fleet with access to nearly 100,000 vCPUs and tens of thousands of concurrent environments, construct approximately 40,000 verifiable tasks, and collect personalized long-horizon workflows across everyday and professional software. We optimize complete trajectories with verifiable rewards and trajectory slicing, while iterative training runs refresh supervised data and recalibrate reinforcement-learning tasks. Across eight benchmarks, Qwen-CUA outperforms Qwen3.7 and remains competitive with leading proprietary systems, reaching 86.2 on OSWorld-Verified and 18.5/48.4 binary/partial completion on OSWorld 2.0. Scaling the same recipe to a model with over one trillion parameters yields Qwen-CUA-Max, improving these scores to 87.6 and 21.2/53.3. Qwen-CUA also reduces RedTeamCUA attack success from 36.6 to 16.4 relative to Qwen3.7. Efficiency analyses, a browser deployment, and Bash-augmented experiments further characterize practical behavior. These results establish native computer use as a broadly capable agent foundation and highlight scalable verifiable interaction and hybrid tool use as key directions.

Dunjie Lu, Shuai Bai, Tianyi Bai +43

Computer-Use Agent BenchmarksComputer-Use AgentsReinforcement Learning with Verifiable RewardsLLM Agent TrainingLong-Horizon LLM AgentsComputer-Use Agent Evaluation
AskPDF
Oct 19, 2026

CUA-Lite: An Open Platform for Computer Use Agents

Open resources for computer use agents (CUAs) -- environments, datasets, and models -- remain scattered: without a standard, these resources cannot be integrated and scaled; additionally, evaluation, RL, and SFT cannot share one infrastructure. We present CUA-Lite, an open platform for developing CUAs, built around three standardized abstractions: (1) Unified environment interface, under which we integrate 12 benchmarks across desktop, browser, and mobile, plus lightweight VM-free sandboxes that replicate OSWorld's desktop at much lower cost, providing over 30,000 verifiable tasks. (2) Unified supervised data format, into which we convert the major public CUA datasets, from GUI grounding and understanding to full rollouts, and record our fresh rollouts from frontier CUAs on these verifiable environments. (3) Model harnesses shared across evaluation, RL, and SFT, one per model and implemented for 14 model families. Each harness runs its model in these computer environments, producing the rollouts that evaluation and RL consume, or renders stored supervised data into the model's own training format for SFT. We release reference scores across the integrated benchmarks, and show that RL and SFT through the same harness improve open-weight models. We release our dataset, code, and models, and the platform is open to contributions.

Zhanhui Zhou, Weichen Zhang, Haoran Liu +6

Computer-Use AgentsComputer-Use Agent BenchmarksComputer-Use Agent Evaluation
AskPDF