Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.
Figures & tables
Figure 1: Overview of CUA-Universe. (1) Scale: 16 real desktop applications spanning creative & interactive, office & knowledge, media & playback, and technical & web domains, each exposed through both a GUI and application-specific CLI interfaces. (2) Closed-loop environment-to-data pipeline: App-Forge adapts an application into a reproducible VM and the command-line surface it discovers, wraps, or generates (installer agent → reproducible VM + metadata → tool construction); Task-Weave synthesizes diverse hybrid GUI+CLI tasks at scale over seed files (operation abstraction → task instantiation → agentic refinement); and Path-Steer rolls agents out onto efficient hybrid paths and harvests high-quality, directly reusable trajectories for post-training. (3) Impact: training on this data shifts interaction behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration, producing more capable and efficient agents with consistent improvements across diverse benchmarks, including CUA-Verse (Score +39.3 pts; −37% steps, −60% tokens), OSWorld (SR +16.8 pts; −57% steps, −44% tokens), and OSWorld-MCP (Score +7.84 pts; −27% steps, −30% tokens).
Figure 2: Training-data composition. Our pipeline scales to diverse applications and produces substantial training data, with 4,923 episodes and approximately 235K training steps across all 16 applications. The balanced coverage of CUA-Universe extensions and OSWorld applications further demonstrates the diversity and scalability of the generated data.
Model
Mode
CUA-Verse
OSWorld
Score
Steps
Token (K)
SR (%)
Steps ↓
Step Gain ↑
Token(K) ↓
Token Gain ↑
GUI
53.7
27.2
224.8
Kimi K2.5
GUI+CLI
0.522
21.1
275
54.5
28.9
1.15
277.3
1.09
GUI
55.7
28.4
188.0
Seed2.1 Pro
GUI+CLI
0.599
40.1
449
59.0
27.7
1.10
215.6
0.94
GUI
66.8
14.7
595.0
Table 1: Combined performance on CUA-Verse and OSWorld. CUA-Verse : 8 apps × 20 tasks; Score is the average VLM judge score, and Token is the average number of tokens per episode in thousands. OSWorld : 244-task controlled scope excluding os and multi-app tasks, comparing GUI and GUI+CLI execution. Token/task is reported in thousands. Step Gain and Token Gain are the mean per-task GUI-to-GUI+CLI cost ratios over jointly solved tasks. A gain of k× indicates that GUI+CLI uses k times fewer steps or tokens on the same successfully solved tasks.
Figure 3: Application-level performance on the eight CUA-Verse applications. Each cell is the score for one model–application pair; darker shading is better. The horizontal rule separates proprietary (top) from open-source (bottom) models. Ours surpasses its Qwen3.5-9B on all eight applications, and is strongest on audio/video apps (Audacity, OBS).
Figure 5
Group
Accept Rate ( ≥0.75 )
Mean score
Avg steps
Avg tokens
Avg cost
Kimi K2.5 w/ Path-Steer
0.51
0.71
22.75
331,988
$0.26
Kimi K2.5 w/o Path-Steer
0.44
0.63
26.68
385,107
$0.31
Seed2.1 Pro w/ Path-Steer
0.54
0.75
25.53
263,500
$0.29
Seed2.1 Pro w/o Path-Steer
0.45
0.67
28.78
303,653
$0.33
Table 3: Rollout efficiency of Path-Steer (enabled vs. disabled) , for the Kimi K2.5 data-generation backbone and Seed2.1 Pro (a cross-backbone check). Accept Rate is the fraction of the 320 tasks scoring ≥0.75 ; tokens are averaged per task.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Steps
CLI %
Score
Kimi K2.5
21.1
0 30.3
0.522
Seed2.1 Pro
40.1
0 49.2
0.599
GPT-5.5
23.1
100.0
0.768
Qwen3.5-9B
56.2
00 0.0
0.189
EvoCUA-8B
41.7
00 3.0
0.330
Ours
35.2
0 25.3
0.582
Appendix
Table 4: Action-modality behavior on CUA-Verse , averaged over the eight applications. Steps is the mean executed action count, CLI % is the CLI share of executed GUI + CLI actions, and Score is the average VLM-judge score. Greater CLI use often accompanies higher efficiency and performance, but CLI usage alone is insufficient: even GPT-5.5, which executes entirely through CLI in these trajectories, does not fully solve the benchmark, reflecting the visually grounded demands of CUA-Verse.
App
Base
8-app LoRA (OOD-only)
16-app LoRA (+in-domain)
Chrome
23.9
26.1
37.0
GIMP
26.9
30.8
42.3
Calc
8.5
10.6
18.8
Impress
27.7
25.8
31.8
Writer
34.8
39.1
52.2
Thunderbird
33.3
46.7
73.3
Appendix
Table 5: Out-of-domain training transfer on OSWorld under the controlled 244-task GUI+CLI scope. Values are success rates in percent. The 8-app OOD-only LoRA is trained on cua-universe extensions applications disjoint from the OSWorld applications, whereas the full 16-app LoRA is trained on both. Parenthesized values in the final row denote absolute percentage-point gains over the untuned Qwen3.5-9B.
Application
Domain
Seed
OSWorld applications
Chrome
Web browsing
Profile
GIMP
Image editing
.xcf
LibreOffice Calc
Spreadsheets
.ods
LibreOffice Impress
Presentations
.odp
LibreOffice Writer
Word processing
.odt
Appendix
Table 6: Desktop applications supported by CUA-Universe. Blue rows indicate applications inherited from OSWorld, whereas orange rows indicate applications added by CUA-Universe.
Figure 5: Agent-visible command count per application, grouped by source: the eight applications reused from OSWorld and the eight CUA-Universe extensions ( ∼ 404 commands in total).
Figure 6: Six opened seed examples used to ground task synthesis across diverse GUI applications. Each seed is a concrete project or file with metadata and verifiable app state, rather than a generic natural-language prompt.
Figure 7: VLC task trajectories w/ Path-Steer and w/o Path-Steer . The Path-Steer trajectory shows optimal solution using CLI tools and GUI actions, whereas the trajectory without sterr-path guidance becomes stuck during filter configuration and fails the task.
App
p
FP
FN
Prec. (%)
FNR (%)
Chrome
0.53
0
3
100.0
10.0
GIMP
0.52
1
3
96.7
10.0
LibreOffice Calc
0.56
0
1
100.0
3.3
LibreOffice Impress
0.51
0
2
100.0
6.7
LibreOffice Writer
0.53
0
0
100.0
0.0
Thunderbird
0.59
0
0
100.0
0.0
Appendix
Table 7: VLM judge vs. human labels, all 16 applications. Three annotators, blind to the judge score, labeled 30 judge-accepted and 30 judge-rejected trajectories per application. p is the true acceptance rate; FP counts accepted trajectories that were not a human “fully success” (out of 30, all of them partial successes—zero hard failures); FN counts rejected trajectories that were in fact fully successful (out of 30). Precision is on the acceptance set and FNR on the rejection set. Reweighting each application by p gives an overall agreement of 97.0% and Cohen’s κ=0.94 . Blue rows are OSWorld applications; orange rows are CUA-Universe extensions.
Split
Episodes
Step records
Total (GB)
CUA-Verse
2,526
140,088
172.97
OSWorld
2,397
95,320
113.67
Total
4,923
235,408
286.64
Appendix
Table 8: Final training splits. “Total (GB)” is the on-disk size including all screenshots; step records reference images by path.
Parameter
Value
Tuner type
LoRA
LoRA rank
8
LoRA α
32
LoRA dropout
0.05
LoRA bias
none
Target modules
all-linear
Appendix
Table 9: LoRA configuration.
Parameter
Value
Optimizer
ms-swift default
Learning rate
1×10−4
Warmup ratio
0.05
Per-device train batch size
2
Gradient accumulation
1
Effective batch size (8 GPUs)
16
Appendix
Table 10: Optimization and sequence settings.
Figure 8: An OSWorld VS Code task under GUI+CLI vs. GUI-only interfaces. GUI+CLI mode combines GUI navigation with direct settings-file editing: it locates the target setting, writes debug.focusEditorOnBreak: false to settings.json , and verifies persistence, completing the task in 5 CLI calls and 12 GUI actions (reward 1 ). GUI-only mode repeatedly searches and clicks through the settings UI but fails to commit the correct configuration, exhausting the 60-action budget (reward 0 ).
Figure 9: CUA-Verse task composition. Mean number of abstract tools per task for each application, split into CLI (orange) and GUI (blue) operations. Every application requires both modalities.
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.
Tongbo Chen, Junbo Niu, Zhengxi Lu +8
Zhejiang University · Peking University · Tsinghua University
Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: https://x-plug.github.io/ToolCUA/
Xuhao Hu, Xi Zhang, Haiyang Xu +6
Tongyi Lab, Alibaba Group · Fudan University · Shanghai Artificial Intelligence Laboratory
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.