OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

Hanzhang Zhou; Long Li; Quyu Kong; Lei Wang; Panrong Tong; Long Chen; Tianyu Xia; Gongjie Zhang

cs.CVOct 15, 2026

OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

Authors: Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong, Long Chen, Tianyu Xia, Gongjie Zhang,

Organizations: Alibaba Token Hub, Alibaba Group

Abstract

Computer use has reached a turning point: the focus is expanding from GUI-only interaction to systems that complete tasks on users' computers through visual and programmatic interfaces. We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces. Yet frontier implementations, such as GPT-6 Astra's computer-use system, remain largely closed, limiting systematic study of their design. We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community. This report presents the first release of OpenGCUA. Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their existing programming capabilities directly to computer interaction. We also propose harness-to-model decision transfer, moving selected observations and task-state decisions from fixed harness rules to the model. On OSWorld-v2, our full configuration improves task scores across all eight evaluated models, including GPT-6 Astra and Claude Opus 5, raising mean partial score by +19.6 points and reducing reported steps per task by an average of 53.0% relative to batched GUI-only execution. We release our implementation and prompts, evaluation trajectories from all eight models, and preliminary versions of a real-device harness and a cross-device SDK. The harness addresses real-device challenges not fully captured by benchmarks, while the SDK provides a common programming interface across computers, mobiles and browsers. Across four real macOS tasks under our real-device harness, selected Qwen3.8-Max runs use 19.4 times as many reported reasoning tokens as GPT-6 Astra, highlighting substantial differences between models in real-device execution efficiency.

Figures & tables

Figure 1: On the offline set of OSWorld-v2, OpenGCUA improves all eight frontier models, raising mean partial score by 19.6 points and reducing steps per task by 53% . On real-device, the gap between models remains large: Qwen3.8-Max uses 19.4 $\times$ GPT-6 Astra’s reasoning tokens but completes fewer tasks.

Figure 2: Overview of Open Generalist CUA.

	Decision at each step
Family	What to observe	How to act	How to organize	Interface choice
GUI agent	Screenshot only $q_{t}\equiv\{\mathrm{screenshot}\}$	One GUI action $p_{t}\in\mathcal{A}_{\mathrm{GUI}}$	Harness rule, e.g., last $k$ screenshots $m_{t+1}=h(m_{t},o_{t})$	No choice (GUI only)
CLI agent	Command output only $q_{t}\equiv\emptyset$ , only $e_{t}$	Shell commands and scripts program over $\mathcal{A}_{\mathrm{CLI}}$	Partly by the model, e.g., context summaries	No choice (shell only)
Multi-agent hybrid	Fixed for each sub-agent $q_{t}\equiv\mathcal{Q}_{k}$	Fixed for each sub-agent $p_{t}$ over $\mathcal{A}_{k}$	Split between orchestrator and sub-agents	Orchestrator, per subtask
Qwen-UI-Agent (GUI+CLI)	Screenshot every step, plus command output when a command runs $q_{t}\equiv\{\mathrm{screenshot}\}$ , plus $e_{t}$	A batch of GUI and CLI actions sequence over $\mathcal{A}_{\mathrm{GUI}}\cup\mathcal{A}_{\mathrm{CLI}}$	Harness rule $m_{t+1}=h(m_{t},o_{t})$	Model, per action (GUI or shell)
GCUA	Chosen per step, plus program output $q_{t}\subseteq\mathcal{Q}$ , plus $e_{t}$	Program over all interfaces program over $\mathcal{A}$	Decided by the model $m_{t+1}\sim\pi_{\theta}$	Model, per step

Table 1: Existing agent families and GCUA compared on the three decisions. Gray text gives the formal notation. The observation after each step contains the views selected by

q_{t}

and the execution output

e_{t}

(Eq. 2 ). Existing families fix some of these decisions in advance. The shaded rows mark the path this report follows, from our earlier GUI+CLI agent to GCUA, in which the model makes all three decisions at each step.

Figure 3: Programs coordinate actions and checks. (a) GUI clicks alternate with checks of generated contracts and local drafts. (b) A loop visits seven webpage sections and checks each active section through the DOM. (c) A draft comparison determines whether to click Send, without verifying delivery.

Figure 4: Programs prepare and verify evidence. (a) A program assembles 25 views into a contact sheet for the model’s next observation. (b) Assertions check background images and transparency, and verify that non-background content remains unchanged.

Figure 5: Model outputs for observation, task state, and intent. (a) The two programs differ in whether they request a new screenshot for the next input. (b) A form record remains unverified until the model reads submission confirmation. (c) Intent states what the program should change and preserve.

Model	Configuration	Task completion		Interaction cost
		Score $\uparrow$	Success $\uparrow$	Steps/task $\downarrow$	Output K/task $\downarrow$
Claude Opus 5	GUI + batch	60.7	29.3	103.1	378.3
Claude Opus 5	OpenGCUA	70.1 (+9.4)	36.6	61.4 ( $-40.4\%$ )	89.3
GPT-6 Astra	GUI + batch	67.1	36.6	102.1	30.8
GPT-6 Astra	OpenGCUA	75.7 (+8.7)	46.3	33.5 ( $-67.2\%$ )	31.9
Gemini 3.8 Flash	GUI + batch	50.1	22.0	166.8	80.3

Table 2: CUA versus OpenGCUA on the OSWorld-v2 offline set. Each model is evaluated with a GUI-only baseline that supports batched actions and the full GCUA configuration with code execution and harness-to-model decision transfer. Bold marks the better value within each model and metric. K denotes 1,000 tokens.

Figure 6: Code-execution ablation on OSWorld-v2. Code execution achieves higher task scores and fewer steps than single-action GUI, batched GUI, and batched GUI+CLI setting across all three models.

Model	Task score $\uparrow$		Steps/task $\downarrow$
	GUI+CLI	GCUA	GUI+CLI	GCUA
Qwen3.8-Max	57.2	64.0	66.1	57.0
DeepSeek-V4.1-Flash	50.9	57.7	86.7	84.9
GLM-5.3-Flash	55.8	64.9	81.7	75.5
Claude Opus 5	65.7	70.1	61.6	61.4

Table 3: Full GCUA versus batched GUI+CLI.

Configuration	Qwen3.8-Max			DeepSeek-V4.1-Flash
	Score $\uparrow$	Steps/task $\downarrow$	Cache hit (%)	Score $\uparrow$	Steps/task $\downarrow$	Cache hit (%)
Code execution	62.4	54.9	75.6	56.7	74.5	61.9
+ screenshot	64.3	63.0	86.1	58.1	90.0	76.3
+ todo	64.0	56.7	–	58.5	74.1	–
+ intent	61.5	53.4	–	53.3	82.4	–
+ micro-compaction	63.4	58.7	68.6	56.8	79.7	58.9

Table 4: Ablation of harness-to-model decision transfer and harness-managed context policies. Each mechanism is added separately to code execution; the final row combines all mechanisms. Cache hit is the fraction of input tokens served from cache; dashes indicate omitted rates.

Figure 7: Task performance under monetary and time constraints. Mean partial scores on the 82-task offline subset of OSWorld-v2 under increasing per-task spending caps (a) and recorded time limits (b). Panel (a) reports average expenditure, including unsuccessful attempts. Top bars identify the highest-scoring models; shading indicates cache-write cost bounds. Time comparisons are exploratory.

Figure 8: Architecture of the Cross-Device SDK. A common programming interface connects agent programs to computers, mobile devices, and browser environments, enabling cross-device workflows through platform-specific adapters.

Figure 9: Five-layer architecture of the real-device harness. The runtime executes model-generated programs through the SDK and platform drivers to interact with applications and the operating system.

Model	Interaction cost			Task completion
	Steps/task $\downarrow$	Out K/task $\downarrow$	Est. USD/task $\downarrow$	Complete $\uparrow$
GPT-6 Astra	34.5	12.9	18.290	4/4
Claude Opus 5	52.3	42.3	17.117	3/4
Gemini 3.8 Flash	36.0	31.9	0.662	2/4
Qwen3.8-Max	50.8	73.4	1.494	3/4
Qwen3.8-Flash	67.3	75.7	0.120	2/4

Table 5: Real-device results on four macOS tasks. Costs and completion use the same selected run per task (run 1). Costs are four-task means; Complete counts tasks judged complete. DeepSeek lacks a hotel trajectory, so its aggregates are unavailable.

Figure 10: Reasoning cost and observation-target selection. GPT-6 Astra captures the selected Amazon page, while Qwen3.8-Max captures the foreground Taobao page and then switches to Amazon. Qwen3.8-Max uses substantially more reasoning tokens.

Figure 11: Task coverage in code execution. GPT-6 Astra fills all 125 required Level answer cells, while Qwen3.8-Max leaves 40 empty.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Workbook interface during the Claude Opus 5 run. The view shows the rules and answer cells. Displayed answers are model outputs, not part of the initial workbook.

Figure 13: First page of the supplied blank application form. Completed forms and applicant details are not shown.

Figure 14: First page of the supplied Combined_Snacks candidate list, showing product images, package descriptions, and original listing links. The screenshot retains the source language.

Figure 15: InterContinental Paris Le Grand website observed during the GPT-6 Astra run.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 161 papers

Oct 19, 2026

CUA-Lite: An Open Platform for Computer Use Agents

Open resources for computer use agents (CUAs) -- environments, datasets, and models -- remain scattered: without a standard, these resources cannot be integrated and scaled; additionally, evaluation, RL, and SFT cannot share one infrastructure. We present CUA-Lite, an open platform for developing CUAs, built around three standardized abstractions: (1) Unified environment interface, under which we integrate 12 benchmarks across desktop, browser, and mobile, plus lightweight VM-free sandboxes that replicate OSWorld's desktop at much lower cost, providing over 30,000 verifiable tasks. (2) Unified supervised data format, into which we convert the major public CUA datasets, from GUI grounding and understanding to full rollouts, and record our fresh rollouts from frontier CUAs on these verifiable environments. (3) Model harnesses shared across evaluation, RL, and SFT, one per model and implemented for 14 model families. Each harness runs its model in these computer environments, producing the rollouts that evaluation and RL consume, or renders stored supervised data into the model's own training format for SFT. We release reference scores across the integrated benchmarks, and show that RL and SFT through the same harness improve open-weight models. We release our dataset, code, and models, and the platform is open to contributions.

Zhanhui Zhou, Weichen Zhang, Haoran Liu +6

Computer-Use Agents Computer-Use Agent Benchmarks Computer-Use Agent Evaluation

Ask PDF

Sep 4, 2026

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.

Haoting Shi, Wenhao Wang, Weicheng Fang +6

Shanghai Jiao Tong University · Zhejiang University

Computer-Use Agents Tool-Using Agents Computer-Use Agent Benchmarks

Ask PDF

Sep 29, 2026

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.

Tongbo Chen, Junbo Niu, Zhengxi Lu +8

Zhejiang University · Peking University · Tsinghua University

Computer-Use Agents GUI Agents LLM Agent Training RL for Tool Use

Ask PDF

Abstract

Computer-Use Agents

Explore similar work

CUA-Lite: An Open Platform for Computer Use Agents

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents