Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Haoqing Wang; Xiaoyi Zhang; Sicheng Fan; Zhexin Zhang; Xinjie Zhang; Jiahao Li; Bin Li; Yan Lu

cs.HCOct 30, 2026

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Authors: Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu

Organizations: Microsoft Research · Fudan University · Tsinghua University

Abstract

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.

Figures & tables

Figure 1: Left: HCU’s programmatic, semantic, and visual channels. Right: OSWorld 2.0 results. Stars mark HCU and the staircase marks the Pareto frontier, detailed in Section 4.2 .

Figure 2: HCU overview. Top: channels, skill, and action guards. Bottom: a compact AX view, batched actions with a wait step, and the returned effect summary and AX diff.

a targeted AX batch or GUI coordinate batch

A

last screenshot

F

, if any

Algorithm 1 Cross-channel action guards (simplified)

System	Model	Partial (%) $\uparrow$	Binary (%) $\uparrow$	Out./1k $\downarrow$	Cost ( $)$ \downarrow$
Computer-use loop	GPT-6 Astra ∗	72.60	–	–	–
	Claude Opus 5	68.31	31.43	102.87	28.43
	Muse Spark 1.3	66.90	32.00	–	–
	GPT-5.6 Sol	62.72	27.34	36.14	13.94
	Gemini 3.8 Flash §	59.00	–	–	–
	Kimi K3 §	58.30	–	–	–

Table 1: OSWorld 2.0 main results. Bold: best. ∗ OpenAI’s offline-subset result; unranked. § Pre-0808: StateAct uses 0624; Gemini/Kimi task counts unspecified.

Configuration	Score (%) $\uparrow$	Steps $\downarrow$	Input (M) $\downarrow$	Time (s) $\downarrow$	Cost ( $)$ \downarrow$
HCU	65.59	28.60	0.941	351.3	2.52
w/o GUI channel	66.08	32.16	0.909	352.4	2.45
w/o AX channel	62.79	33.28	1.051	401.8	2.82
w/o skill	64.00	33.47	0.962	385.5	2.57
CUA Driver	62.30	40.74	2.181	500.8	5.65

Table 2: GPT-5.4 ablations, OSWorld 1.0. Darker red: larger losses vs. HCU. Runs: Appendix C.1 .

Figure 3: Skill effects.

Figure 4: Interface use. (a) Mean per-task tool-call shares (Programmatic: the agent’s programmatic tools; CUA Driver includes browser tools). (b) Within-HCU AX/GUI shares: observation (AX reads, GUI screenshots) versus execution (AX actions, GUI inputs). (c) AX/GUI acting calls with at least two actions (%).

	Score (%) $\uparrow$			Change with HCU (%) $\downarrow$
Model	CUA Driver	HCU	$\Delta$ (pp)	Steps	Input	Time	Cost
GPT-5.4	62.30	65.59	+3.29	$-$ 30	$-$ 57	$-$ 30	$-$ 55
Claude Opus 5.5	68.68	71.62	+2.94	$-$ 1	$-$ 32	+6	$-$ 8
Gemini 3.8 Flash	54.67	67.57	+12.90	$-$ 17	$-$ 39	$-$ 29	$-$ 44
Grok 4.7	67.90	74.71	+6.81	$-$ 22	$-$ 18	$-$ 13	$-$ 13

Table 3: HCU versus CUA Driver on OSWorld 1.0. Resource changes are relative to CUA Driver; darker blue indicates larger improvements. Details: Appendix D.1 .

Figure 5: (a) Contributions to the HCU–CUA Driver Score difference, grouped by HCU AX/GUI use. (b) Tasks using each interface’s desktop tools; labels show HCU–CUA Driver differences in points.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Evaluation of HCU across benchmarks and platforms (GPT-5.4). (a) OSWorld 1.0 (Linux, 360 tasks): Score, Cost per task, and tool-call shift from Shell to desktop tools across CUA Driver, evaluated HCU, and released HCU. (b) WeaveBench (Linux, 114 tasks): mean score and Cost per task across Official OpenClaw, CUA Driver, and HCU. (c) MacAgentBench UI-dependent subset (macOS, 28 tasks) and (d) Windows Agent Arena (Windows, 111 tasks): task performance and resource reductions relative to CUA Driver.

Metric	CUA Driver	Evaluated HCU	Released HCU	vs. Evaluated	vs. CUA Driver
Score (%)	62.47	65.77	70.13	+4.36 pp	+7.66 pp
Full-score tasks (%)	60.00	62.41	66.67	+4.26 pp	+6.67 pp
Steps	40.58	28.45	24.92	$-$ 12.4%	$-$ 38.6%
Tool calls	33.67	24.94	21.63	$-$ 13.3%	$-$ 35.8%
Desktop calls	12.02	9.83	13.10	+33.2%	+9.0%
Shell commands	21.66	15.11	8.52	$-$ 43.6%	$-$ 60.7%

Appendix

Table 4: CUA Driver, evaluated HCU, and released HCU on OSWorld 1.0 with GPT-5.4 (xhigh) in Codex, 360 tasks. CUA Driver and Evaluated HCU: mean of three runs. Values are per task unless noted; tool calls count desktop calls (AX and GUI) and shell commands (programmatic channel).

Metric	Official OpenClaw	HCU	CUA Driver
Mean score (%)	43.84	53.25	46.37
Tasks with score $\geq 0.8$	23/114	24/114	12/114
Total tool calls	9,458	9,465	11,494
Tool calls	82.96	83.03	100.82
Input tokens	3,537,196	4,478,251	8,712,900
Output tokens	32,677	26,821	31,671

Appendix

Table 5: Three-arm comparison on all 114 WeaveBench tasks with GPT-5.4 (medium). Values are per task unless noted.

Metric	CUA Driver	HCU	Change
Completed tasks	24/28 (85.7%)	25/28 (89.3%)	+3.6 pp
Steps	52.21	39.61	$-$ 24.1%
Tool calls	44.54	35.64	$-$ 20.0%
Time, mean (s)	876.0	544.5	$-$ 37.8%
Time, median (s)	659.5	419.3	$-$ 36.4%
Input tokens	3,637,053	1,344,550	$-$ 63.0%

Appendix

Table 6: HCU and CUA Driver on the 28 UI-dependent macOS tasks of MacAgentBench with GPT-5.4 (xhigh). Values are per task unless noted.

Metric	CUA Driver	HCU	Change
Score (%)	71.89 (79.80/111)	80.86 (89.75/111)	+8.97 pp
Full / partial / zero tasks	78 / 2 / 31	88 / 2 / 21	+10 full
Steps, mean (median)	24.45 (20.5)	17.35 (14.0)	$-$ 29.0%
Tool calls, mean (median)	23.81 (20.0)	16.43 (13.0)	$-$ 31.0%
Time, mean (s)	344.3	220.3	$-$ 36.0%
Time, median (s)	199.5	171.8	$-$ 13.9%

Appendix

Table 7: HCU and CUA Driver on 111 Windows Agent Arena tasks with GPT-5.4 in Codex. Values are per task unless noted.

Configuration	Direct code/CLI	AX	GUI	Skill
HCU	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
HCU w/o GUI	$\checkmark$	$\checkmark$	–	$\checkmark$
HCU w/o AX	$\checkmark$	–	$\checkmark$	$\checkmark$
HCU w/o skill	$\checkmark$	$\checkmark$	$\checkmark$	–

Appendix

Table 8: Direct tool access in the channel and skill ablations. A check mark means that the configuration exposes that component directly.

Metric	CUA Driver	HCU
Partial (%)	68.82	72.37
Binary (%)	31.5	32.4
Steps	94	85
Tool calls	111.7	78.4
Input tokens / task (M)	6.9	6.9
Output tokens / task (k)	25.8	25.4

Appendix

Table 9: HCU versus CUA Driver with GPT-6 Astra in Codex, OSWorld 2.0 full set (0808). Code Mode; 500-step limit.

Configuration	Run 1	Run 2	Run 3	Mean $\pm$ SD
HCU	66.16	65.65	64.95	65.59 $\pm$ 0.61
HCU w/o GUI	65.64	64.70	67.92	66.08 $\pm$ 1.66
HCU w/o AX	63.53	62.30	62.55	62.79 $\pm$ 0.65
HCU w/o skill	62.84	65.33	63.83	64.00 $\pm$ 1.25
CUA Driver	63.29	61.09	62.51	62.30 $\pm$ 1.12

Appendix

Table 10: Per-run Score (%); mean

\pm

sample standard deviation across three runs.

Metric (numerator / denominator)	HCU w/o skill	HCU
Schema rejection / calls	2460/14160	147/10783
Task-runs with schema rejection	601/1083	132/1083
Task-runs with consecutive schema failures	484/1083	4/1083
All AX/GUI errors / calls	2686/14160	638/10783
Stale-category errors / calls	107/14160	88/10783
Execution or assertion errors / calls	66/14160	380/10783

Appendix

Table 11: Skill diagnostics pooled across three runs per configuration (2,166 traces; 361 tasks). Cells report event or task-run counts.

Figure 7: Skill effects across 2,166 OSWorld 1.0 traces (GPT-5.4). (a) Collapse of invocation errors, schema-error chains, and repeated same-tool failures. (b) Adoption of composable AX features (default diff feedback, multi-action batches, find , and in-call checks).

Model	GPT-style	Claude-style	Difference (pp)
Claude Opus 5.5	71.70	72.51	+0.82
GPT-5.4	65.59	63.56	-2.03

Appendix

Table 12: GUI action-space replacement within HCU: Score (%); differences are Claude-style minus GPT-style.

Figure 8: HCU–CUA Driver Score differences by application; tasks are averaged over runs before pairing. $n$ : paired tasks; asterisk: Claude Opus 5.5’s 14-task Thunderbird comparison.

Figure 9: Recorded GIMP interaction, HCU (GPT-5.4), OSWorld 1.0. A screenshot reveals the value omitted by AX; execution resumes through the earlier OK reference.

Figure 10: Steps decomposition by desktop-tool use across models. (a) Mean Steps per task and equivalent task counts

N_{g}

on tasks where both interfaces use desktop tools. (b) Additive contributions

\Delta_{g}

to the overall Steps difference (negative favors HCU) and net per-task Steps.

Figure 11: Termination, step distributions, and limit-grouped Score gains across models (Appendix D.1 ). (a) Termination shares; counts denote termination at the 100-step limit. (b) Within-group cumulative Steps distributions;

n

: HCU / CUA Driver tasks. (c) Additive Score-gain contributions (pp) grouped by whether CUA Driver reaches the 100-step limit, with mean task counts beneath each bar.

Figure 12: Post-error behavior across models. (a) Next-call outcome shares over desktop-tool error events, with total error counts

n

on the right. (b) Destination channel shares conditional on HCU switching channels.

Operation	Main request fields	Returned evidence and use
AX apps	No target	List top-level windows as wN with title, application, and focus.
AX observe	app or window_ref ; scope , root	Return a compact window tree with state and rN references for subsequent calls.
AX find	app or window_ref , query , scope ; or cursor	Retrieve matching targets and local context without reading an entire window tree.
AX act	Up to 64 ordered actions and wait steps; observe	Execute steps in order, stop at the first failed step, and report per-target effects, window changes, and post-action observation.
GUI observation	actions with a single screenshot or zoom	Return the full-desktop frame, binding its window layout for subsequent coordinate checks, or an enlarged region of that frame.
GUI input	Up to 64 ordered actions : pointer, drag, scroll, text, keys, or wait	Execute mouse and keyboard input against the frame-bound layout and return the final screenshot.

Appendix

Table 13: Desktop operations and main request fields.

Figure 13: HCU interface and MCP server boundaries. HCU unifies programmatic, semantic, and visual channels; the HCU MCP server provides the AX and GUI tools over a shared Go service, while programmatic tools live in the agent and the evaluation runner remains external. Platform backends: Figure 19 .

Figure 14: Same-capture AX excerpts (‘o200k_base‘). Counts cover complete view bodies, excluding headers and images; projection retains all issued references.

Figure 15: Obsidian screenshot, stock OSWorld 2.0 XML, and compact AX excerpts.

Figure 16: Recorded AX interaction with GPT-5.6 Sol on OSWorld 2.0. The search batch returns the entered value, and a subsequent find supplies the next target.

Figure 17: Cross-channel state and action guards. GUI mutation invalidates working observations while retaining identities and eligible diff snapshots. The GUI guard checks window layout before coordinate input, and uncertain preparation stops the batch.

Figure 18: Recorded Writer layout-guard interaction, HCU (GPT-5.4), OSWorld 1.0. The GUI batch is rejected before input and returns a fresh screenshot; the first two crops share region and scale. Retrying with revised pixel coordinates dispatches seven actions.

Figure 19: Shared HCU MCP server and native backends. Rows show capabilities, not a fixed workflow; AX can also return screenshots, and its wait steps are optional. Dashed branches are alternative backends, one selected per build, while the agent’s programmatic tools remain in the harness.

Operation	GPT-style ( type field)	Claude-style ( action field)
Observe	screenshot ; zoom of a frame region	screenshot , zoom , cursor_position
Click	click with button and modifier keys ; double_click	left_click , right_click , middle_click , double_click , triple_click
Pointer	move ; drag along a path	mouse_move ; left_click_drag , left_mouse_down , left_mouse_up
Scroll	scroll with scroll_x and scroll_y deltas	scroll by direction, in wheel clicks
Keyboard	type ; keypress with a key array	type ; key with a key-chord string; hold_key
Wait	wait , one second	wait

Appendix

Table 14: GUI action vocabularies. The GPT-style space is HCU’s default.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 171 papers

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Sep 4, 2026Haoting Shi, Wenhao Wang, Weicheng Fang +6Computer-Use Agents Tool-Using Agents
Shanghai Jiao Tong University · Zhejiang University
HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
Sep 29, 2026Tongbo Chen, Junbo Niu, Zhengxi Lu +8Computer-Use Agents GUI Agents
Zhejiang University · Peking University · Tsinghua University
Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
Aug 4, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Computer-Use Agents RL for LLM Agents
University of Electronic Science and Technology of China · AI Platform, Xiaohongshu Inc. · Gaoling School of Artificial Intelligence, Renmin University of China +1

Sep 4, 2026J/K move · Enter open · S save

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang+5

Shanghai Jiao Tong University · Zhejiang University

Computer-Use Agents Tool-Using Agents Computer-Use Agent Benchmarks

Open paper AskPDF arXiv

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.

Abstract

Computer-Use Agents

Explore similar work

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents