OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents

Bowen Yang; Kaiming Jin; Zichen Ding; Zhenyu Wu; Zehao Li; Zhoumianze Liu; Kanzhi Cheng; Fangzhi Xu

cs.LGOct 15, 2026

OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents

Authors: Bowen Yang, Kaiming Jin, Zichen Ding, Zhenyu Wu, Zehao Li, Zhoumianze Liu, Kanzhi Cheng, Fangzhi Xu,

Organizations: University of Science and Technology of China · Shanghai AI Laboratory · National University of Singapore · Shanghai Jiao Tong University ·

Abstract

Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at https://os-copilot.github.io/OS-Helix/.

Figures & tables

Figure 1: Overview of GUI-only, code-only, and hybrid computer use.

Figure 2: Overview of the OS-Helix pipeline. Stage 1 constructs executable desktop environments by composing applications and artifacts. Stage 2 explores each environment to synthesize instructions and evaluators, yielding verifiable tasks. Stage 3 uses teacher-generated GUI/code trajectories for SFT cold start. Stage 4 performs online agentic RL in live sandboxes.

Agentic Frameworks
Method	Setting	Success Rate (%)
Method	Setting	OS	Office	Daily	Prof.	Work.	Avg.
UiPath w/ GPT-5 ( 2025 )	GUI	73.91	49.52	62.12	71.43	37.30	53.69
CoAct-1 w/ GPT-5 ( 2026 )	Hybrid	75.00	62.93	57.94	71.43	47.87	59.93
GTA1 w/ GPT-5 ( 2026d )	GUI	79.17	63.91	62.56	79.59	50.91	63.41
Agent S3 w/ GPT-5 ( 2025 )	Hybrid	77.50	66.46	61.23	69.80	51.37	62.63

Table 1: Main results of OS-Helix on OSWorld. ♣ marks results reproduced by us; other baseline results are sourced from the official leaderboard. OS contains system-level OS tasks; Office contains LibreOffice Calc, Impress, and Writer tasks; Daily contains Chrome, Thunderbird, and VLC tasks; Prof. contains GIMP and VS Code tasks; and Work. denotes Workflow tasks involving multiple applications.

Model	Succ. (%)
Agent S3 w/ GPT-5 ( 2025 )	54.10
OS-Symphony w/ GPT-5 ( 2026a )	63.45
Qwen3.5-27B ( 2026a ) ♣	56.41
OS-Helix-27B	57.95
OS-Helix-27B-RL	63.87

Table 2: Main results of OS-Helix on WindowsAgentArena. ♣ marks the baseline reproduced by us. All entries use a 50-step budget and a hybrid action space.

Figure 3: Execution efficiency on OSWorld.

Figure 4: Training dynamics, scaling, and coverage analysis.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Synthesis Grounding	Application Scaling	Real Artifact Scaling	Cross-App Tasks	Rule-Based Evaluator
OS-Genesis ( 2025 )	E	$\times$	$\times$	$\times$	$\times$
AUTO PLAY ( 2026 )	K/E/C	$\times$	$\times$	$\times$	$\times$
TreeCUA ( 2026 )	K/E/C	$\times$	$\times$	$\times$	$\times$
EvoCUA-1.5 ( 2026 )	K	$\times$	$\checkmark$	$\times$	$\checkmark$
Gym-Anything ( 2026 )	K	$\checkmark$	$\checkmark$	$\times$	$\times$
CUA-Gym ( 2026a )	K/C	$\checkmark$	$\times$	$\checkmark$	$\checkmark$

Appendix

Table 6: Comparison of recent CUA task- and environment-synthesis pipelines. Synthesis grounding distinguishes external knowledge or application priors (K), task-specific environment or file inspection before the final instruction is proposed (E), and cross-instance coverage or capability memory (C). Application scaling denotes support for adding and configuring applications beyond an inherited benchmark; real artifact scaling denotes authentic or realistic file-backed inputs; rule-based evaluators programmatically inspect terminal states. A green check denotes explicit support, a red cross denotes no explicit support, and an orange check denotes preliminary but not yet systematic support.

Category	Count	Applications
Web, communication, and knowledge work	5	Chrome , ThunderBird , Notion , Zotero , Foliate
Office, documents, and publishing	9	LibreOffice Writer , LibreOffice Calc , LibreOffice Impress , WPS Presentation , WPS Writer , WPS Spreadsheets , PDF Arranger , TeXstudio , Text Editor
Programming, databases, and version control	5	PyCharm , VSCode , DBeaver , GitKraken , Meld
Graphics, diagrams, CAD, and photography	6	Gimp , Inkscape , Krita , Darktable , Draw.io , LibreCAD
Audio, video, music, and 3D media	9	Blender , Audacity , Handbrake , Mixxx , OBS Studio , MuseScore , Spotify , Shotcut , VLC
Scientific, engineering, geospatial, and network tools	8	Scilab , Wireshark , KiCad , Grass , ChimeraX , KAlgebra , Celestia , Google Earth

Appendix

Table 7: Functional classification of all 45 applications in the Ubuntu desktop infrastructure.

File group	Types
Documents, publishing, and office files	.doc , .docx , .pptx , .xlsx , .pdf , .epub , .txt , .tex
Code, projects, and structured data	Project Folders , .py , .sql , .csv , .sci , .xhb
Web, geospatial, and network artifacts	url , .html , .kmz , .pcapng
Images, vectors, diagrams, and design files	.png , .jpeg , .svg , .drawio , .kra
Audio, video, and music files	.mp3 , .wav , .mp4 , .mscz
CAD and 3D assets	.blend , .dxf

Appendix

Table 8: File and artifact types covered by the complete inventory of 45 applications.

Type	High-level group	Primary source
.docx	Office documents	superdoc-dev/docx-corpus
.pptx	Office documents	ForceLess/PPTAgent_parsed_data
.xlsx/.csv	Office documents	Multilingual-Multimodal-NLP/TableBench
.pdf	Documents	lmms-lab/DocVQA converted to PDF format
.png/.jpeg	Images	moonshotai/WorldVQA
.mp3	Audio	thang101020/musical_audio_cache

Appendix

Table 9: Primary sources of the file library by file type. Public upstream sources are written in repository-style form, including GitHub and dataset repositories where applicable. When an upstream source contains more files than needed for local deployment, we randomly sample a subset from that source. File types outside this table are covered through lightweight collection from small example libraries, application demo assets, simple conversions, or manually constructed seed files rather than large-scale harvesting.

Figure 5: Distributions of synthesized instructions, tool actions in successful filtered SFT samples, task-level evaluator composition, and outcomes of the full desktop SFT sample pool. Percentages are computed from the raw counts for each panel and displayed to two decimal places.

Parameter	Value
Training machine	$8\times$ Nvidia H200
Training framework	Megatron-SWIFT
Training type	Full-parameter fine-tuning
Tensor parallel size	4
Pipeline parallel size	1
Precision	bfloat16

Appendix

Table 10: OS-Helix SFT training parameters.

Parameter	Value
Basic Configuration
Base reinforcement learning framework	Slime
Total GPUs	$32\times$ Nvidia H200
Training GPUs	$16\times$ Nvidia H200
Rollout GPUs	$16\times$ Nvidia H200
Training steps	1,000

Appendix

Table 11: RL training parameters.

Agentic Frameworks
Method	Setting	Max Steps	Success Rate (%)
Method	Setting	Max Steps	Office	Web	Sys.	Code	Media	Util.	Avg.
Agent S3 w/ GPT-5 ( 2025 )	Hybrid	50	–	–	–	–	–	–	54.10
Agent S3 w/ GPT-5 ( 2025 )	Hybrid	100	–	–	–	–	–	–	56.60
OS-Symphony w/ GPT-5-Mini ( 2026a )	Hybrid	50	42.86	73.00	79.17	68.42	48.66	66.67	62.15
OS-Symphony w/ GPT-5 ( 2026a )	Hybrid	50	54.76	73.00	75.00	42.11	70.09	75.00	63.45

Appendix

Table 12: Additional results on WindowsAgentArena. Office includes Writer and Calc; Web includes Edge and Chrome; Sys. denotes Windows system tasks; Code includes VS Code; Media includes VLC; Util. includes Windows utilities. ♣ marks baselines reproduced by us.

Model	OSWorld-G	ScreenSpot-V2
Qwen3-VL-32B-Instruct	65.10	95.80
Qwen3-VL-32B-Instruct ♣	61.35	95.05
OS-Helix-32B	57.21	93.47

Appendix

Table 13: Grounding benchmark results on OSWorld-G and ScreenSpot-V2. ♣ represents the result reproduced by us.

Model	Setting	Step	OS	Office	Daily	Prof.	Work.	Avg.
Qwen3-VL-32B-Instruct ♣	GUI	50	41.67	31.54	51.23	44.90	15.57	34.17
Qwen3-VL-32B-Instruct ♣	Hybrid	50	50.00	33.25	39.31	38.78	18.78	32.68
Qwen3-VL-32B-Thinking ♣	GUI	50	58.33	35.84	54.44	55.10	18.71	39.08
Qwen3-VL-32B-Thinking ♣	Hybrid	50	62.50	40.97	49.81	53.85	27.95	42.88
OS-Helix-32B	Hybrid	50	66.67 ( $\uparrow$ 16.7)	52.91 ( $\uparrow$ 19.7)	56.14 ( $\uparrow$ 16.8)	51.02 ( $\uparrow$ 12.2)	38.54 ( $\uparrow$ 19.8)	50.56 ( $\uparrow$ 17.9)
OS-Helix-32B-Thinking	Hybrid	50	70.83 ( $\uparrow$ 8.3)	52.91 ( $\uparrow$ 11.9)	61.21 ( $\uparrow$ 11.4)	51.02 ( $\downarrow$ 2.8)	37.35 ( $\uparrow$ 9.4)	51.67 ( $\uparrow$ 8.8)

Appendix

Table 14: Backbone replacement analysis on OSWorld. ♣ marks baselines reproduced by us.

Training Recipe	Inference Context	OS	Office	Daily	Prof.	Work.	Avg.
Normal	Normal	66.67	52.91	56.14	51.02	38.54	50.56
Normal	$-$ CoT	54.17 ( $\downarrow$ 12.5)	28.98 ( $\downarrow$ 23.9)	53.58 ( $\downarrow$ 2.6)	38.77 ( $\downarrow$ 12.3)	31.43 ( $\downarrow$ 7.1)	37.82 ( $\downarrow$ 12.7)
$-$ CoT	Normal	50.00 ( $\downarrow$ 16.7)	35.55 ( $\downarrow$ 17.4)	46.99 ( $\downarrow$ 9.2)	38.77 ( $\downarrow$ 12.3)	28.87 ( $\downarrow$ 9.7)	37.74 ( $\downarrow$ 12.8)
$-$ History CoT	Normal	60.87 ( $\downarrow$ 5.8)	43.51 ( $\downarrow$ 9.4)	54.19 ( $\downarrow$ 2.0)	39.80 ( $\downarrow$ 11.2)	31.30 ( $\downarrow$ 7.2)	43.22 ( $\downarrow$ 7.3)
$+$ Failure Data	Normal	54.17 ( $\downarrow$ 12.5)	48.67 ( $\downarrow$ 4.2)	54.80 ( $\downarrow$ 1.3)	50.28 ( $\downarrow$ 0.7)	35.16 ( $\downarrow$ 3.4)	47.06 ( $\downarrow$ 3.5)

Appendix

Table 15: Training-recipe ablations on OSWorld with Qwen3-VL-32B-Instruct. “

-

CoT” denotes the removal of all CoT text while retaining action text only; “

-

History CoT” denotes the removal of historical CoT while retaining the current-step CoT and action text; “

+

Failure Data” denotes augmenting the SFT samples with partial rejection-sampled failure samples.

Metric	Comparison	$\Delta$ (pp)	95% CI
Primary score (Pass@1)	SFT $-$ Hybrid baseline	+16.21	[+10.75, +21.66]
Primary score (Pass@1)	RL $-$ Hybrid baseline	+19.89	[+14.24, +25.59]
Primary score (Pass@1)	RL $-$ SFT	+3.68	[+0.93, +8.33]
Pass@3	RL $-$ SFT	+5.56	[+2.24, +8.97]

Appendix

Table 16: Paired-bootstrap confidence intervals on OSWorld. We resample the 361 tasks with replacement for 100,000 replicates and recompute the paired mean-score difference.

Figure 6: Successful GUI–code coordination cases. The top case first uses GUI observation to infer the URL pattern from an opened lecture slide, then uses code to download the remaining weekly PDFs. The bottom case uses code to extract payment records from a spreadsheet and then returns to the GUI to fill the e-mail recipients.

Figure 7: Successful cross-modality solve-and-verify cases. The top case creates a spreadsheet chart with code and then detects from the GUI that one data series is missing before repairing it. The bottom case completes slide editing through GUI actions and then verifies the saved file with code.

Figure 8: Representative failure cases. Top: In a Calc-to-Writer transfer task requiring preservation of the original formatting, code-based conversion fails to retain the alternating row shading. The subsequent GUI observation identifies a mismatch but misattributes it to missing currency symbols, so the agent adds dollar signs instead of restoring the table style. Bottom: After correctly completing the first PowerPoint subgoal through GUI actions, the agent edits and saves the second with code. The code-side save does not include the unsaved GUI state, and selecting “Don’t Save” subsequently discards the earlier update.

Task ID	Outcome	Key Modality Flow	Handoff Purpose	Label
2c9fc0de	Success	Code only	Command-line completion; no handoff	A
e135df7c	Success	Code only	Programmatic conversion; no handoff	A
a503b07f	Success	Code only	Programmatic artifact creation; no handoff	A
26660ad1	Success	GUI $\rightarrow$ Code	Transfer measured network results to a text record	B
4c26e3f3	Success	GUI $\rightarrow$ Code	Identify the visual target, then extract and edit its asset	B
f5c13cdd	Success	Code $\rightarrow$ GUI	Transfer filtered recipients into an e-mail draft	B

Appendix

Table 17: Manual audit of 30 representative Work. trajectories. Task IDs are shown by unique eight-character prefixes; Key Modality Flow is a compressed summary of the principal stages relevant to the reported handoff; Handoff Purpose states what information, artifact, or application state is transferred or checked; Labels follow the definitions: A denotes single-modality execution without a substantive handoff, B denotes a task-grounded cross-modal handoff, C denotes cross-modal verification or repair, and D denotes a failed handoff.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Sep 4, 2026Haoting Shi, Wenhao Wang, Weicheng Fang +6Computer-Use Agents Tool-Using Agents
Shanghai Jiao Tong University · Zhejiang University
ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
Jun 22, 2026Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang +3Distribution Shift Robustness Computer-Use Agent Benchmarks
University of Pennsylvania · 2Peking University
OS-Wizard: Scaling Computer-Using Agents to Specialized Software via Knowledge-Guided Exploration
Oct 24, 2026Zhoumianze Liu, Zichen Ding, Bowen Yang +8Computer-Use Agents GUI Automation
Fudan University · Shanghai AI Laboratory · University of Science and Technology of China +4

Sep 4, 2026J/K move · Enter open · S save

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang+5

Shanghai Jiao Tong University · Zhejiang University

Computer-Use Agents Tool-Using Agents Computer-Use Agent Benchmarks

Open paper AskPDF arXiv

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.

Abstract

Computer-Use Agents

Explore similar work

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents

OS-Wizard: Scaling Computer-Using Agents to Specialized Software via Knowledge-Guided Exploration