Organizations: Fudan University · Shanghai AI Laboratory · University of Science and Technology of China · Shanghai Jiaotong University · East China Normal University · Nanjing University · The University of Hong Kong
Computer-using agents are increasingly capable of performing complex tasks in the digital world. However, high-quality interaction trajectories remain scarce for specialized desktop applications, where limited application knowledge, cumbersome GUI workflows, and incomplete exploration hinder automated collection. We introduce OS-Wizard, an automated trajectory collection framework. It combines tutorial-guided exploration with a hybrid GUI--Code action space, providing application-specific knowledge and simplifying GUI workflows through direct execution of application-native commands. The framework also checkpoints and revisits pivotal states during tree exploration to explore alternative workflows, complementing linear rollouts for broader coverage. We collect 10.9K trajectories comprising 54.9K state--action samples. Fine-tuning open-source base models on this corpus yields a best-performing model that improves performance on a benchmark of specialized desktop workflows by 58.2 percentage points to 64.1%, surpassing reported human performance of 59.8% and establishing a new state of the art. The fine-tuned model also improves performance on general desktop workflows using GUI actions only, suggesting that the collected supervision benefits general desktop tasks as well. Comparative experiments and ablations demonstrate the efficiency of trajectory collection and the importance of code actions.
Figures & tables
Figure 1: Three interaction paths for the same visualization task in ChimeraX. Top: A GUI-only workflow takes 10 actions to open the dialog, configure parameters, and apply the settings. Middle: Our domain-specific code interface executes the equivalent application-native command with a single action. Bottom: Entering the command through the GUI takes 3 actions to click the input, type the command, and press Enter.
Figure 2: Overview of trajectory collection in OS-Wizard. Left: Tutorials guide exploration with GUI and code actions. Center: Cold-start navigation builds an initial tree and checkpoints pivotal states. Right: Parallel tree exploration and linear rollouts collect trajectories. Bottom: Execution checks and reward-model verification ensure trajectory quality.
Figure 3: Left: Two-stage collection in the tree exploration component of OS-Wizard. Right: Snapshot-assisted state recovery (§ 3.4 ): find and restore the nearest saved ancestor, then replay all actions along the path to the target while checking screenshot similarity.
Model
Astron
Biochem
GIS
Alg
ATP
Doc
Overall
Human †
51.5
69.0
55.9
74.2
42.3
68.8
59.8
Base models
Qwen3-VL-8B ( Bai et al., 2025 )
12.1
10.3
2.9
9.7
0.0
0.0
6.5
Qwen3.5-27B ( Qwen Team, 2026a )
12.1
13.8
2.9
0.0
0.0
6.3
5.9
Proprietary models
GPT-5.5 ( OpenAI, 2026a )
18.2
41.4
11.8
9.7
3.8
31.3
18.3
Table 1: ScienceBoard success rates (%). † denotes results reported in the original paper; all other results are evaluated by us with a maximum step limit of 50. Results in the Ours block are reported as mean ± standard deviation over three evaluation runs, while results of SCALECUA-Qwen3.5-9B are reported as means. Best results are bolded and the second best results are underlined .
Model
Overall
Base models
Qwen3-VL-8B †
33.9
Qwen3.5-27B
39.1
Ours
OS-Wizard-Qwen3.5-27B
42.9
Figure 4: (a) OSWorld success rates (%). OS-Wizard-Qwen3.5-27B uses a maximum step limit of 50, and † denotes results reported in the original paper. (b) ScienceBoard ablations on domain-specific code supervision and trajectory collection strategies using Qwen3.5-27B. The code ablation randomly samples trajectories containing no code calls throughout, yielding 20K training samples.
Figure 5: Scaling and recovery-efficiency analysis of OS-Wizard. (a) ScienceBoard success rate as a function of training data size for Qwen3.5-4B, 9B, and 27B. (b) Average replay overhead of standard BFS, standard BFS with pivotal-checkpoint recovery, and standard BFS with pivotal checkpoints and continuous single-branch execution.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Action
Parameters
Execution semantics
key
keys (list)
Press a single key or a key combination, releasing the keys afterwards.
key_down
key (string)
Hold a key without releasing it.
key_up
key (string)
Release a held key.
type
text (string)
Enter text using pyautogui.typewrite .
mouse_move
coordinate
Move the cursor to the specified location.
left_click
coordinate
Click the left mouse button.
Appendix
Table 2: Model-facing action space in Qwen tool-call format and its environment execution semantics. Parameters are fields in the arguments of computer_use ; coordinate is an (x,y) pair.
Application
Syntax
Operation
ChimeraX
CODE <command>
Execute a native ChimeraX command.
GRASS GIS
CODE gcmd <cmdline>
Execute a console command and display its output in the Tool window.
CODE switch <location> <mapset>
Switch to the specified location and mapset under /home/user/grassdata .
CODE load <layer>
Load the named layer from the current mapset.
KAlgebra
CODE calc <expr>
Submit an expression to the Calculator.
CODE 2d <expr>
Add a two-dimensional curve.
Appendix
Table 3: CODE action definitions. Angle brackets denote arguments supplied by the agent. File operations are shared by Lean 4 and TeXstudio; TeXstudio additionally supports forwarding commands to a running ChimeraX or KAlgebra instance.
Application
Trajectories
Training samples
Mean steps
Celestia
1,120
7,401
6.61
ChimeraX
1,411
6,458
4.58
GRASS GIS
1,561
5,487
3.52
KAlgebra
1,636
7,841
4.79
Lean 4
1,486
7,546
5.08
TeXstudio
591
2,763
4.68
Appendix
Table 4: Application-level dataset statistics. Samples and mean trajectory lengths include terminal steps. The overall mean is weighted by trajectory count.
Setting
Qwen3.5-27B
Fine-tuning method
Full-parameter SFT
DeepSpeed ZeRO stage
2
Hardware
16× H200
Precision
BF16
Maximum sequence length
32,768
Per-device batch size
1
Appendix
Table 5: Training settings for the main Qwen3.5-27B experiment on the full corpus. Other experiments use similar training settings with minor adjustments. The effective global batch size is the per-device batch size multiplied by the gradient accumulation steps and the number of data-parallel workers.