Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions. However, maintaining strong performance on unseen applications remains challenging, as practical tasks often require long-horizon procedural knowledge and application-specific operation patterns. In addition, constructing executable tasks and collecting corresponding trajectories for training still heavily rely on human-annotated data, incurring substantial cost. In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training. During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs. To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction. During training, MAGE converts manuals into executable task-plan pairs and uses the plans to guide environment rollouts toward task-relevant states; the resulting trajectories are then used to optimize the policy without human-authored demonstrations. This unified design enables effective utilization of manuals for both task execution and training, reducing reliance on human-authored training data. Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments. We provide the code and reproduction scripts at https://github.com/ku-autoai/MAGE.
Figures & tables
Figure 1: Task execution with user manuals. MAGE retrieves relevant manual sections, then generates a retrieval-augmented plan, and finally performs GUI task execution conditioned on the plan.
Figure 2: Training CUAs with a user manual. MAGE uses manual-derived task-plan pairs to collect episodes for policy optimization.
Model
Size
Runs
Calc
Impress
Writer
GIMP
VSCode
Overall
o3
-
1
8.51
4.26
21.74
38.46
21.74
15.66
Claude-Sonnet-4.5
-
1
65.96
57.49
65.22
57.69
69.57
62.66
UI-TARS-DPO ( Qin et al., 2025 )
72B
1
12.77
25.45
43.48
61.54
47.83
33.11
OpenCUA ( Wang et al., 2025 )
32B
3
13.48 ± 1.23
38.28 ± 4.26
40.58 ± 2.51
69.23 ± 3.85
55.07 ± 10.94
38.75 ± 1.52
OpenCUA ( Wang et al., 2025 )
72B
1
31.91
44.66
60.87
76.92
73.91
52.40
GUI-Owl-1.5 ( Ye et al., 2025 )
32B
1
48.94
63.81
60.87
76.92
86.96
64.45
Table 1: Task success rates (%) on OSWorld applications under a 50-step budget. For models evaluated over multiple runs, we report mean ± standard deviation. Bold and underlined values indicate the highest and second-highest mean success rates, respectively, among systems with base CUAs of 8B parameters or fewer.
Table 4
Figure 3: Qualitative comparison of GUI execution on the same task before and after applying MAGE.
Calc
Impress
Writer
GIMP
Slack
Average
API Cost ($)
79.80
30.66
33.74
28.95
15.99
37.83
Episode Generation (h)
70.90
26.63
26.13
26.63
33.35
36.73
Policy Optimization (h)
24.24
8.44
8.73
4.24
3.80
9.89
Table 4: Adaptation cost of MAGE across applications using UI-TARS-1.5 as the base agent.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Manual
Tasks
Plan
Trajectory
Application
# of Sections
w/o Setup
All
Avg. Length
Trajectories
Avg. Steps
SR (%)
Calc
536
34
228
7.47
1,665
29.74
52.13
Impress
107
66
102
5.53
609
30.71
47.62
Writer
537
33
105
6.25
710
28.45
46.76
GIMP
1,870
21
100
8.31
536
35.72
35.26
VSCode
1,636
121
260
6.78
1,417
40.52
21.88
Appendix
Table 5: Data generation and execution statistics of MAGE across six applications.
Gemini 3.1 Pro Preview
Method
w/o Setup (18 tasks)
All (30 tasks)
UI-TARS
27.78±6.80
26.67±5.27
MAGE
56.67±6.09
49.33±11.88
Appendix
Table 6: Slack evaluation with and without setup tasks.
Table 9Table 10Table 11
Category
Action Primitive
Description
Required Arguments
Keyboard
key
Presses and releases the specified key(s).
keys (array)
key_down
Presses and holds the specified key(s).
keys (array)
key_up
Releases the specified key(s).
keys (array)
type
Types the specified text.
text (string)
Mouse
mouse_move
Moves the cursor to the specified coordinates.
coordinate (x,y)
left_click
Performs a left click.
coordinate (x,y)
Appendix
Table 13: Native action space of the EvoCUA agent.
Action
Definition
click(x, y)
Clicks at coordinates (x,y) .
left_double(x, y)
Double-clicks at coordinates (x,y) .
right_single(x, y)
Right-clicks at coordinates (x,y) .
drag(x1, y1, x2, y2)
Drags from (x1,y1) to (x2,y2) .
hotkey(key)
Presses the specified keyboard shortcut.
type(content)
Types the specified text content.
Appendix
Table 14: Action space of the UI-TARS-1.5 ( Qin et al., 2025 ) agent.