Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.
Figures & tables
Figure 1: Overview of the UI-Mate data flywheel. The pipeline jointly scales instruction curation, environment construction, trainable task generation, and rollout infrastructure. SFT data combines filtered agent rollouts with validated and repaired human trajectories; RL data is built from verifiable task bundles whose evaluators are refined using complementary probes and rollout feedback. Coverage and outcome diagnostics are fed back to rebalance capabilities and continuously improve the diversity, difficulty, and reliability of the training distribution.
Figure 2: Representative subset of the collected GUI capability tree. The hierarchy proceeds from application domains to fine-grained capabilities and representative workflows; sector sizes are schematic.
Figure 3: Agentic RL system of UI-Mate. Our RL Pipeline can be divided into rollout stage (left) and update stage (right). Adaptive sampler drives online sandbox rollouts under a fixed policy snapshot; completed trajectories are pooled per task group for outcome verification, optional process credit assignment, and asynchronous GRPO updates.
Figure 4: Demonstration representation. Offline (top), a human recording is annotated and segmented into subtasks with goals and verifiable completion criteria. Online (bottom), the agent follows the current subtask using live screenshots and advances upon completion.
Figure 5: A complete demo-workflow for a Godot task in which the agent adds timer-based bullet firing to a simple game. Panel (a) shows the context available to the agent at one point during execution. It includes the original task instruction, one user turn containing the demonstration, and the agent’s interaction history. Panel (b) breaks down the demonstration into a subtask checklist, the current subtask and its completion criterion, and key milestones.
Figure 7: OSWorkerBench benchmark construction pipeline. Candidate task instances are synthesized and validated through human review, evaluator tests, and pilot-agent rollouts. Forty-five targets are additionally paired with human-recorded demonstrations of related but non-identical tasks for the variant-demo setting. The 33 same-task strong-agent rollouts used in the self-demo evaluation are constructed separately and are not depicted.
Figure 8: OSWorkerBench dataset overview. All statistics are computed over the canonical 100-task set. Application aliases and mock variants are normalized before counting.
Model
Size
Average Score
General-purpose multimodal models
Kimi-K2.6
1T-A32B
73.1
Qwen3.7-Plus
Closed-source
73.3
Qwen3.6-27B
27B
52.5 ∗
GPT-5.5
Closed-source
78.7
Claude Sonnet 5
Closed-source
81.2
Table 1: Overall average scores (%) on OSWorld-Verified. We report the overall average score of each model under the OSWorld evaluation protocol. Size denotes the publicly disclosed model scale when available. Higher is better. Our models are shaded. ∗ denotes a result reproduced by following the official OSWorld repository.
Model
Model Size
Progress Score (%)
Binary Success Rate (%)
Overall (100)
Multi- App (49)
Long- Memory (67)
Overall (100)
Closed-weight models
GPT-5.6-Sol
87.67
65.31
67.16
71.00
Claude Opus 4.8
81.54
53.06
55.22
62.00
Claude Sonnet 5
81.46
42.86
50.75
55.00
Open-weight models
Table 2: Main results on OSWorkerBench under the instruction-only protocol. We report the average checkpoint-based progress score across all 100 tasks, followed by strict binary success rates on the overlapping requirement-based Multi-App (49 tasks) and Long-Memory (67 tasks) subsets and on the full benchmark. Models are grouped by weight availability, with model sizes omitted when they are not publicly disclosed. All values are percentages, and higher is better. The best baseline result in each column is shown in bold.
Figure 9: Decision-turn distributions for GPT-5.6-Sol, Kimi-K2.6, and UI-Mate-27B on the same 100 OSWorkerBench tasks. A decision turn is one observation-to-action model response. A response containing multiple UI actions still counts as one turn because no new observation occurs within the batch; terminal responses and duplicate log records are excluded. Trajectory-length histograms using 20-turn bins show the prevalence of extended interactions, while empirical cumulative distributions summarize the full trajectory-length profiles; open triangles mark trajectories censored by the turn limit or technical failures. Each system has one rollout per task, and decision-turn counts characterize end-to-end interaction patterns rather than atomic UI-operation counts.
Model
Access / Size
Total Score (%)
General-purpose multimodal models
Kimi-K2.6
Open, 1T-A32B
63.3
Qwen3.6-27B
Open, 27B
47.1
Qwen3.5-9B
Open, 9B
37.5
GPT-5.5
Closed-source
70.4
Claude Sonnet 5
Closed-source
68.8
Table 3: Agent success rates on WindowsAgentArena. All values are percentages, and higher is better. Our models are shaded.
No-Demo
Demo
Task
1
2
3
4
5
Avg.
1
2
3
4
5
Avg.
Δ
godot-01
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
0.00
godot-02
71.43
64.29
78.57
78.57
57.14
70.00
71.43
71.43
64.29
71.43
64.29
68.57
-1.43
godot-03
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
0.00
godot-04
22.22
38.89
44.44
38.89
38.89
36.67
55.56
50.00
55.56
66.67
50.00
55.56
18.89
godot-05
76.47
82.35
82.35
82.35
76.47
80.00
64.71
76.47
70.59
76.47
88.24
75.29
-4.71
Table 4: GameDev performance with and without demonstrations on UI-Mate-27B. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while Δ denotes the Demo - No-Demo difference in percentage points. Higher is better.
Evaluation Set
Condition
Performance (%)
Paired Gain (pp)
Progress Score
Binary Success
Δ Progress
Δ Success
OSWorld (Subset-30)
Instruction only
40.27
–
–
–
Demo guided
65.75
–
+25.48
–
OSWorkerBench (Subset-33)
Instruction only
67.85
17.17
–
–
Demo guided
81.14
35.35
+13.29
+18.18
Table 5: Paired self-demo performance on the OSWorld-Subset30 and OSWorkerBench-Subset33 with UI-Mate-27B. Each target is evaluated under instruction-only and self-demo-guided conditions with identical initial states, budgets, and evaluators. The self-demo is a successful stronger-agent rollout of the same task. OSWorld-Subset results are averaged over five runs per target, while OSWorkerBench-Subset results are averaged over three runs per target. Paired gains are computed as self-demo guided minus instruction only and reported in percentage points (pp). Higher is better.
Figure 10: Demonstrations help the agent modify the correct underlying object. The reference demonstration and demo-conditioned run modify the reusable Player template, whereas the no-demo run modifies only one Player copy inside a game level. The two operations look similar in the interface, but only the former persists in the intended source object.
Figure 11: Demonstrations teach reliable viewport control for document-grounded reasoning. The reference demonstration and demo-conditioned run maximize the PDF viewer, locate and cross-check the critical values, and submit consistent form entries and supporting evidence. The no-demo run enlarges the document view without adequately repositioning the viewport, leading to incorrect visual/OCR readings and erroneous downstream decisions.
Task
Human Demo
No-Demo
Demo
Steps
Avg. Steps
Score (%)
Avg. Steps
Score (%)
godot-01
35
29.8
100.00
30.0
100.00
godot-02
223
178.8
70.00
186.6
68.57
godot-03
202
173.8
100.00
131.0
100.00
godot-04
247
410.4
36.67
357.8
55.56
godot-05
380
580.0
80.00
362.6
75.29
Table 6: Trajectory lengths and task scores on GameDev on UI-Mate-27B. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Tasks requiring longer trajectories are generally associated with lower scores.
Figure 12: UI-Mate App’s four-layer architecture: the frontend controls runs and demonstrations, the backend configures requests, the harness drives the agent loop, and the bridge provides macOS observation and input.
Figure 13: UI-Mate interface and runtime latency. (a) Execution trace with an expanded step. (b) Per-step latency and breakdown in a real CUA run.
Where the model runs
Served as
Per step
Hosted gateway
Vendor’s own
varies
Self-hosted, one 8-GPU machine
27B, bf16
3–5 s
27B, ours
2–3 s
On device, one Apple Silicon Mac
9B, 6-bit
10–20 s
Table 7: Serving arrangements and approximate step times on our devices.
ID
Dimension
Item
Type
Judging Method
S1
Trajectory-level
Whole-Task Completion
VLM
A Skeptical-Auditor prompt is fed to the VLM together with a multi-frame visual chain (initial + uniformly sampled intermediate + last- K frames); the VLM returns a binary verdict on whether the full task is completed.
S2
Subtask-level
Per-Subtask Completion
VLM
For every subtask, its boundary frames and its self-report sentences are passed to the VLM, which independently returns passed(s)∈{True,False} for each subtask s∈S .
S3
Demo-following
Subtask Completion Rate
Rule
Fraction of subtasks judged as passed by S2, computed as #{s∈S∣passed(s)=True}/∣S∣ .
S4
Demo-following
Execution-Order Alignment
Rule
Longest-common-subsequence similarity between the agent’s actual visit order oagent (extracted from the trajectory) and the reference visit order odemo defined by the demo workflow, normalized as LCS(oagent,odemo)/∣S∣ .
Table 8: Scoring rules used to evaluate each rollout. Every rollout is assessed along three orthogonal dimensions. K is a small constant specifying how many trailing frames are appended to the visual chain used by the VLM in rule S1. Let S be the set of subtasks defined by the workflow and ∣S∣ its cardinality (i.e., the total number of subtasks).
ID
Item
Type
Repair Method
R1
Duplicate Reports
Rule
Detect adjacent report calls that share the same subtask_id and status , and collapse them into one.
R2
Unclosed Trajectory
Rule
If the trajectory terminates without a final report(subtask_id, done) , append one to close the last subtask.
R3
Missing Boundary
LLM
When a subtask transition is present in the action stream but the corresponding boundary report is absent, an LLM synthesizes the missing report from surrounding context.
R4
Step-Level Cleanup
Rule
Re-derive subtask boundaries from the report stream and rewrite each step’s subtask_id to align with the correct segment; in the same pass, strip any leading WAIT or empty action so that the first step is action-effective.
Table 9: Repair rules applied to rollouts. Each rule targets one action/report issue, executed by a deterministic program ( Rule ) or language model ( LLM ). Subtask reports refer to report(subtask_id, status) calls at boundaries.
No-Demo
Demo
Task
1
2
3
4
5
Avg.
1
2
3
4
5
Avg.
Δ
godot-01
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
0.00
godot-02
92.86
71.43
92.86
78.57
92.86
85.71
92.86
71.43
50.00
85.71
85.71
77.14
-8.57
godot-03
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
0.00
godot-04
78.95
52.63
84.21
63.16
68.42
69.47
94.74
84.21
68.42
84.21
84.21
83.16
13.68
godot-05
82.35
41.18
64.71
82.35
29.41
60.00
70.59
58.82
70.59
76.47
70.59
69.41
9.41
Table 12: Kimi K2.6 performance on GameDev without and with demonstrations. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while Δ denotes the Demo - No-Demo difference in percentage points. Higher is better.
Task
Human Demo
No-Demo
Demo
Steps
Avg. Steps
Score (%)
Avg. Steps
Score (%)
godot-01
35
22.2
100.00
31.8
100.00
godot-02
223
253.4
85.71
190.6
77.14
godot-03
202
103.0
100.00
145.8
100.00
godot-04
247
387.4
69.47
418.2
83.16
godot-05
380
450.6
60.00
445.0
69.41
Table 13: Trajectory lengths and task scores for Kimi K2.6 on GameDev. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Tasks requiring longer trajectories are generally associated with lower scores.
No-Demo
Demo
Task
1
2
3
4
5
Avg.
1
2
3
4
5
Avg.
Δ
chrome-01
100.00
0.00
0.00
0.00
100.00
40.00
100.00
100.00
100.00
100.00
100.00
100.00
60.00
chrome-02
0.00
0.00
0.00
0.00
0.00
0.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
chrome-03
0.00
0.00
0.00
0.00
0.00
0.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
chrome-04
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
gimp-01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
100.00
0.00
0.00
100.00
40.00
40.00
Table 14: OSWorld Per-task performance with and without demonstrations on UI-Mate-27B. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while Δ denotes the Demo - No-Demo difference in percentage points. Higher is better.
Task
Human Demo
No-Demo
Demo
Steps
Avg. Steps
Score (%)
Avg. Steps
Score (%)
chrome-01
5
9.0
40.00
8.8
100.00
chrome-02
13
87.4
0.00
21.6
100.00
chrome-03
24
10.8
0.00
18.0
100.00
chrome-04
17
34.4
0.00
64.0
0.00
gimp-01
4
4.8
0.00
6.6
40.00
Table 15: Trajectory lengths and task scores with and without demonstrations on OSWorld-Subset 30 problems with UI-Mate-27B. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Demonstrations make the model trajectory length track the human demonstration much more closely (rank correlation with the human step count rises from 0.66 to 0.86).
Task
No-Demo
Demo
Avg. Steps
Score (%)
Avg. Steps
Score (%)
task-01
250.3
95.67
265.3
98.33
task-02
145.0
78.89
203.3
68.33
task-03
224.0
53.33
313.3
18.33
task-04
174.3
43.75
264.0
41.67
task-05
270.3
79.58
278.0
81.25
Table 16: Trajectory lengths and task scores for UI-Mate-27B on the 33-task OSWorkerBench self-demo subset. For each target, Demo provides a successful stronger-agent rollout of that same task (self-demo setting), represented by screenshots and action types without concrete pixel coordinates. No-Demo and Demo report the average model steps and normalized task scores over three runs under otherwise identical conditions. Demos improve average task score by 13.29 percentage points but also increase trajectory length, because these tasks involve repetitive or branching subtasks that the model may fail to infer from instructions alone and thus declares completion prematurely. The longer trajectories reflect higher task completeness rather than lower efficiency.
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Online reinforcement learning (RL) with verifiable rewards has driven rapid progress in mathematics, coding, and tool use, but scaling this success to GUI agents remains challenging due to the high cost of interactive environments and the scarcity of realistic training tasks. We introduce MAI-Gym, an open all-in-one platform that unifies GUI agent evaluation, data collection, and online RL training across mobile, computer, and web environments. Through a shared rollout pipeline, MAI-Gym ensures consistent agent behavior across evaluation and training, supports ten established benchmarks, and scales to 20,000 parallel environment instances. Beyond scalable infrastructure, MAI-Gym enables automated synthesis of diverse GUI applications and challenging tasks, including resettable mocked apps and coherent cross-app worlds with execution-validated task verifiers. Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified. Importantly, RL in a mocked train-ticketing app transfers to its real-world counterpart, improving task success on a phys- ical phone from 28.8% to 48.1%. We release the framework, synthesis skills, mocked apps, 2,000 verified tasks, and the model checkpoint to support repro- ducible research and further advances in scalable GUI agent training.
Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.