UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Zihan Ding; Longxu Dou; Qi Gao; Xiangwu Guo; Shengchao Hu; Zilong Huang; Zihang Jiang; Lei Ke

cs.AIAug 16, 2026

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Authors: Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke,

Organizations: Tencent Hy Frontier Team

Abstract

Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.

Figures & tables

Figure 1: Overview of the UI-Mate data flywheel. The pipeline jointly scales instruction curation, environment construction, trainable task generation, and rollout infrastructure. SFT data combines filtered agent rollouts with validated and repaired human trajectories; RL data is built from verifiable task bundles whose evaluators are refined using complementary probes and rollout feedback. Coverage and outcome diagnostics are fed back to rebalance capabilities and continuously improve the diversity, difficulty, and reliability of the training distribution.

Figure 2: Representative subset of the collected GUI capability tree. The hierarchy proceeds from application domains to fine-grained capabilities and representative workflows; sector sizes are schematic.

Figure 3: Agentic RL system of UI-Mate. Our RL Pipeline can be divided into rollout stage (left) and update stage (right). Adaptive sampler drives online sandbox rollouts under a fixed policy snapshot; completed trajectories are pooled per task group for outcome verification, optional process credit assignment, and asynchronous GRPO updates.

Figure 4: Demonstration representation. Offline (top), a human recording is annotated and segmented into subtasks with goals and verifiable completion criteria. Online (bottom), the agent follows the current subtask using live screenshots and advances upon completion.

Figure 5: A complete demo-workflow for a Godot task in which the agent adds timer-based bullet firing to a simple game. Panel (a) shows the context available to the agent at one point during execution. It includes the original task instruction, one user turn containing the demonstration, and the agent’s interaction history. Panel (b) breaks down the demonstration into a subtask checklist, the current subtask and its completion criterion, and key milestones.

Figure 7: OSWorkerBench benchmark construction pipeline. Candidate task instances are synthesized and validated through human review, evaluator tests, and pilot-agent rollouts. Forty-five targets are additionally paired with human-recorded demonstrations of related but non-identical tasks for the variant-demo setting. The 33 same-task strong-agent rollouts used in the self-demo evaluation are constructed separately and are not depicted.

Panel of Figure 8 — Figure 8: OSWorkerBench dataset overview. All statistics are computed over the canonical 100-task set. Application aliases and mock variants are normalized before counting.

Model	Size	Average Score
General-purpose multimodal models
Kimi-K2.6	1T-A32B	73.1
Qwen3.7-Plus	Closed-source	73.3
Qwen3.6-27B	27B	52.5 ∗
GPT-5.5	Closed-source	78.7
Claude Sonnet 5	Closed-source	81.2

Table 1: Overall average scores (%) on OSWorld-Verified. We report the overall average score of each model under the OSWorld evaluation protocol. Size denotes the publicly disclosed model scale when available. Higher is better. Our models are shaded. ∗ denotes a result reproduced by following the official OSWorld repository.

Model	Model Size	Progress Score (%)	Binary Success Rate (%)
		Overall (100)	Multi- App (49)	Long- Memory (67)	Overall (100)
Closed-weight models
GPT-5.6-Sol		87.67	65.31	67.16	71.00
Claude Opus 4.8		81.54	53.06	55.22	62.00
Claude Sonnet 5		81.46	42.86	50.75	55.00
Open-weight models

Table 2: Main results on OSWorkerBench under the instruction-only protocol. We report the average checkpoint-based progress score across all 100 tasks, followed by strict binary success rates on the overlapping requirement-based Multi-App (49 tasks) and Long-Memory (67 tasks) subsets and on the full benchmark. Models are grouped by weight availability, with model sizes omitted when they are not publicly disclosed. All values are percentages, and higher is better. The best baseline result in each column is shown in bold.

Panel of Figure 9 — Figure 9: Decision-turn distributions for GPT-5.6-Sol, Kimi-K2.6, and UI-Mate-27B on the same 100 OSWorkerBench tasks. A decision turn is one observation-to-action model response. A response containing multiple UI actions still counts as one turn because no new observation occurs within the batch; terminal responses and duplicate log records are excluded. Trajectory-length histograms using 20-turn bins show the prevalence of extended interactions, while empirical cumulative distributions summarize the full trajectory-length profiles; open triangles mark trajectories censored by the turn limit or technical failures. Each system has one rollout per task, and decision-turn counts characterize end-to-end interaction patterns rather than atomic UI-operation counts.

Model	Access / Size	Total Score (%)
General-purpose multimodal models
Kimi-K2.6	Open, 1T-A32B	63.3
Qwen3.6-27B	Open, 27B	47.1
Qwen3.5-9B	Open, 9B	37.5
GPT-5.5	Closed-source	70.4
Claude Sonnet 5	Closed-source	68.8

Table 3: Agent success rates on WindowsAgentArena. All values are percentages, and higher is better. Our models are shaded.

	No-Demo						Demo
Task	1	2	3	4	5	Avg.	1	2	3	4	5	Avg.	$\Delta$
godot-01	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	0.00
godot-02	71.43	64.29	78.57	78.57	57.14	70.00	71.43	71.43	64.29	71.43	64.29	68.57	-1.43
godot-03	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	0.00
godot-04	22.22	38.89	44.44	38.89	38.89	36.67	55.56	50.00	55.56	66.67	50.00	55.56	18.89
godot-05	76.47	82.35	82.35	82.35	76.47	80.00	64.71	76.47	70.59	76.47	88.24	75.29	-4.71

Table 4: GameDev performance with and without demonstrations on UI-Mate-27B. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while

\Delta

denotes the Demo - No-Demo difference in percentage points. Higher is better.

Evaluation Set	Condition	Performance (%)		Paired Gain (pp)
		Progress Score	Binary Success	$\Delta$ Progress	$\Delta$ Success
OSWorld (Subset-30)	Instruction only	40.27	–	–	–
	Demo guided	65.75	–	$+25.48$	–
OSWorkerBench (Subset-33)	Instruction only	67.85	17.17	–	–
	Demo guided	81.14	35.35	$+13.29$	$+18.18$

Table 5: Paired self-demo performance on the OSWorld-Subset30 and OSWorkerBench-Subset33 with UI-Mate-27B. Each target is evaluated under instruction-only and self-demo-guided conditions with identical initial states, budgets, and evaluators. The self-demo is a successful stronger-agent rollout of the same task. OSWorld-Subset results are averaged over five runs per target, while OSWorkerBench-Subset results are averaged over three runs per target. Paired gains are computed as self-demo guided minus instruction only and reported in percentage points (pp). Higher is better.

Figure 10: Demonstrations help the agent modify the correct underlying object. The reference demonstration and demo-conditioned run modify the reusable Player template, whereas the no-demo run modifies only one Player copy inside a game level. The two operations look similar in the interface, but only the former persists in the intended source object.

Figure 11: Demonstrations teach reliable viewport control for document-grounded reasoning. The reference demonstration and demo-conditioned run maximize the PDF viewer, locate and cross-check the critical values, and submit consistent form entries and supporting evidence. The no-demo run enlarges the document view without adequately repositioning the viewport, leading to incorrect visual/OCR readings and erroneous downstream decisions.

Task	Human Demo	No-Demo		Demo
	Steps	Avg. Steps	Score (%)	Avg. Steps	Score (%)
godot-01	35	29.8	100.00	30.0	100.00
godot-02	223	178.8	70.00	186.6	68.57
godot-03	202	173.8	100.00	131.0	100.00
godot-04	247	410.4	36.67	357.8	55.56
godot-05	380	580.0	80.00	362.6	75.29

Table 6: Trajectory lengths and task scores on GameDev on UI-Mate-27B. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Tasks requiring longer trajectories are generally associated with lower scores.

Figure 12: UI-Mate App’s four-layer architecture: the frontend controls runs and demonstrations, the backend configures requests, the harness drives the agent loop, and the bridge provides macOS observation and input.

Figure 13: UI-Mate interface and runtime latency. (a) Execution trace with an expanded step. (b) Per-step latency and breakdown in a real CUA run.

Where the model runs	Served as	Per step
Hosted gateway	Vendor’s own	varies
Self-hosted, one 8-GPU machine	27B, bf16	3–5 s
	27B, ours	2–3 s
On device, one Apple Silicon Mac	9B, 6-bit	10–20 s

Table 7: Serving arrangements and approximate step times on our devices.

ID	Dimension	Item	Type	Judging Method
S1	Trajectory-level	Whole-Task Completion	VLM	A Skeptical-Auditor prompt is fed to the VLM together with a multi-frame visual chain (initial $+$ uniformly sampled intermediate $+$ last- $K$ frames); the VLM returns a binary verdict on whether the full task is completed.
S2	Subtask-level	Per-Subtask Completion	VLM	For every subtask, its boundary frames and its self-report sentences are passed to the VLM, which independently returns $\mathrm{passed}(s)\!\in\!\{\text{True},\text{False}\}$ for each subtask $s\!\in\!\mathcal{S}$ .
S3	Demo-following	Subtask Completion Rate	Rule	Fraction of subtasks judged as passed by S2, computed as $\displaystyle\#\{\,s\!\in\!\mathcal{S}\mid\mathrm{passed}(s)=\text{True}\,\}\,/\,\|\mathcal{S}\|$ .
S4	Demo-following	Execution-Order Alignment	Rule	Longest-common-subsequence similarity between the agent’s actual visit order $\mathbf{o}_{\text{agent}}$ (extracted from the trajectory) and the reference visit order $\mathbf{o}_{\text{demo}}$ defined by the demo workflow, normalized as $\displaystyle\mathrm{LCS}(\mathbf{o}_{\text{agent}},\,\mathbf{o}_{\text{demo}})\,/\,\|\mathcal{S}\|$ .

Table 8: Scoring rules used to evaluate each rollout. Every rollout is assessed along three orthogonal dimensions.

K

is a small constant specifying how many trailing frames are appended to the visual chain used by the VLM in rule S1. Let

\mathcal{S}

be the set of subtasks defined by the workflow and

|\mathcal{S}|

its cardinality (i.e., the total number of subtasks).

ID	Item	Type	Repair Method
R1	Duplicate Reports	Rule	Detect adjacent report calls that share the same subtask_id and status , and collapse them into one.
R2	Unclosed Trajectory	Rule	If the trajectory terminates without a final report(subtask_id, done) , append one to close the last subtask.
R3	Missing Boundary	LLM	When a subtask transition is present in the action stream but the corresponding boundary report is absent, an LLM synthesizes the missing report from surrounding context.
R4	Step-Level Cleanup	Rule	Re-derive subtask boundaries from the report stream and rewrite each step’s subtask_id to align with the correct segment; in the same pass, strip any leading WAIT or empty action so that the first step is action-effective.

Table 9: Repair rules applied to rollouts. Each rule targets one action/report issue, executed by a deterministic program ( Rule ) or language model ( LLM ). Subtask reports refer to report(subtask_id, status) calls at boundaries.

	No-Demo						Demo
Task	1	2	3	4	5	Avg.	1	2	3	4	5	Avg.	$\Delta$
godot-01	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	0.00
godot-02	92.86	71.43	92.86	78.57	92.86	85.71	92.86	71.43	50.00	85.71	85.71	77.14	-8.57
godot-03	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	0.00
godot-04	78.95	52.63	84.21	63.16	68.42	69.47	94.74	84.21	68.42	84.21	84.21	83.16	13.68
godot-05	82.35	41.18	64.71	82.35	29.41	60.00	70.59	58.82	70.59	76.47	70.59	69.41	9.41

Table 12: Kimi K2.6 performance on GameDev without and with demonstrations. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while

\Delta

denotes the Demo - No-Demo difference in percentage points. Higher is better.

Task	Human Demo	No-Demo		Demo
	Steps	Avg. Steps	Score (%)	Avg. Steps	Score (%)
godot-01	35	22.2	100.00	31.8	100.00
godot-02	223	253.4	85.71	190.6	77.14
godot-03	202	103.0	100.00	145.8	100.00
godot-04	247	387.4	69.47	418.2	83.16
godot-05	380	450.6	60.00	445.0	69.41

Table 13: Trajectory lengths and task scores for Kimi K2.6 on GameDev. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Tasks requiring longer trajectories are generally associated with lower scores.

	No-Demo						Demo
Task	1	2	3	4	5	Avg.	1	2	3	4	5	Avg.	$\Delta$
chrome-01	100.00	0.00	0.00	0.00	100.00	40.00	100.00	100.00	100.00	100.00	100.00	100.00	60.00
chrome-02	0.00	0.00	0.00	0.00	0.00	0.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
chrome-03	0.00	0.00	0.00	0.00	0.00	0.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
chrome-04	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
gimp-01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	100.00	0.00	0.00	100.00	40.00	40.00

Table 14: OSWorld Per-task performance with and without demonstrations on UI-Mate-27B. Results are evaluated over five runs. The two Avg. columns report the corresponding mean success rates, while

\Delta

denotes the Demo - No-Demo difference in percentage points. Higher is better.

Task	Human Demo	No-Demo		Demo
	Steps	Avg. Steps	Score (%)	Avg. Steps	Score (%)
chrome-01	5	9.0	40.00	8.8	100.00
chrome-02	13	87.4	0.00	21.6	100.00
chrome-03	24	10.8	0.00	18.0	100.00
chrome-04	17	34.4	0.00	64.0	0.00
gimp-01	4	4.8	0.00	6.6	40.00

Table 15: Trajectory lengths and task scores with and without demonstrations on OSWorld-Subset 30 problems with UI-Mate-27B. Human Demo reports the number of steps in the human demonstration, while No-Demo and Demo report the average model steps and normalized task scores over five runs. Demonstrations make the model trajectory length track the human demonstration much more closely (rank correlation with the human step count rises from 0.66 to 0.86).

Task	No-Demo		Demo
	Avg. Steps	Score (%)	Avg. Steps	Score (%)
task-01	250.3	95.67	265.3	98.33
task-02	145.0	78.89	203.3	68.33
task-03	224.0	53.33	313.3	18.33
task-04	174.3	43.75	264.0	41.67
task-05	270.3	79.58	278.0	81.25

Table 16: Trajectory lengths and task scores for UI-Mate-27B on the 33-task OSWorkerBench self-demo subset. For each target, Demo provides a successful stronger-agent rollout of that same task (self-demo setting), represented by screenshots and action types without concrete pixel coordinates. No-Demo and Demo report the average model steps and normalized task scores over three runs under otherwise identical conditions. Demos improve average task score by

13.29

percentage points but also increase trajectory length, because these tasks involve repetitive or branching subtasks that the model may fail to infer from instructions alone and thus declares completion prematurely. The longer trajectories reflect higher task completeness rather than lower efficiency.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
Jul 30, 2026Hanzhang Zhou, Panrong Tong, Xu Zhang +13Mobile GUI Automation Computer-Use Agents
MAI-UI Team Alibaba Token Hub, Alibaba Group
MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
Oct 14, 2026Panrong Tong, Quyu Kong, Pengxiang Li +12RL for GUI Agents Mobile GUI Agents
Alibaba Token Hub, Alibaba Group
UI-Venus-2 Technical Report
Aug 27, 2026Venus Team, Zhuohan Cai, Haoxing Chen +28GUI Agents Multimodal Agents
Ant Group

Jul 30, 2026J/K move · Enter open · S save

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong+12

MAI-UI Team Alibaba Token Hub, Alibaba Group

Mobile GUI Automation Computer-Use Agents GUI Agents LLM Agent Self-Improvement Mobile GUI Agents LLM Agent Training+1

Open paper AskPDF arXiv

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

Abstract

GUI Agents

Explore similar work

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

UI-Venus-2 Technical Report