Learning Reliable GUI Agents under Imperfect Priors

Bo Han; Qianyi Wang; Shuai Liu; Xiong Zifan; Changqiao Wu; Yuanfa Li; Pengzhi Gao; Wei Liu

cs.LGSep 30, 2026

Learning Reliable GUI Agents under Imperfect Priors

Authors: Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan, Changqiao Wu, Yuanfa Li, Pengzhi Gao, Wei Liu, +4 more

Organizations: Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.

Abstract

GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.

Figures & tables

Figure 1: (a) GUI drift patterns across app versions; (b) task success drops on newer versions.

Figure 2: Overview of SAGE . Semantic-guided exploration builds a deduplicated UI graph $\mathcal{G}$ and task-aligned priors. Drift-aware operators $\{T_{k}\}$ construct $\mathcal{D}_{\Pi}$ with reliability labels. Teacher-generated thoughts supervise a LoRA-fine-tuned policy that assesses priors as Follow , Partial , or Ignore before acting.

$k$	Operator $T_{k}$	Empirical drift analog	Simulated failure mode	$r^{\star}$
$0$	identity	clean retrieval	none	Follow
$1$	surface rephrasing	control renaming	lexical/stylistic variation	Follow
$2$	sub-path deletion	step removal	missing intermediate operation	Partial
$3$	suffix branch replacement	entry-point/path shift	correct prefix, stale suffix	Partial
$4$	irrelevant-op insertion	ads, permissions, overlays	noisy or unrelated step	Partial
$5$	other-goal path substitution	wrong-goal retrieval	coherent but irrelevant prior	Ignore

Table 1: Drift-aware perturbation operators.

T_{2}

–

T_{4}

additionally annotate trustworthy edge indices.

T_{3},T_{5}

draw from other paths in the same

\mathcal{G}

, keeping noisy priors in-distribution .

Panel of Figure 3 — Figure 3: Exploration efficiency and downstream utility. (a) # of average deduplicated screens vs. steps. (b) # of average activities. (c) Subtask coverage (%) on MobileWorld / AndroidLab / AndroidWorld under the same budget; numbers in parentheses denote subtask counts.

Backbone	Method	Accuracy gain over no-prior (pp) $\uparrow$
Backbone	Method	Correct	Local	Noise	Omit	Rewrite	Other	Avg.
Qwen3-VL-2B	SFT	6.53	0.41	6.53	0.82	3.27	0.00	2.93
Qwen3-VL-2B	Ours	21.63 $\uparrow$ 15.10	4.90 $\uparrow$ 4.49	13.06 $\uparrow$ 6.53	9.80 $\uparrow$ 8.98	11.84 $\uparrow$ 8.57	8.98 $\uparrow$ 8.98	11.70 $\uparrow$ 8.77
Qwen3-VL-4B	SFT	18.37	3.67	11.02	4.49	8.57	4.49	8.44
Qwen3-VL-4B	Ours	21.63 $\uparrow$ 3.26	6.53 $\uparrow$ 2.86	13.47 $\uparrow$ 2.45	7.35 $\uparrow$ 2.86	13.47 $\uparrow$ 4.90	4.90 $\uparrow$ 0.41	11.23 $\uparrow$ 2.79
Qwen3-VL-8B	SFT	13.06	-3.67	3.67	-1.63	4.90	-1.63	2.45

Table 2: Accuracy gains (in percentage points) over the no-prior baseline across different prior conditions. Higher is better. Shaded rows denote our method; subscripts indicate absolute improvement over the SFT baseline (

\uparrow

\downarrow

). Avg. is the unweighted mean over all prior conditions.

Backbone	Method	Mean success rate gain over no-prior (pp) $\uparrow$
Backbone	Method	AndroidWorld	AndroidLab	MobileWorld
Qwen3-VL-4B	SFT	-6.80	-10.32	-6.20
Qwen3-VL-4B	Ours	-2.60 $\uparrow$ 4.20	0.31 $\uparrow$ 10.63	7.70 $\uparrow$ 13.90
GUI-OWL-1.5-4B	SFT	-7.66	-1.52	-2.34
GUI-OWL-1.5-4B	Ours	-1.70 $\uparrow$ 5.96	1.26 $\uparrow$ 2.78	5.22 $\uparrow$ 7.56

Table 3: Mean online task success rate gains (

\Delta

, in percentage points) over each method’s own no-prior baseline on 144 tasks across three benchmarks. Values are averaged equally over all prior conditions. Higher is better. Shaded rows denote our method; subscripts indicate absolute improvement over the SFT baseline (

\uparrow

\downarrow

Model	Train $\rightarrow$ Test	Corr.		Local		Noise		Other		Rewrite		Omit		Avg.
Model	Train $\rightarrow$ Test	Ours	SFT	Ours	SFT	Ours	SFT	Ours	SFT	Ours	SFT	Ours	SFT	Ours	SFT
Qwen3-2B	CMGUI $\rightarrow$ CUTG	13.06	6.94	6.53	-3.27	7.35	4.08	2.86	-4.08	4.90	2.45	4.90	0.41	6.60	1.09
	CMGUI $\rightarrow$ ChiM	15.50	3.10	12.40	0.00	15.50	3.88	1.55	-4.65	13.95	3.10	13.18	5.43	12.01	1.81
	CUTG $\rightarrow$ CMGUI	7.19	3.77	3.77	3.42	6.85	5.48	-4.79	-3.42	3.42	2.40	7.53	6.16	3.99	2.97
	CUTG $\rightarrow$ ChiM	10.85	2.33	8.53	0.00	6.98	2.33	-1.55	-1.55	7.75	3.88	10.85	7.75	7.23	2.46
	ChiM $\rightarrow$ CMGUI	6.51	0.00	5.48	-1.71	4.79	1.71	-0.34	-2.40	5.48	3.77	7.88	2.05	4.97	0.57

Table 4: Cross-dataset generalization under different prior conditions. Values are accuracy gains (in percentage points) over the no-prior setting. Each pair of columns reports Ours (shaded) and SFT ; the larger value in each pair is bolded .

Model	CUTG				CMGUI				ChiM-Nav				Average
Model	Ori.	SFT	Part- CoT	Ours	Ori.	SFT	Part- CoT	Ours	Ori.	SFT	Part- CoT	Ours	Ori.	SFT	Part- CoT	Ours
Qwen3-VL-2B	3.06	2.93	6.53	11.70	1.03	3.66	2.17	4.97	4.39	1.03	8.66	9.82	2.83	2.54	5.79	8.83
Qwen3-VL-4B	4.83	8.44	4.42	11.23	0.57	1.49	2.28	2.40	6.59	6.72	5.95	10.21	4.00	5.55	4.22	7.95
Qwen3-VL-8B	5.31	2.45	9.59	11.84	0.57	2.05	2.40	1.77	4.00	5.82	11.11	10.59	3.29	3.44	7.70	8.07
MAI-UI-2B	0.48	3.81	5.03	7.08	4.62	2.57	4.28	6.96	4.78	2.33	5.56	7.50	3.29	2.90	4.96	7.18
GELab-Zero-4B	0.54	1.63	3.33	7.08	2.00	1.43	2.28	1.66	3.36	3.75	10.21	13.18	1.97	2.27	5.27	7.31

Table 5: In-domain ablation. Values are mean accuracy gains (pp) over each variant’s own no-prior baseline, averaged equally across prior conditions. Average further averages over the three datasets. Ours is shaded; the best value in each group is bolded.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Direct	Full-CoT	No-CoT	Images
CUTG	1,789 / 245	10,734 / 1,470	10,734 / 1,470	707
CMGUI	1,996 / 292	11,976 / 1,752	11,976 / 1,752	2,257
ChiM-Nav	555 / 129	3,330 / 774	3,330 / 774	684
Total	4,340 / 666	26,040 / 3,996	26,040 / 3,996	3,648

Appendix

Table 6: Dataset statistics. Direct contains one sample per decision state, while Full-CoT and No-CoT expand each Direct sample into six knowledge conditions. Counts are reported as train/test samples; image counts denote the number of unique screen images used by each source.

Panel of Figure 4 — Figure 4: Exploration strategies for different interface types. Red boxes indicate the controls being explored. In (a), only two homogeneous video items are explored as they lead to similar video playback interfaces; in (b), diverse functional controls are explored comprehensively.

App	DroidBot		DroidAgent		LLM-Explorer		Humanoid		Ours
App	Scr.	Act.	Scr.	Act.	Scr.	Act.	Scr.	Act.	Scr.	Act.
Ctrip	32	12	11	8	60	1	16	4	219	19
Bilibili	34	9	30	16	56	10	54	15	196	45
NetEase Cloud Music	19	12	2	3	84	1	40	22	188	34
Amap	10	2	11	2	103	6	17	2	142	2
Douban	44	30	–	–	45	8	69	38	120	52

Appendix

Table 7: Per-app exploration results at the 600th exploration step. Scr. and Act. denote the number of deduplicated screens and distinct activities reached, respectively. The best value in each (app, metric) cell is bolded ; “–” marks runs that crashed before completion.

Thought:Action:Tool	Qwen3-VL-2B	Qwen3-VL-4B
$1{:}1{:}1$ (default)	+11.70	+11.23
$2{:}1{:}1$	+9.52	+8.91
$1{:}2{:}1$	+10.51	+9.13
$1{:}1{:}2$	+8.76	+10.05

Appendix

Table 8: Loss-weight sensitivity on CUTG. Values are mean accuracy gains (pp) over each model’s own no-prior baseline, averaged over six prior conditions. Bold indicates the best result per backbone.

Backbone	Follow F1	Partial F1	Ignore F1	Macro-F1
GELab-Zero-4B	92.09	64.08	64.97	73.71
MAI-UI-2B	91.40	56.57	60.24	69.41
Qwen3-VL-2B	91.33	54.55	65.54	70.47
Qwen3-VL-4B	90.00	56.41	58.62	68.34
Average	91.21	58.82	63.57	70.48

Appendix

Table 9: Reliability assessment on held-out real cross-version GUI drift. We report class-wise F1 and Macro-F1 (%) against manually annotated Follow / Partial / Ignore labels. Best results in each column are bolded.

Explore similar work

Xiaomi-GUI-0 Technical Report
Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Graphical User Interface Agents Multimodal Agents
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
May 28, 2026Tianpeng Bu, Xin Liu, Qihua Chen +7Graphical User Interface Agents
Alibaba Cloud Computing.
UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
May 28, 2026Yuxiang Chai, Han Xiao, Xinyu Fu +3Graphical User Interface Agents Graphical User Interface
1CUHK MMLab · 2Huawei Research · 3Shenzhen Loop Area Institute +1

Jun 30, 2026, cs.AIJ/K move · Enter open · S save

Xiaomi-GUI-0 Technical Report

Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao+28

Graphical User Interface Agents Multimodal Agents Device Technical Report

Open paper AskPDF arXiv

Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.

Abstract

Explore similar work

Xiaomi-GUI-0 Technical Report

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents