GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

Rui Yang; Qianhui Wu; Zhaoyang Wang; Hanyang Chen; Ke Yang; Hao Cheng; Huaxiu Yao; Baolin Peng

cs.LGFeb 25, 2026

GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

Authors: Rui Yang, Qianhui Wu, Zhaoyang Wang, Hanyang Chen, Ke Yang, Hao Cheng, Huaxiu Yao, Baolin Peng, +3 more

Organizations: UIUC · Microsoft · UNC-Chapel Hill

Abstract

Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks. This gap stems from two limitations: a shortage of high-quality, action-aligned reasoning data, and the direct adoption of generic post-training pipelines that overlook the unique challenges of GUI agents. We identify two fundamental issues in these pipelines: (i) standard SFT with CoT reasoning often hurts grounding, and (ii) step-wise RLVR-tyle training faces partial verifiability, where multiple actions can be correct but only a single demonstrated action is used for verification. This makes offline step-wise metrics weak predictors of online task success. In this work, we present GUI-Libra, a tailored training recipe that addresses these challenges. First, to mitigate the scarcity of action-aligned reasoning data, we introduce a data construction and filtering pipeline and release a curated 81K GUI reasoning dataset. Second, to reconcile reasoning with grounding, we propose action-aware SFT that mixes reasoning-then-action and direct-action data and reweights tokens to emphasize action and grounding. Third, to stabilize RL under partial verifiability, we identify the overlooked importance of KL regularization in RLVR and show that a KL trust region is critical for improving offline-to-online predictability; we further introduce success-adaptive scaling to downweight unreliable negative gradients. Across diverse web and mobile benchmarks, GUI-Libra consistently improves both step-wise accuracy and end-to-end task completion. Our results suggest that carefully designed post-training and data curation can unlock significantly stronger task-solving capabilities without costly online data collection. We release our dataset, code, and models to facilitate further research on data-efficient post-training for reasoning-capable GUI agents.

Figures & tables

Figure 1: Overview of GUI-Libra. Using only a subset of existing open-source GUI trajectories, we tackle key limitations of prior training pipelines through action-aligned reasoning data curation, action-aware SFT, and conservative RL, yielding consistent gains on online benchmarks.

Name	Task Type	Reasoning	SFT	RL	Open Weights	Open Data	Open Code
OS-Atlas ( Wu et al., 2025b )		✗	✓	✗	✓	✓	✗
AGUVIS ( Xu et al., 2025c )		short	✓	✗	✓	✓	✓
UGround ( Gou et al., 2025 )		✗	✓	✗	✓	✓	✓
ScaleCUA ( Liu et al., 2025c )		long	✓	✗	✓	✓	✓
OpenCUA ( Wang et al., 2025d )		long	✓	✗	✓	✓	✓
UI-TARS ( Qin et al., 2025b )		short	✓	✓	✓	✗	✗

Table 1: Comparison of existing training recipes from different perspectives. The task type column indicates the training target of the released models: denotes GUI grounding, denotes GUI navigation (both online and offline), and denotes offline step-wise action prediction.

Dataset	Avg Thought Token Per Step	Total Steps	#Traj
MM-Mind2Web ( Zheng et al., 2024 )	0	8K	1K
AndroidControl ( Li et al., 2024 )	11	75K	14K
GUIAct ( Chen et al., 2025c )	0	17K	2.5k
AMEX ( Chai et al., 2025 )	0	35K	3K
GUI-Net-1M ( Zhang et al., 2025a )	37	4M	1M
ScaleCUA ( Liu et al., 2025c )	0	170K	19K

Table 2: Comparison of our GUI reasoning dataset with previous open-source datasets in web and mobile domains. Token statistics are computed using the Qwen2.5-VL-3B-Instruct tokenizer.

Figure 2: Example data format in GUI-Libra-81K . Each sample includes the current visual observation (screenshot) and textual context (system prompt, user instruction, and interaction history/previous actions). The model output is split into (1) a CoT reasoning trace and (2) a structured executable action (JSON), specifying the action type, a brief action description, the target element (if available), and action arguments such as text values or coordinates.

Figure 3: (a)(b) Data source distribution for SFT and RL. (c) Action type distribution of GUI-Libra-81K . (d) Comparison of step index distributions between our SFT and RL datasets.

Figure 4: (a) Grounding accuracy on ScreenSpot-v2 versus response length for base models and CoT-SFT models, showing that overly long responses correlate with degraded grounding. (b) Average grounding accuracy under different SFT strategies, where excessively long reasoning traces lead to a substantial drop.

Figure 5: Overall training framework of GUI-Libra: Stage 1 applies action-aware SFT with mixed supervision and token reweighting; Stage 2 performs KL-regularized GRPO with success-adaptive negative gradient scaling.

Figure 6: Limitations of current offline benchmarks. (a) Symbolic action history not in natural language in MM-Mind2Web, (b) Action type mismatch and (c) coordinate mismatch in AndroidControl.

	High Level		Low Level
Model	Pass@1	Pass@4	Pass@1	Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	57.0	66.3	78.4	85.4
GPT-4.1 + UGround-v1-7B	57.5	63.3	78.4	83.2
GPT-5-mini + UGround-v1-7B	52.8	58.8	77.1	83.2
GPT-5 + UGround-v1-7B	61.3	69.4	86.2	90.0

Table 3: Step accuracy Performance on AndroidControl-v2.

	Cross-Task		Cross-Website		Cross-Domain		Average
Model	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	35.7	38.9	33.9	37.6	39.1	42.2	36.2	39.6
GPT-4.1 + UGround-v1-7B	41.1	44.8	36.2	39.7	43.0	46.4	40.1	43.6
GPT-5-mini + UGround-v1-7B	44.2	48.0	40.4	44.2	45.8	48.1	43.5	46.7
GPT-5 + UGround-v1-7B	47.7	51.7	45.0	47.6	48.2	51.6	47.0	50.3

Table 4: Step accuracy Performance on Multimodal-Mind2Web-v2.

Figure 7: Trajectory Example of GUI-Libra-7B on AndroidWorld.

Table 5: Performance on the online benchmark AndroidWorld in 20 steps. ∗ denotes numbers reported by original papers. Left: Native Models (single VLM). Right: Agent Frameworks (

\geq

2 VLM modules).

	GitLab	MAP	Reddit	Shopping	ShoppingAdmin	Average
Native Models
Aguvis-72B ∗	-	-	-	-	-	5.8
Qwen2.5-VL-72B ∗	-	-	-	-	-	15.6
InternVL3.5-241B-A28B ∗	-	-	-	-	-	11.7
UI-TARS-1.5-7B ∗	-	-	-	-	-	20.8
UI-TARS-72B-DPO ∗	-	-	-	-	-	23.4

Table 6: Performance comparison on WebArena-Lite-v2 in 15 steps. ∗ denotes numbers reported in Liu et al. (2025c) .

Panel of Table 7 — Table 7: Performance comparison on Online-Mind2Web in 30 steps using o4-mini and WebJudge-7B as judges.

Figure 9: Comparison of training and evaluation metrics with and without KL regularization: (a) training reward, (b) policy entropy during training, (c) offline evaluation performance on AndroidControl-High, and (d) online evaluation performance on AndroidWorld.

Figure 10: (a) Correlation between offline and online performance. (b) Comparison of Pearson and Spearman correlations with and without KL regularization.

Figure 11: Ablation study of data filtering at the (a) SFT and (b) RL stages. Data filtering consistently improves both Pass@1 and Pass@4 performance across three benchmarks.

Model	MM-Mind2Web-v2		AC-v2 (High)		AC-v2 (Low)		AndroidWorld
	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
Base Model	23.4	28.3	36.4	50.8	71.1	79.2	3.5
SFT	28.5	36.9	45.7	59.8	73.1	83.2	5.2
SFT+ Mixed Data	30.2	42.0	45.5	64.8	72.6	85.4	11.3
ASFT	32.0	41.3	44.5	64.6	75.4	86.9	13.0
GUI-Libra w/o ASFT (KL_reg=0.0)	40.9	45.1	50.5	54.8	78.6	81.9	17.4

Table 9: Ablations study on ASFT, KL regularization, and reasoning across benchmarks with Qwen2.5-VL-3B as the base model.

Model	MM-Mind2Web-v2		AC-v2 (High)		AC-v2 (Low)		AndroidWorld	WebArena-Lite-v2
	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
GUI-Libra-4B (w/o SNGS)	49.1	55.1	59.8	69.9	87.7	92.0	39.1	22.2
GUI-Libra-4B (w/ SNGS)	50.0	55.6	62.3	68.6	86.4	93.0	42.6	24.4

Table 10: Comparison between GUI-Libra w/ SNGS and w/o SNGS across benchmarks.

	SS-v2	SS-Pro	MM-Mind2Web-v2	AC-v2 (high)	AC-v2 (low)
Qwen3-VL-4B	91.7	52.8	41.2	49.3	78.9
GUI-Libra-4B	92.3	54.3	49.1	59.8	87.7
GUI-Libra-4B + Mix Grounding 20k	94.6 $\uparrow$ (+2.3)	61.4 $\uparrow$ (+7.1)	43.9 $\downarrow$ (-5.2)	61.3 $\uparrow$ (+1.5)	84.9 $\downarrow$ (-2.8)
Qwen3-VL-8B	92.1	52.7	43.8	54.8	77.6
GUI-Libra-8B	90.7	54.1	50.3	65.6	88.7
GUI-Libra-8B + Mix Grounding 20k	94.8 $\uparrow$ (+4.1)	59.9 $\uparrow$ (+5.8)	49.5 $\downarrow$ (-0.8)	61.7 $\downarrow$ (-3.9)	86.4 $\downarrow$ (-2.3)

Table 11: Effect of mixing grounding data into RL training on ScreenSpot-v2 (SS-v2), ScreenSpot-Pro (SS-Pro), MM-Mind2Web-v2, and AndroidControl-v2 (AC-v2). We report Pass@1 on all benchmarks. Green arrows indicate performance gains, and red arrows indicate degradations relative to the corresponding GUI-Libra models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

action_type	action_target	value	point_2d	details
Answer	None	answer text	[-100,-100]	Return the final answer to the user’s question.
Click	element description	None	[x,y]	Tap/click a specific UI element and provide its coordinates.
Select	element description	option value	[-100,-100]	Select an item in a list or dropdown menu.
LongPress	element description	None	[x,y]	Press-and-hold on a UI element (mobile only) and provide its coordinates.
Write	element description or None	input text	[x,y] or [-100,-100]	Enter text into a specific input field; if point_2d is [-100,-100] , type at the current focus.
KeyboardPress	None	key name (e.g., enter )	[-100,-100]	Press a specific key on the keyboard.

Appendix

Table 12: Unified action space. Each action is a tuple (action_type, action_target, value, point_2d) .

Category	Configuration
Backbone	Qwen2.5-VL / Qwen3-VL
Training Strategy	Full-parameter fine-tuning
Epochs	1
Learning Rate	$1\times 10^{-5}$
Scheduler	Cosine
Warmup Ratio	0.01

Appendix

Table 13: SFT configuration used in our experiments.

Category	Configuration
RL Training Framework	VERL
Distributed Training Backend	FSDP
Inference Engine	vLLM
Backbone	Qwen2.5-VL, Qwen3-VL
Rollout
Rollout Samples per Prompt	8

Appendix

Table 14: RL configuration for our experiments.

Figure 12: RL for grounding exhibits stable improvements and strong cross-benchmark predictability. (a) ScreenSpot-V2 and (b) ScreenSpot-Pro performance over RL training. (c) Correlation between the two benchmark scores across checkpoints. (d) Pearson and Spearman correlations with and without KL regularization.

Figure 13: Performance comparison of different models for reasoning generation. All models are used to augment the same 30K web samples from AGUVIS and are fine-tuned on the same 3B base model. This controlled evaluation reveals substantial performance differences among generator models.

Model	MM-Mind2Web-v2		AC-v2 (High)		AC-v2 (Low)		AndroidWorld
	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
GUI-Libra ( $\lambda_{g}=0.75$ )	42.2	48.5	54.5	64.1	86.7	90.2	19.1
GUI-Libra ( $\lambda_{g}=0.9$ )	42.7	48.1	52.3	61.1	87.4	90.7	20.0
GUI-Libra	42.7	50.2	55.8	65.8	89.5	91.5	25.2

Appendix

Table 15: Ablations study of negative scaling strategy on 3B base model.

	High Level		Low Level
Model	Pass@1	Pass@4	Pass@1	Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	58.7	67.3	83.4	88.3
GPT-4.1 + UGround-v1-7B	60.5	64.6	83.0	86.1
GPT-5-mini + UGround-v1-7B	61.9	66.8	83.9	87.9
GPT-5 + UGround-v1-7B	74.4	82.1	93.7	94.6

Appendix

Table 16: Grounding Accuracy on AndroidControl-v2.

	High Level		Low Level
Model	Pass@1	Pass@4	Pass@1	Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	73.9	82.7	89.2	93.7
GPT-4.1 + UGround-v1-7B	72.9	78.9	89.5	93.0
GPT-5-mini + UGround-v1-7B	70.9	77.4	89.7	93.5
GPT-5 + UGround-v1-7B	74.1	78.9	88.4	91.5

Appendix

Table 17: Type Accuracy on AndroidControl-v2.

	Cross-Task		Cross-Website		Cross-Domain		Average
Model	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4	Pass@1	Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	42.0	44.0	40.6	42.9	44.9	47.2	42.5	44.7
GPT-4.1 + UGround-v1-7B	47.4	49.3	43.5	45.5	48.7	51.4	46.5	48.7
GPT-5-mini + UGround-v1-7B	52.5	53.1	50.7	51.6	53.6	54.3	52.3	53.0
GPT-5 + UGround-v1-7B	55.3	56.6	53.1	53.4	55.8	57.3	54.7	55.8

Appendix

Table 18: Grounding accuracy (%) on Multimodal-Mind2Web-v2.

	Cross-Task		Cross-Website		Cross-Domain		Average
Model	Best@1	Best@4	Best@1	Best@4	Best@1	Best@4	Best@1	Best@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B	70.1	78.9	70.9	79.6	67.7	75.7	69.6	78.1
GPT-4.1 + UGround-v1-7B	72.7	81.6	71.1	81.0	71.1	79.4	71.6	80.7
GPT-5-mini + UGround-v1-7B	80.7	88.8	77.3	86.8	77.9	85.3	78.6	87.0
GPT-5 + UGround-v1-7B	79.7	89.2	79.6	88.9	79.1	88.2	79.5	88.8

Appendix

Table 19: Operation F1 score on Multimodal-Mind2Web-v2.

Figure 14: Trajectory Example of GUI-Libra-7B for Task 18 ExpenseDeleteMultiple: Delete the following expenses from pro expense: School Supplies, Religious, Flight Tickets.

Figure 15: Trajectory Example of the base model Qwen2.5-VL-7B-Instruct for Task 18 ExpenseDeleteMultiple: Delete the following expenses from pro expense: School Supplies, Religious, Flight Tickets.

Figure 16: Trajectory Example of GUI-Libra-4B for WebArena-Lite-v2 Task: Follow [’lahwaacz’, ’Koushik’, ’Vinta Chen’] on Gitlab.

Figure 17: Trajectory Example of Qwen3-VL-4B-Instruct for WebArena-Lite-v2 Task: Follow [’lahwaacz’, ’Koushik’, ’Vinta Chen’] on Gitlab.

Explore similar work

Learning Reliable GUI Agents under Imperfect Priors
Sep 30, 2026Bo Han, Qianyi Wang, Shuai Liu +9GUI Agents LLM Agent Training
Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.
GUI Agents with Reinforcement Learning: Toward Digital Inhabitants
Apr 30, 2026Junan Hu, Jian Liu, Jingxiang Lai +6Reinforcement Learning RL for GUI Agents
1Shandong University · 2The Hong Kong University of Science and Technology · 3The Hong Kong University +2
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University

Sep 30, 2026J/K move · Enter open · S save

Learning Reliable GUI Agents under Imperfect Priors

Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan+8

Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.

GUI Agents LLM Agent Training

Open paper AskPDF arXiv

GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.

Abstract

Explore similar work

Learning Reliable GUI Agents under Imperfect Priors

GUI Agents with Reinforcement Learning: Toward Digital Inhabitants

GUI-C2^22: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning