Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks. This gap stems from two limitations: a shortage of high-quality, action-aligned reasoning data, and the direct adoption of generic post-training pipelines that overlook the unique challenges of GUI agents. We identify two fundamental issues in these pipelines: (i) standard SFT with CoT reasoning often hurts grounding, and (ii) step-wise RLVR-tyle training faces partial verifiability, where multiple actions can be correct but only a single demonstrated action is used for verification. This makes offline step-wise metrics weak predictors of online task success. In this work, we present GUI-Libra, a tailored training recipe that addresses these challenges. First, to mitigate the scarcity of action-aligned reasoning data, we introduce a data construction and filtering pipeline and release a curated 81K GUI reasoning dataset. Second, to reconcile reasoning with grounding, we propose action-aware SFT that mixes reasoning-then-action and direct-action data and reweights tokens to emphasize action and grounding. Third, to stabilize RL under partial verifiability, we identify the overlooked importance of KL regularization in RLVR and show that a KL trust region is critical for improving offline-to-online predictability; we further introduce success-adaptive scaling to downweight unreliable negative gradients. Across diverse web and mobile benchmarks, GUI-Libra consistently improves both step-wise accuracy and end-to-end task completion. Our results suggest that carefully designed post-training and data curation can unlock significantly stronger task-solving capabilities without costly online data collection. We release our dataset, code, and models to facilitate further research on data-efficient post-training for reasoning-capable GUI agents.
Figures & tables
Figure 1: Overview of GUI-Libra. Using only a subset of existing open-source GUI trajectories, we tackle key limitations of prior training pipelines through action-aligned reasoning data curation, action-aware SFT, and conservative RL, yielding consistent gains on online benchmarks.
Name
Task Type
Reasoning
SFT
RL
Open Weights
Open Data
Open Code
OS-Atlas ( Wu et al., 2025b )
✗
✓
✗
✓
✓
✗
AGUVIS ( Xu et al., 2025c )
short
✓
✗
✓
✓
✓
UGround ( Gou et al., 2025 )
✗
✓
✗
✓
✓
✓
ScaleCUA ( Liu et al., 2025c )
long
✓
✗
✓
✓
✓
OpenCUA ( Wang et al., 2025d )
long
✓
✗
✓
✓
✓
UI-TARS ( Qin et al., 2025b )
short
✓
✓
✓
✗
✗
Table 1: Comparison of existing training recipes from different perspectives. The task type column indicates the training target of the released models: denotes GUI grounding, denotes GUI navigation (both online and offline), and denotes offline step-wise action prediction.
Dataset
Avg Thought Token Per Step
Total Steps
#Traj
MM-Mind2Web ( Zheng et al., 2024 )
0
8K
1K
AndroidControl ( Li et al., 2024 )
11
75K
14K
GUIAct ( Chen et al., 2025c )
0
17K
2.5k
AMEX ( Chai et al., 2025 )
0
35K
3K
GUI-Net-1M ( Zhang et al., 2025a )
37
4M
1M
ScaleCUA ( Liu et al., 2025c )
0
170K
19K
Table 2: Comparison of our GUI reasoning dataset with previous open-source datasets in web and mobile domains. Token statistics are computed using the Qwen2.5-VL-3B-Instruct tokenizer.
Figure 2: Example data format in GUI-Libra-81K . Each sample includes the current visual observation (screenshot) and textual context (system prompt, user instruction, and interaction history/previous actions). The model output is split into (1) a CoT reasoning trace and (2) a structured executable action (JSON), specifying the action type, a brief action description, the target element (if available), and action arguments such as text values or coordinates.
Figure 3: (a)(b) Data source distribution for SFT and RL. (c) Action type distribution of GUI-Libra-81K . (d) Comparison of step index distributions between our SFT and RL datasets.
Figure 4: (a) Grounding accuracy on ScreenSpot-v2 versus response length for base models and CoT-SFT models, showing that overly long responses correlate with degraded grounding. (b) Average grounding accuracy under different SFT strategies, where excessively long reasoning traces lead to a substantial drop.
Figure 5: Overall training framework of GUI-Libra: Stage 1 applies action-aware SFT with mixed supervision and token reweighting; Stage 2 performs KL-regularized GRPO with success-adaptive negative gradient scaling.
Figure 6: Limitations of current offline benchmarks. (a) Symbolic action history not in natural language in MM-Mind2Web, (b) Action type mismatch and (c) coordinate mismatch in AndroidControl.
High Level
Low Level
Model
Pass@1
Pass@4
Pass@1
Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
57.0
66.3
78.4
85.4
GPT-4.1 + UGround-v1-7B
57.5
63.3
78.4
83.2
GPT-5-mini + UGround-v1-7B
52.8
58.8
77.1
83.2
GPT-5 + UGround-v1-7B
61.3
69.4
86.2
90.0
Table 3: Step accuracy Performance on AndroidControl-v2.
Cross-Task
Cross-Website
Cross-Domain
Average
Model
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
35.7
38.9
33.9
37.6
39.1
42.2
36.2
39.6
GPT-4.1 + UGround-v1-7B
41.1
44.8
36.2
39.7
43.0
46.4
40.1
43.6
GPT-5-mini + UGround-v1-7B
44.2
48.0
40.4
44.2
45.8
48.1
43.5
46.7
GPT-5 + UGround-v1-7B
47.7
51.7
45.0
47.6
48.2
51.6
47.0
50.3
Table 4: Step accuracy Performance on Multimodal-Mind2Web-v2.
Figure 7: Trajectory Example of GUI-Libra-7B on AndroidWorld.
Table 5: Performance on the online benchmark AndroidWorld in 20 steps. ∗ denotes numbers reported by original papers. Left: Native Models (single VLM). Right: Agent Frameworks ( ≥ 2 VLM modules).
GitLab
MAP
Reddit
Shopping
ShoppingAdmin
Average
Native Models
Aguvis-72B ∗
-
-
-
-
-
5.8
Qwen2.5-VL-72B ∗
-
-
-
-
-
15.6
InternVL3.5-241B-A28B ∗
-
-
-
-
-
11.7
UI-TARS-1.5-7B ∗
-
-
-
-
-
20.8
UI-TARS-72B-DPO ∗
-
-
-
-
-
23.4
Table 6: Performance comparison on WebArena-Lite-v2 in 15 steps. ∗ denotes numbers reported in Liu et al. (2025c) .
Table 7: Performance comparison on Online-Mind2Web in 30 steps using o4-mini and WebJudge-7B as judges.
Figure 15
Figure 9: Comparison of training and evaluation metrics with and without KL regularization: (a) training reward, (b) policy entropy during training, (c) offline evaluation performance on AndroidControl-High, and (d) online evaluation performance on AndroidWorld.
Figure 10: (a) Correlation between offline and online performance. (b) Comparison of Pearson and Spearman correlations with and without KL regularization.
Figure 11: Ablation study of data filtering at the (a) SFT and (b) RL stages. Data filtering consistently improves both Pass@1 and Pass@4 performance across three benchmarks.
Model
MM-Mind2Web-v2
AC-v2 (High)
AC-v2 (Low)
AndroidWorld
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
Base Model
23.4
28.3
36.4
50.8
71.1
79.2
3.5
SFT
28.5
36.9
45.7
59.8
73.1
83.2
5.2
SFT+ Mixed Data
30.2
42.0
45.5
64.8
72.6
85.4
11.3
ASFT
32.0
41.3
44.5
64.6
75.4
86.9
13.0
GUI-Libra w/o ASFT (KL_reg=0.0)
40.9
45.1
50.5
54.8
78.6
81.9
17.4
Table 9: Ablations study on ASFT, KL regularization, and reasoning across benchmarks with Qwen2.5-VL-3B as the base model.
Model
MM-Mind2Web-v2
AC-v2 (High)
AC-v2 (Low)
AndroidWorld
WebArena-Lite-v2
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
GUI-Libra-4B (w/o SNGS)
49.1
55.1
59.8
69.9
87.7
92.0
39.1
22.2
GUI-Libra-4B (w/ SNGS)
50.0
55.6
62.3
68.6
86.4
93.0
42.6
24.4
Table 10: Comparison between GUI-Libra w/ SNGS and w/o SNGS across benchmarks.
SS-v2
SS-Pro
MM-Mind2Web-v2
AC-v2 (high)
AC-v2 (low)
Qwen3-VL-4B
91.7
52.8
41.2
49.3
78.9
GUI-Libra-4B
92.3
54.3
49.1
59.8
87.7
GUI-Libra-4B + Mix Grounding 20k
94.6 ↑ (+2.3)
61.4 ↑ (+7.1)
43.9 ↓ (-5.2)
61.3 ↑ (+1.5)
84.9 ↓ (-2.8)
Qwen3-VL-8B
92.1
52.7
43.8
54.8
77.6
GUI-Libra-8B
90.7
54.1
50.3
65.6
88.7
GUI-Libra-8B + Mix Grounding 20k
94.8 ↑ (+4.1)
59.9 ↑ (+5.8)
49.5 ↓ (-0.8)
61.7 ↓ (-3.9)
86.4 ↓ (-2.3)
Table 11: Effect of mixing grounding data into RL training on ScreenSpot-v2 (SS-v2), ScreenSpot-Pro (SS-Pro), MM-Mind2Web-v2, and AndroidControl-v2 (AC-v2). We report Pass@1 on all benchmarks. Green arrows indicate performance gains, and red arrows indicate degradations relative to the corresponding GUI-Libra models.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
action_type
action_target
value
point_2d
details
Answer
None
answer text
[-100,-100]
Return the final answer to the user’s question.
Click
element description
None
[x,y]
Tap/click a specific UI element and provide its coordinates.
Select
element description
option value
[-100,-100]
Select an item in a list or dropdown menu.
LongPress
element description
None
[x,y]
Press-and-hold on a UI element (mobile only) and provide its coordinates.
Write
element description or None
input text
[x,y] or [-100,-100]
Enter text into a specific input field; if point_2d is [-100,-100] , type at the current focus.
KeyboardPress
None
key name (e.g., enter )
[-100,-100]
Press a specific key on the keyboard.
Appendix
Table 12: Unified action space. Each action is a tuple (action_type, action_target, value, point_2d) .
Category
Configuration
Backbone
Qwen2.5-VL / Qwen3-VL
Training Strategy
Full-parameter fine-tuning
Epochs
1
Learning Rate
1×10−5
Scheduler
Cosine
Warmup Ratio
0.01
Appendix
Table 13: SFT configuration used in our experiments.
Category
Configuration
RL Training Framework
VERL
Distributed Training Backend
FSDP
Inference Engine
vLLM
Backbone
Qwen2.5-VL, Qwen3-VL
Rollout
Rollout Samples per Prompt
8
Appendix
Table 14: RL configuration for our experiments.
Figure 12: RL for grounding exhibits stable improvements and strong cross-benchmark predictability. (a) ScreenSpot-V2 and (b) ScreenSpot-Pro performance over RL training. (c) Correlation between the two benchmark scores across checkpoints. (d) Pearson and Spearman correlations with and without KL regularization.
Figure 13: Performance comparison of different models for reasoning generation. All models are used to augment the same 30K web samples from AGUVIS and are fine-tuned on the same 3B base model. This controlled evaluation reveals substantial performance differences among generator models.
Model
MM-Mind2Web-v2
AC-v2 (High)
AC-v2 (Low)
AndroidWorld
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
GUI-Libra ( λg=0.75 )
42.2
48.5
54.5
64.1
86.7
90.2
19.1
GUI-Libra ( λg=0.9 )
42.7
48.1
52.3
61.1
87.4
90.7
20.0
GUI-Libra
42.7
50.2
55.8
65.8
89.5
91.5
25.2
Appendix
Table 15: Ablations study of negative scaling strategy on 3B base model.
High Level
Low Level
Model
Pass@1
Pass@4
Pass@1
Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
58.7
67.3
83.4
88.3
GPT-4.1 + UGround-v1-7B
60.5
64.6
83.0
86.1
GPT-5-mini + UGround-v1-7B
61.9
66.8
83.9
87.9
GPT-5 + UGround-v1-7B
74.4
82.1
93.7
94.6
Appendix
Table 16: Grounding Accuracy on AndroidControl-v2.
High Level
Low Level
Model
Pass@1
Pass@4
Pass@1
Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
73.9
82.7
89.2
93.7
GPT-4.1 + UGround-v1-7B
72.9
78.9
89.5
93.0
GPT-5-mini + UGround-v1-7B
70.9
77.4
89.7
93.5
GPT-5 + UGround-v1-7B
74.1
78.9
88.4
91.5
Appendix
Table 17: Type Accuracy on AndroidControl-v2.
Cross-Task
Cross-Website
Cross-Domain
Average
Model
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
Pass@1
Pass@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
42.0
44.0
40.6
42.9
44.9
47.2
42.5
44.7
GPT-4.1 + UGround-v1-7B
47.4
49.3
43.5
45.5
48.7
51.4
46.5
48.7
GPT-5-mini + UGround-v1-7B
52.5
53.1
50.7
51.6
53.6
54.3
52.3
53.0
GPT-5 + UGround-v1-7B
55.3
56.6
53.1
53.4
55.8
57.3
54.7
55.8
Appendix
Table 18: Grounding accuracy (%) on Multimodal-Mind2Web-v2.
Cross-Task
Cross-Website
Cross-Domain
Average
Model
Best@1
Best@4
Best@1
Best@4
Best@1
Best@4
Best@1
Best@4
Proprietary Models with SeeAct-V Framework
GPT-4o + UGround-v1-7B
70.1
78.9
70.9
79.6
67.7
75.7
69.6
78.1
GPT-4.1 + UGround-v1-7B
72.7
81.6
71.1
81.0
71.1
79.4
71.6
80.7
GPT-5-mini + UGround-v1-7B
80.7
88.8
77.3
86.8
77.9
85.3
78.6
87.0
GPT-5 + UGround-v1-7B
79.7
89.2
79.6
88.9
79.1
88.2
79.5
88.8
Appendix
Table 19: Operation F1 score on Multimodal-Mind2Web-v2.
Figure 14: Trajectory Example of GUI-Libra-7B for Task 18 ExpenseDeleteMultiple: Delete the following expenses from pro expense: School Supplies, Religious, Flight Tickets.
Figure 15: Trajectory Example of the base model Qwen2.5-VL-7B-Instruct for Task 18 ExpenseDeleteMultiple: Delete the following expenses from pro expense: School Supplies, Religious, Flight Tickets.
Figure 16: Trajectory Example of GUI-Libra-4B for WebArena-Lite-v2 Task: Follow [’lahwaacz’, ’Koushik’, ’Vinta Chen’] on Gitlab.
Figure 17: Trajectory Example of Qwen3-VL-4B-Instruct for WebArena-Lite-v2 Task: Follow [’lahwaacz’, ’Koushik’, ’Vinta Chen’] on Gitlab.
GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.
Bo Han, Qianyi Wang, Shuai Liu +9
Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.
Graphical User Interface (GUI) agents have emerged as a promising paradigm for intelligent systems that perceive and interact with graphical interfaces visually. Yet supervised fine-tuning alone cannot handle long-horizon credit assignment, distribution shifts, and safe exploration in irreversible environments, making Reinforcement Learning (RL) a central methodology for advancing automation. In this work, we present the first comprehensive overview of the intersection between RL and GUI agents, and examine how this research direction may evolve toward digital inhabitants. We propose a principled taxonomy that organizes existing methods into Offline RL, Online RL, and Hybrid Strategies, and complement it with analyses of reward engineering, data efficiency, and key technical innovations. Our analysis reveals several emerging trends: the tension between reliability and scalability is motivating the adoption of composite, multi-tier reward architectures; GUI I/O latency bottlenecks are accelerating the shift toward world-model-based training, which can yield substantial performance gains; and the spontaneous emergence of System-2-style deliberation suggests that explicit reasoning supervision may not be necessary when sufficiently rich reward signals are available. We distill these findings into a roadmap covering process rewards, continual RL, cognitive architectures, and safe deployment, aiming to guide the next generation of robust GUI automation and its agent-native infrastructure.
Junan Hu, Jian Liu, Jingxiang Lai +6
1Shandong University · 2The Hong Kong University of Science and Technology · 3The Hong Kong University +2
Existing agentic reinforcement learning methods for GUI grounding have limitations at two levels. At the data level, current approaches typically treat all training samples equally, although their training value to the baseline model varies with difficulty. Overlooking this can greatly reduce training efficiency or even cause collapse. At the strategy level, existing frameworks struggle to balance the trade-off between cropping larger regions for sufficient context and smaller ones for reduced redundancy, a tension inherent to tool-augmented grounding agents. In addition, overly complex decision-making is difficult for small-parameter models and significantly increases inference time. To address these issues, at the data level, we propose GUI-D, a data mining and difficulty scoring pipeline that identifies the training-worthy samples by proper testing and assigns difficulty scores to guide subsequent training weights. At the strategy level, we propose GUI-C2, which employs an area-gated coarse-to-fine refinement mechanism that progressively narrows the visual field via model-internal uncertainty signals, adaptively reserving context for large targets while amplifying precision for small ones, reinforced by improvement-aware stage rewards that ensure each refinement genuinely advances grounding. Meanwhile, we simplify the decision-making process to greatly reduce additional inference time. Finally, extensive experiments show that our method achieves state-of-the-art performance. The code and data will be publicly available.