Organizations: Artificial Intelligence Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Baidu Inc. Beijing, China · Department of Electrical and Computer Engineering University of Alberta, Edmonton, Alberta, Canada
Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) have shown strong potential for automating tasks across diverse digital environments, where reinforcement learning (RL) has become a dominant training paradigm. However, widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward-gradient misalignment, leading to inefficient and unstable optimization. Recent work addresses this issue by reformulating RL with verifiable rewards (RLVR) as contrastive or classification-based objectives, which improve stability by eliminating problematic gradient behaviors. Despite this progress, existing contrastive RLVR methods rely primarily on outcome-level supervision and fail to capture fine-grained differences in trajectory quality within the same outcome category. In this paper, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive RLVR framework that incorporates trajectory-level quality signals into policy optimization. LACL-GUI introduces structured preferences within both successful and failed trajectories, encouraging concise successful executions and differentiating failure quality based on divergence from successful trajectories, while preserving optimization stability. Experiments on GUI agent benchmarks show that LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods, highlighting the value of trajectory-level supervision in contrastive RLVR.
Figures & tables
Figure 1: Overview of LACL-GUI framework. A decoupled architecture collects informative trajectory groups and maintains a Success Buffer for reference trajectories. The proposed objective introduces length-aware positive preference and divergence-aware negative refinement on top of REAL contrastive objective.
Model
Max Steps
gimp
calc
impress
writer
os
thunderbird
vlc
vs_code
Overall
Closed-source Models
OpenAI CUA o3 ( 24 )
100
38.5
10.6
10.6
30.4
62.5
26.7
39.2
39.1
27.8
OpenAI CUA ( 24 )
50
34.6
14.9
29.7
26.1
70.8
66.7
11.6
69.6
36.5
Claude 3.7 Sonnet ( 1 )
50
38.5
31.9
36.1
43.5
50.0
53.3
23.5
56.5
40.1
Seed1.5-VL-250717 ( 12 )
100
50.0
34.8
48.9
56.5
39.1
73.3
35.3
56.5
45.1
Claude 4 Sonnet ( 2 )
50
50.0
31.9
46.7
60.9
45.8
73.3
41.2
60.9
47.3
Table 1: Task success rate (%) on the OSWorld benchmark across different application domains. Results are reported following the official OSWorld evaluation protocol. Qwen3-VL-GRPO and Qwen3-VL-REAL denote models trained with GRPO and REAL objectives, respectively, based on the corresponding Qwen3-VL backbones.
Configuration
Success Rate (%)
Gain (%)
Prompt Baseline
42.7
–
+ Decoupled Architecture
45.4
+2.7
+ Success Buffer
46.5
+1.1
+ Trajectory Filter
47.2
+0.7
+ LACL-GUI Objective
50.0
+2.8
Table 2: Ablation study of different components in the LACL-GUI framework. Each row incrementally introduces one additional component over the previous configuration. The prompt-based baseline uses Qwen3-VL-8B-Thinking, while GRPO is adopted for all training configurations except the proposed LACL-GUI objective.
Figure 2: Effect of length-aware preference modeling in LACL-GUI. We compare average trajectory lengths on tasks where all three methods achieve the same outcome.
Figure 3: Sensitivity analysis of the similarity threshold Θ for divergence estimation in LACL-GUI. The default setting ( Θ=0.85 ) is highlighted, and results are reported across different domains.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Number of Tasks
Gimp
26
LibreOffice Calc
47
LibreOffice Impress
47
LibreOffice Writer
23
OS
24
Thunderbird
15
Appendix
Table 3: Distribution of the 222 selected OSWorld tasks across application domains.
Domain
Base
LACL-GUI
REAL
Rel. to REAL
Gimp
36.46
12.08
13.38
−9.7%
Calc
17.50
10.17
16.00
−36.4%
Impress
4.25
4.25
4.25
0.0%
Writer
10.73
5.36
5.27
+1.7%
OS
21.67
4.25
5.50
−22.7%
Thunderbird
7.86
6.71
7.29
−8.0%
Appendix
Table 4: Average trajectory lengths on the 70 tasks successfully completed by all three methods. The last column shows the relative change of LACL-GUI with respect to REAL.
Domain
Base
LACL-GUI
REAL
Rel. to REAL
Gimp
33.80
5.20
25.00
−79.2%
Calc
39.91
30.12
33.97
−11.3%
Impress
30.48
27.86
31.24
−10.8%
Writer
37.83
46.67
42.67
+9.4%
OS
35.67
10.67
22.67
−52.9%
Thunderbird
50.00
26.75
27.75
−3.6%
Appendix
Table 5: Average trajectory lengths on the 93 tasks where all three methods fail. The last column shows the relative change of LACL-GUI with respect to REAL.
Figure 4: LACL-GUI Step 1 (Success)
Figure 5: LACL-GUI Step 2 (Success)
Figure 6: LACL-GUI Step 3 (Success)
Figure 7: LACL-GUI Step 4 (Success)
Figure 8: LACL-GUI Step 5 (Success)
Figure 9: LACL-GUI Step 6 (Success)
Figure 10: A redundant interaction in the long successful trajectory (Step 5).
Figure 11: An erroneous repeated action after the UI state changes (Step 17).
Figure 12: Premature termination after an incomplete state transition (Step 4 in 4-step Failed Trajectory).
Figure 13: Additional ineffective actions after divergence from the successful execution (Step 5 in 6-step Failed Trajectory).
Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data synthesis methods for GUI Agents rely on specific environments and struggle to generate diverse data, while existing evaluators either suffer from limited scalability or provide inaccurate and unreliable reward signals. To overcome these challenges, we introduce GSAR (Goal-State-Anchor Reward), a RL reward framework that supports scalable task generation and delivers reliable reward signals for stable and efficient policy optimization. Our approach features self-evolving data synthesis, which produces multiple environments through task execution and generates diverse tasks and goal states. Complementing this, a state-anchor mechanism automatically annotates task-relevant UI elements in successful goal states as reference anchors. During RL training, these reference anchors provide accurate, scalable reward signals that substantially enhance efficiency. Extensive evaluations demonstrate that our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods. Furthermore, agents trained using our reward framework exhibit strong performance on both AndroidWorld and our constructed benchmark, establishing a scalable approach for GUI agent training.
As Multimodal Large Language Models (MLLMs) mature, GUI agents are evolving from static interactions to complex navigation. While Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks, its effective application faces a dilemma. Standard Offline RL often relies on static step-level data, neglecting global trajectory semantics such as task completion and execution quality. Conversely, Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability. To bridge this gap, we propose SOLAR-RL (Semi-Online Long-horizon Assignment Reinforcement Learning). Instead of relying solely on expensive online interactions, our framework integrates global trajectory insights directly into the offline learning process. Specifically, we reconstruct diverse rollout candidates from static data, detect the first failure point using per-step validity signals, and retroactively assign dense step-level rewards with target-aligned shaping to reflect trajectory-level execution quality, effectively simulating online feedback without interaction costs. Extensive experiments demonstrate that SOLAR-RL significantly improves long-horizon task completion rates and robustness compared to strong baselines, offering a sample-efficient solution for autonomous GUI navigation.
Jichao Wang, Liuyang Bian, Yufeng Zhou +9
1vivo AI Lab · 2Zhejiang Lab · 5Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences +2
Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT), which enables the continuous co-evolution of data and models in a fully autonomous loop. The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts and constructs dense step-level supervision from successful trajectories to correct failed ones. Extensive experiments on AndroidWorld show that our 4B model achieves an 81.0% Pass@1 success rate, outperforming numerous recent baselines and exceeding human-level performance. Ablation and case studies further verify the effectiveness of GRSD. Our method represents a significant leap toward efficient, self-evolving, and high-performance mobile GUI automation without expensive manual data annotation.