While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.
Figures & tables
Figure 1: Overview of the iOS agent architecture and training setup.
UINavBench
iOSWorld
AndroidWorld
SFT Baseline
16.7%
4.5%
4.7%
PPO (terminal only), Step 75
17.6%
3.0%
9.6%
PPO (terminal only), Step 168
17.6%
5.3%
6.9%
PPO (terminal only), Step 336
21.6%
7.5%
19.8%
PPO, Step 70
14.7%
6.0%
11.2%
PPO, Step 168
25.5%
6.0%
20.7%
Table 1: Success rates across UINavBench, iOSWorld, and AndroidWorld. We compare our SFT baseline against our RL-trained agents (PPO and GiGPPO), including an ablation of PPO trained with only terminal rewards. For each RL checkpoint, we report the number of update steps, where Steps 168 and 336 correspond to one and two epochs, respectively. We also include results for the UI-TARS agent and three frontier models (Qwen3.8-27B, Gemini-3.5-Flash, and Gemini-3.1-Pro) evaluated using our agent setup. UI-TARS was not evaluated on iOSWorld.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
3b
Gemini-3.5-Flash
3b w/ Gemini-3.5-Flash (swipe only)
UINavBench
12.7%
14.7%
16.7%
iOSWorld
3.0%
4.5%
1.5%
Appendix
Table 2: Performance of the SFT trained Planning agent paired with different grounding models.
Figure 2: Example traces of the GiGPPO-trained planning agent and the SFT baseline on the UINavBench task: ”Find a route from Apple Park to the Santa Clara Caltrain Station.” Pink crosshairs indicate the agent’s tap and text-entry actions at each step. Because the Apple Maps workflow requires selecting the destination before the starting location, the agents are easily confused by the task prompt’s phrasing, which mentions the starting point first. The GiGPPO agent initially sets the destination as Apple Park, but it corrects itself by exiting the directions search and restarting with the Santa Clara Caltrain Station as the destination, demonstrating strong error recovery. In contrast, the SFT agent opens the wrong directions search page and gets stuck repeatedly trying to set the starting point of a custom route until it reaches the maximum step limit.
Figure 3: Example traces of the GiGPPO-trained planning agent and the SFT baseline on the complex, multi-app iOSWorld task: “Find my most recent Chipotle order in QuickBite and note the items and total. Search my Mail inbox for the QuickBite receipt email and note the receipt amount. Check my MyBank credit card transactions for the corresponding charge. Compare the amounts across all three sources (totals may differ due to tips, fees, or processing). What are the order items, QuickBite total, email receipt amount, bank charge, and any discrepancies?” Pink crosshairs indicate the agent’s tap and text-entry actions at each step, while blue arrows indicate swipe actions. Although the SFT agent opens all three apps, it fails to search them for the necessary information. In contrast, the GiGPPO agent successfully navigates through each app and locates all the required details.
Figure 4: Example traces of the PPO-trained planning agent and the PPO agent trained only on terminal rewards for the AndroidWorld task: ”Delete the following recipes from Broccoli app: Chicken Alfredo Pasta, Tomato Basil Bruschetta, Grilled Cheese with Tomato and Basil.” Pink crosshairs indicate the agent’s tap and text-entry actions at each step, while blue arrows indicate swipe actions. Because the app contains many other recipes, the PPO agent with dense rewards efficiently searches for and deletes each specified recipe. In contrast, the PPO agent without dense rewards manually scrolls through the list to locate each one, wasting numerous steps and failing to complete the task within the maximum step limit.
We present LiteGUI, a new framework for building lightweight GUI agents. GUI interaction poses unique challenges due to the long-horizon nature of complex tasks and the existence of multiple valid interaction paths, which are difficult for lightweight GUI agents to handle effectively. To address these challenges, LiteGUI introduces a two-stage post-training framework operating on logged GUI states. First, we propose Guided On-Policy Distillation, which provides training-time privileged teacher guidance at each GUI state by selecting the most-matched valid action from human-verified multi-solution action annotations, while leaving the student's on-policy rollout unchanged. Second, we develop Multi-Solution Dual-Level GRPO, which combines action-level supervision with history-conditioned, per-state planning-quality supervision while accounting for multiple valid actions at each logged GUI state. Together, these stages support multi-step GUI tasks without requiring the student to reproduce fixed demonstration trajectories. We further develop a scalable data generation pipeline and a corresponding multi-solution dataset, Lite-Dataset, to support the training and evaluation of GUI agents, addressing a gap in the existing literature. Extensive experiments across multiple GUI benchmarks demonstrate that LiteGUI substantially improves the performance of lightweight GUI agents over existing training paradigms, including SFT, on-policy distillation and GRPO, and achieves competitive or superior performance compared with substantially larger models.
We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.
Graphical user interface (GUI)-based mobile agents automate digital tasks on mobile devices by interpreting natural-language instructions and interacting with the screen. While recent methods apply reinforcement learning (RL) to train vision-language-model(VLM) agents in interactive environments with a primary focus on performance, generalization remains underexplored due to the lack of standardized benchmarks and open-source RL systems. In this work, we formalize the problem as a Contextual Markov Decision Process (CMDP) and introduce AndroidWorld-Generalization, a benchmark with three increasingly challenging regimes for evaluating zero-shot generalization to unseen task instances, templates, and applications. We further propose an RL training system that integrates Group Relative Policy Optimization (GRPO) with a scalable rollout collection system, consisting of containerized infrastructure and asynchronous execution % , and error recovery to support reliable and efficient training. Experiments on AndroidWorld-Generalization show that RL enables a 7B-parameter VLM agent to surpass supervised fine-tuning baselines, yielding a 26.1% improvement on unseen instances but only limited gains on unseen templates (15.7%) and apps (8.3%), underscoring the challenges of generalization. As a preliminary step, we demonstrate that few-shot adaptation at test-time improves performance on unseen apps, motivating future research in this direction. To support reproducibility and fair comparison, we open-source the full RL training system, including the environment, task suite, models, prompt configurations, and the underlying infrastructure \footnote{https://github.com/zihuanjiang/AndroidWorld-Generalization}.
Li Gu, Zihuan Jiang, Zhixiang Chi +5
Mila – Québec AI Institute · Concordia University · University of Toronto +3