Generalization in Online Reinforcement Learning for Mobile Agents
Authors: Li Gu, Zihuan Jiang, Zhixiang Chi, Huan Liu, Ziqiang Wang, Yuanhao Yu, Glen Berseth, Yang Wang
Organizations: Mila – Québec AI Institute · Concordia University · University of Toronto · McMaster University · Université de Montréal · CIFAR AI Chair
Graphical user interface (GUI)-based mobile agents automate digital tasks on mobile devices by interpreting natural-language instructions and interacting with the screen. While recent methods apply reinforcement learning (RL) to train vision-language-model(VLM) agents in interactive environments with a primary focus on performance, generalization remains underexplored due to the lack of standardized benchmarks and open-source RL systems. In this work, we formalize the problem as a Contextual Markov Decision Process (CMDP) and introduce AndroidWorld-Generalization, a benchmark with three increasingly challenging regimes for evaluating zero-shot generalization to unseen task instances, templates, and applications. We further propose an RL training system that integrates Group Relative Policy Optimization (GRPO) with a scalable rollout collection system, consisting of containerized infrastructure and asynchronous execution % , and error recovery to support reliable and efficient training. Experiments on AndroidWorld-Generalization show that RL enables a 7B-parameter VLM agent to surpass supervised fine-tuning baselines, yielding a 26.1% improvement on unseen instances but only limited gains on unseen templates (15.7%) and apps (8.3%), underscoring the challenges of generalization. As a preliminary step, we demonstrate that few-shot adaptation at test-time improves performance on unseen apps, motivating future research in this direction. To support reproducibility and fair comparison, we open-source the full RL training system, including the environment, task suite, models, prompt configurations, and the underlying infrastructure \footnote{https://github.com/zihuanjiang/AndroidWorld-Generalization}.
Figures & tables
Figure 1: Sample task instructions with corresponding screenshots from the train and test sets of the three unseen regimes in the AndroidWorld-Generalization benchmark. Red highlights the unseen scenarios: Instance, Template, and Application.
Regime
Overlap
Train
Test
Instance
Template
App
I/T/A
Diff.
I/T/A
Diff.
Unseen Instance
✗
✓
✓
1149 / 78 / 17
1.68
234 / 78 / 17
1.68
Unseen Template
✗
✗
✓
836 / 57 / 14
1.70
54 / 18 / 14
1.72
Unseen App
✗
✗
✗
905 / 62 / 12
1.68
48 / 16 / 5
1.69
Table 1: Statistics of the AndroidWorld-Generalization benchmark . “I”, “T”, and “A” denote Instances, Templates, and Applications; “Diff.” denotes average difficulty. Bold numbers indicate the number of train/test examples in each regime, while gray numbers indicate the templates and applications from which those examples are generated.
Figure 2: RL training system for mobile agent. We integrate GRPO with a scalable rollout collection system that parallelizes multiple environments. Docker containerization provides resource isolation and decouples trainer and environments through HTTP communication for reliability. Asynchronous rollouts eliminate synchronization bottlenecks, enabling more agent steps per unit time. Together, these three techniques facilitate reliable and efficient large-scale training.
Models
API-Based
Open-Source
Reproducible
Average (SR)
Proprietary model
GPT-4o ( Hurst et al., 2024 )
✓
✓
34.5
Claude Computer Use ( Anthropic, 2024 )
✓
✓
27.9
UGround+GPT-4o ( Gou et al., 2024 )
✓
✓
✓
44.0
Aria-UI+GPT-4o ( Yang et al., 2024 )
✓
✓
✓
44.8
Agent S2 ( Agashe et al., 2025 )
✓
54.3
Table 2: Comparison of methods on AndroidWorld. “API-Based” denotes use of proprietary inference APIs. “Open-Source” denotes availability of model weights. “Reproducible” denotes availability of full codebase.
Figure 3: Training dynamics on Unseen Instance with curriculum learning. Colored areas denote curriculum stages: blue (Easy), red (Easy + Medium), green (All). (Left) Average training and evaluation success rates. (Right) Average evaluation success rates by task type (Task Completion, Information Retrieval) and difficulty (Easy, Medium, Hard).
Figure 4: Training dynamics of GRPO across the three unseen regimes. We report training success rates and evaluation success rates with standard deviations.
Figure 5: Training dynamics of PPO across the three unseen regimes.
Figure 6: Few-shot adaptation vs. non-adaptation, averaged. over Unseen App test-set.
Figure 7: (Left) Time profiling of a policy update vs. the collection of 16 rollouts per training step. (Right) Performance of async vs. sync rollouts with varying environment counts.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
AndroidWorld
AndroidWorld-Generalization
Environment
116 templates across 20 apps
Same 20 apps and templates (subset splits)
Task Suite
116 eval instances (1 seed / template)
Multi-seed eval instances; thousands of non-overlapping train instances
Generalization
None
Three unseen regimes (instance, template, app)
RL Support
No; evaluation-only
Yes; standardized train set for RL
Infrastructure
Sequential rollouts
Parallel rollout collection for scalable evaluation & RL training
Appendix
Table 3: Comparison between AndroidWorld and AndroidWorld-Generalization.
Models
Easy (SR)
Medium (SR)
Hard (SR)
Average (SR)
Close-source Models
GPT-4o ( Hurst et al., 2024 )
-
-
-
34.5
Claude Computer Use ( Anthropic, 2024 )
-
-
-
27.9
UGround+GPT-4o ( Gou et al., 2024 )
-
-
-
44.0
Aria-UI+GPT-4o ( Yang et al., 2024 )
-
-
-
44.8
Agent S2 ( Agashe et al., 2025 )
-
-
-
54.3
Appendix
Table 4: AndroidWorld evaluation performance for Mobile Agents. 3 3 3 Prior methds marked with ∗ are evaluated on a sub-set of AndroidWorld.
Figure 8: Case study 1.
Figure 9: Case study 2.
Figure 10: Case study 3.
Figure 11: Case study 4.
Figure 12: Case study 5.
Figure 17
Action Type
Description
Click (x,y)
Tap at screen coordinate (x,y)
Long press (x,y)
Long press at screen coordinate (x,y)
Type (text)
Enter variable-length natural-language text
Scroll (d)
Scroll in direction d∈{up, down, left, right}
Navigate home ()
Navigate Back to home screen
Navigate back ()
Navigate Back to previous screen
Appendix
Table 5: Action space of mobile use agents in AndroidWorld.
GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods. Code is available at https://github.com/genalyu/WM-R1 .
Yu Han, Tianwen Qian
School of Computer Science and Technology, East China Normal University
We introduce OpenMobile-2, a near-frontier mobile agent with fully open training environments and recipes. We make three key advances: (1) Diverse environments with simulated commercial apps: We build MobileGym++, featuring 35 realistic, functionally rich commercial-style apps with cross-app workflows, while preserving full controllability for reset and verification. Together with newly configured apps in Android emulators, this yields a diverse playground spanning over 110 apps. (2) Open training data at scale: Building on this foundation, we curate nearly 12K mobile interaction trajectories for supervised fine-tuning and the largest open collection of verifiable RL training data for mobile agents, comprising over 2K executable tasks with automatic rewards. (3) Hybrid GUI and app-native tool use: We explore an experimental mobile-use setting where apps expose selected functionalities as app-native tools alongside their GUIs, allowing agents to interleave GUI actions and tool calls within a task. We implement this setting in MobileGym++ with over 300 carefully scoped tools across 50+ apps. Additionally, we introduce MobileGym++ Bench for complex, long-horizon commercial mobile scenarios, supporting both GUI-only and hybrid GUI-tool evaluation on a shared task suite. OpenMobile-2 performs competitively across established benchmarks, including AndroidWorld (79.9) and MobileWorld (50.4), while showing promising transfer to real-device mobile use, nearly doubling SPA-Bench performance from 31.9 to 59.6. All resources are available at https://os-copilot.github.io/OpenMobile2-Home/.
Chenyang Yan, Yingying Zhang, Kanzhi Cheng +21
Nanjing University · SenseTime · The University of Hong Kong +3
Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap, e.g., nearly 70% success on AndroidWorld. However, these systems keep their training data closed and remain opaque about their task and trajectory synthesis recipes. We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories, with two key components: (1) The first is a scalable task synthesis pipeline that constructs a global environment memory from exploration, then leverages it to generate diverse and grounded instructions. and (2) a policy-switching strategy for trajectory rollout. By alternating between learner and expert models, it captures essential error-recovery data often missing in standard imitation learning. Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks: notably, our fine-tuned Qwen2.5-VL and Qwen3-VL reach 51.7% and 64.7% on AndroidWorld, far surpassing existing open-data approaches. Furthermore, we conduct transparent analyses on the overlap between our synthetic instructions and benchmark test sets, and verify that performance gains stem from broad functionality coverage rather than benchmark overfitting. We release data and code at https://njucckevin.github.io/openmobile/ to bridge the data gap and facilitate broader mobile agent research.
Kanzhi Cheng, Zehao Li, Zheng Ma +11
1Nanjing University · 4Shanghai AI Laboratory · 2SenseTime +3