Autonomous agents powered by vision-language models (VLMs) show great potential in navigating mobile graphical user interfaces (GUIs) to complete complex tasks. However, existing paradigms face fundamental scalability bottlenecks: fine-tuning requires massive high-fidelity datasets or prohibitive training time and compute, while gradient-free approaches rely on custom static heuristics or costly trial-and-error trajectory exploration where feedback is deferred until the end of execution. To address these limitations, we introduce MAPLE (Multimodal Agent with Playbook-Learned Experience), a novel gradient-free multi-agent framework that enables dynamic, step-wise experience sharing and execution-time prompt evolution. Unlike traditional action-reflection architectures that depend on costly, post-hoc self-reflection, MAPLE enables agents to cross-reference and distribute runtime insights in real time at each operational step. This dynamic exchange eliminates blind trial-and-error looping and steers agents along direct, grounded interaction paths. Extensive evaluations on the AndroidWorld and MobileWorld benchmarks demonstrate that MAPLE achieves state-of-the-art performance on complex, long-horizon mobile tasks, while proving scalable and adaptable across novel environments.
Figures & tables
Figure 1: Overview of the MAPLE framework. MAPLE integrates intra-episodic UI execution with a cross-episodic learning loop. Given a user goal, the agent retrieves visually grounded priors from a persistent Playbook and conditions on short-term Working Memory to execute actions. Post-execution reflection updates local state tracking while distilling generalizable rules back into the Playbook for continuous knowledge accumulation.
(a) AndroidWorld Benchmark (116 Tasks)
Method / Agent
Backbone Model
Paradigm
Success Rate (%)
Unrestricted Leaderboard Frontiers ‡
MobileUseAgent [ 20 ]
Seed1.8-GUI
Unrestricted Agent
97.4
AutoDevice [ 2 ]
Gemini-3-Pro + Sonnet
Unrestricted Agent
94.8
Controlled Academic Baselines (Gradient-Free)
Stable Planner [ 11 ]
Qwen2.5-VL-72B
Zero-Shot
63.8
Table 1: Main Results on Mobile Agent Benchmarks. MAPLE demonstrates superior execution across AndroidWorld ( 1× step budget) and MobileWorld (161 tasks). In AndroidWorld, MAPLE significantly outperforms all strictly zero-shot/gradient-free academic baselines, performing on par with gradient-based RL frameworks. In the highly complex MobileWorld environment, despite utilizing a lightweight Flash backbone, MAPLE achieves Rank #3 Overall and secures the Rank #1 SOTA on User-Interactive tasks across the entire public leaderboard.
Model
Method
Overall SR (%)
GUI-Only (%)
User-Int. (%)
Trajectory Length
Gemini-3-Flash
MAPLE (Ours)
60.25
57.26
68.18
24.14
M3A Baseline
53.42
53.85
52.27
25.65
Gemini-3-Flash-Lite
MAPLE (Ours)
39.13
35.04
50.00
19.13
M3A Baseline
31.68
31.62
31.82
21.11
Qwen3.6-35B
MAPLE (Ours)
27.95
28.21
27.27
26.17
M3A Baseline
25.47
27.35
20.45
25.23
Table 2: Ablation Study on Model Generalization. With identical Discoverability Protocol prompting (Appendix A.2 ), MAPLE improves success rates over the M3A baseline across all four VLM backbones while reducing trajectory steps on three.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Discoverability Protocol
User-Int. SR (%)
Solved Tasks
M3A Baseline (Adapted)
Disabled
31.82
14 / 44
M3A Baseline (Adapted)
Enabled (Table 2)
52.27
23 / 44
MAPLE (Ours)
Enabled (Table 2)
68.18
30 / 44
Appendix
Table 3: Ablation on the Discoverability Protocol on MobileWorld User-Interactive tasks (44 tasks, Gemini-3-Flash ). The last two rows match Table 1(b) and Table 2.
Budget
Method
Overall SR (%)
GUI-Only SR (%)
User-Int. SR (%)
Avg. Steps
Minimal
MAPLE (Ours)
22.36
21.37
25.00
15.24
M3A Baseline
29.19
29.91
27.27
17.05
High
MAPLE (Ours)
39.13
35.04
50.00
19.13
M3A Baseline
31.68
31.62
31.82
21.11
Appendix
Table 4: Ablation on VLM Thinking Budget using Gemini-3-Flash-Lite on the MobileWorld benchmark. MAPLE strictly requires a High thinking budget to process its dynamic memory, effectively unlocking performance scaling where the baseline plateaus.
Model
Method
Overall (%)
GUI-Only (%)
User-Int. (%)
Avg. Steps
Claude-Opus-4.8
MAPLE (Ours)
63.35
64.96
59.09
22.20
M3A Baseline
63.35
64.10
61.36
23.24
Gemini-3.6-Flash
MAPLE (Ours)
59.63
58.12
63.64
20.40
M3A Baseline
62.73
62.39
63.64
21.70
Appendix
Table 5: Frontier Model Saturation. Evaluating MAPLE on state-of-the-art frontier models. Explicit playbook scaffolding causes performance to plateau or regress (the “scaffolding paradox”) by over-constraining the models’ native zero-shot exploratory capabilities, though trajectory lengths continue to strictly decrease.
Model
Temperature
Top- p
Top- k
Max Tokens
Thinking Budget
Gemini 3.6 Flash
Default
Default
-
Default
Default (Medium)
Gemini 3 Flash
0.0
0.95
-
8192
Default (High)
Gemini 3 Flash-Lite
0.0
0.95
-
8192
High
Claude Opus 4.8
Default
-
-
8192
Adaptive
Gemma 4-26B-A4B
1.0
0.95
64
8192
6144 tokens (75%)
Qwen 3.6-35B
0.6 (MM) / 0.7 (Text)
0.95
-
8192
6144 tokens (75%)
Appendix
Table 6: Hyperparameter configurations across evaluated VLMs. Endpoint and special configuration details are elaborated in the text.
Figure 2: Set-of-Mark (SoM) visual grounding example in MobileWorld.
Model
Method
Success Rate (%)
Avg. Steps
Claude-Opus-4.8
MAPLE (Ours)
81.90
11.47
M3A Baseline
77.60
11.94
Gemini-3.6-Flash
MAPLE (Ours)
80.00
11.81
M3A Baseline
77.00
11.96
Gemini-3-Flash
MAPLE (Ours)
75.90
12.75
M3A Baseline
71.60
13.11
Appendix
Table 7: Cross-Architecture Generalization on AndroidWorld. Evaluating MAPLE across six distinct VLM families on the deterministic AndroidWorld benchmark. MAPLE acts as a universal enhancement, consistently delivering absolute Success Rate improvements while strictly reducing trajectory lengths across both lightweight and state-of-the-art frontier models.
Budget Setting
Method
Success Rate (%)
Avg. Trajectory Length
1X Budget (Standard)
MAPLE (Ours)
75.90
12.75
3X Budget (Relaxed)
MAPLE (Ours)
81.00
15.39
Appendix
Table 8: Ablation on Step Budgets. Evaluating the MAPLE framework (using Gemini-3-Flash) under standard and relaxed step constraints on AndroidWorld. The 3X budget yields a state-of-the-art 81.00% Success Rate. The minimal increase in trajectory length proves the extra budget is utilized for deliberate error correction rather than stochastic guessing.
Agent
Paradigm
Model
Method
Full SR
Subset SR
Baseline
Gradient-Free
Gemini-3.1-Flash
Minimal Guidance
73.30%
82.50%
MAPLE (Ours)
Gradient-Free
Gemini-3.1-Flash + Lite
Playbook + Context
76.70%
84.47%
MobileUse
Gradient-Free
Qwen2.5-VL-72B-Inst
Hierarchical Refl.
62.90%
N/A
Agent-SAMA
Gradient-Free
GPT-4o + Qwen-VL+
Dynamic FSM
63.70%
N/A
Stable Planner
Gradient-Free
Qwen2.5-VL-72B
EFSM Traversal
63.80%
N/A
K2
Fine-Tuned
Qwen2.5-VL (72B+7B)
RL + Self-Evolve
76.70%
86.40%
Appendix
Table 9: Performance on the Full Benchmark vs. the Solvable Subset. Comparing Success Rates (SR) on the standard 116-task suite and the purified 103-task Solvable Subset (omitting missions with systemic validation/emulator flaws). Note: This audit evaluates a hybrid Gemini-3.1-Flash setup (pairing Flash-Lite for auxiliary memory modules in MAPLE), so Full SR differs slightly from the uniform single-backbone Gemini-3-Flash setup in Table 1 (a). Subset SR is reported only for baselines with publicly available trajectories.
Figure 3: A complete, 7-step successful trajectory in AndroidWorld for recording and saving an audio clip. The captions denote the exact JSON action generated by the agent at each respective step, demonstrating precise visual grounding and action formatting.
As large multimodal models move from understanding content to operating on digital environments, mobile GUI has emerged as a challenging and consequential testbed for digital embodied intelligence. Mobile agents operate under three coupled constraints: precise perception of complex interfaces, scalable acquisition of high-quality interaction data, and robust long-horizon decision making under compounding execution errors. This report presents HyMobileAgent, a mobile GUI agent built on Hy3.0-VL-A3B, a vision-native foundation model featuring native any-resolution input, an A3B-scale deployment budget, and a 32K context window to model extended interaction histories. Rather than relying solely on model scaling, we develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction. Our framework integrates a GUI perception flywheel combining mock-interface synthesis, rejection sampling, and icon-specific augmentation; a knowledge pipeline that transforms tutorial videos into structured interaction data; a million-scale action data pipeline deployed across more than 2000 sandbox and real-device instances with automated failure attribution; the PhoneWorld Mock App Factory, providing a resettable training environment with 34 mock applications and over 34000 tasks; and a structured Planning-and-Reflection mechanism with explicit dead-loop detection for reliable long-horizon execution. We also introduce a progressive training recipe consisting of mid-training, supervised fine-tuning, and reinforcement learning with task-specific reward designs.
Vision-language models (VLMs) have become a common foundation for autonomous mobile GUI agents, but most existing systems rely on the VLM for both planning and action grounding at nearly every interaction step, leading to substantial latency and model-serving cost. We introduce Jev-Mobile, which shifts this paradigm to low-frequency VLM planning and high-frequency lightweight execution: the VLM specifies local goals, the accessibility tree defines a structured executable action space, and Jev, a fast typed decision model, repeatedly selects actions within this space. This design allows multiple GUI actions to be executed under a single VLM decision, reducing expensive VLM inference while preserving adaptive interaction. On the full AndroidWorld task suite, Jev-Mobile achieves 79% task success, compared with 78% for SeeAct-V and 84% for a Step-wise VLM baseline. Among successful trajectories, it reduces mean end-to-end execution time by 32.7% and mean model API cost by 73.4% relative to Step-wise VLM. These results show that decoupling high-level VLM reasoning from low-level action execution can substantially improve mobile GUI agent efficiency while maintaining competitive task performance.
Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap, e.g., nearly 70% success on AndroidWorld. However, these systems keep their training data closed and remain opaque about their task and trajectory synthesis recipes. We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories, with two key components: (1) The first is a scalable task synthesis pipeline that constructs a global environment memory from exploration, then leverages it to generate diverse and grounded instructions. and (2) a policy-switching strategy for trajectory rollout. By alternating between learner and expert models, it captures essential error-recovery data often missing in standard imitation learning. Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks: notably, our fine-tuned Qwen2.5-VL and Qwen3-VL reach 51.7% and 64.7% on AndroidWorld, far surpassing existing open-data approaches. Furthermore, we conduct transparent analyses on the overlap between our synthetic instructions and benchmark test sets, and verify that performance gains stem from broad functionality coverage rather than benchmark overfitting. We release data and code at https://njucckevin.github.io/openmobile/ to bridge the data gap and facilitate broader mobile agent research.
Kanzhi Cheng, Zehao Li, Zheng Ma +11
1Nanjing University · 4Shanghai AI Laboratory · 2SenseTime +3