cs.LGOct 31, 2026
SaveMAPLE: A Multimodal Mobile Agent with Playbook-Learned Experience
Organizations: Google · Washington University in St. Louis
Abstract
Autonomous agents powered by vision-language models (VLMs) show great potential in navigating mobile graphical user interfaces (GUIs) to complete complex tasks. However, existing paradigms face fundamental scalability bottlenecks: fine-tuning requires massive high-fidelity datasets or prohibitive training time and compute, while gradient-free approaches rely on custom static heuristics or costly trial-and-error trajectory exploration where feedback is deferred until the end of execution. To address these limitations, we introduce MAPLE (Multimodal Agent with Playbook-Learned Experience), a novel gradient-free multi-agent framework that enables dynamic, step-wise experience sharing and execution-time prompt evolution. Unlike traditional action-reflection architectures that depend on costly, post-hoc self-reflection, MAPLE enables agents to cross-reference and distribute runtime insights in real time at each operational step. This dynamic exchange eliminates blind trial-and-error looping and steers agents along direct, grounded interaction paths. Extensive evaluations on the AndroidWorld and MobileWorld benchmarks demonstrate that MAPLE achieves state-of-the-art performance on complex, long-horizon mobile tasks, while proving scalable and adaptable across novel environments.
Figures & tables
Figure 1: Overview of the MAPLE framework. MAPLE integrates intra-episodic UI execution with a cross-episodic learning loop. Given a user goal, the agent retrieves visually grounded priors from a persistent Playbook and conditions on short-term Working Memory to execute actions. Post-execution reflection updates local state tracking while distilling generalizable rules back into the Playbook for continuous knowledge accumulation.
| (a) AndroidWorld Benchmark (116 Tasks) | |||||
|---|---|---|---|---|---|
| Method / Agent | Backbone Model | Paradigm | Success Rate (%) | ||
| Unrestricted Leaderboard Frontiers ‡ | |||||
| MobileUseAgent [ 20 ] | Seed1.8-GUI | Unrestricted Agent | 97.4 | ||
| AutoDevice [ 2 ] | Gemini-3-Pro + Sonnet | Unrestricted Agent | 94.8 | ||
| Controlled Academic Baselines (Gradient-Free) | |||||
| Stable Planner [ 11 ] | Qwen2.5-VL-72B | Zero-Shot | 63.8 | ||
Table 1: Main Results on Mobile Agent Benchmarks. MAPLE demonstrates superior execution across AndroidWorld ( step budget) and MobileWorld (161 tasks). In AndroidWorld, MAPLE significantly outperforms all strictly zero-shot/gradient-free academic baselines, performing on par with gradient-based RL frameworks. In the highly complex MobileWorld environment, despite utilizing a lightweight Flash backbone, MAPLE achieves Rank #3 Overall and secures the Rank #1 SOTA on User-Interactive tasks across the entire public leaderboard.
| Model | Method | Overall SR (%) | GUI-Only (%) | User-Int. (%) | Trajectory Length |
|---|---|---|---|---|---|
| Gemini-3-Flash | MAPLE (Ours) | 60.25 | 57.26 | 68.18 | 24.14 |
| M3A Baseline | 53.42 | 53.85 | 52.27 | 25.65 | |
| Gemini-3-Flash-Lite | MAPLE (Ours) | 39.13 | 35.04 | 50.00 | 19.13 |
| M3A Baseline | 31.68 | 31.62 | 31.82 | 21.11 | |
| Qwen3.6-35B | MAPLE (Ours) | 27.95 | 28.21 | 27.27 | 26.17 |
| M3A Baseline | 25.47 | 27.35 | 20.45 | 25.23 |
Table 2: Ablation Study on Model Generalization. With identical Discoverability Protocol prompting (Appendix A.2 ), MAPLE improves success rates over the M3A baseline across all four VLM backbones while reducing trajectory steps on three.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Discoverability Protocol | User-Int. SR (%) | Solved Tasks |
|---|---|---|---|
| M3A Baseline (Adapted) | Disabled | 31.82 | 14 / 44 |
| M3A Baseline (Adapted) | Enabled (Table 2) | 52.27 | 23 / 44 |
| MAPLE (Ours) | Enabled (Table 2) | 68.18 | 30 / 44 |
Table 3: Ablation on the Discoverability Protocol on MobileWorld User-Interactive tasks (44 tasks, Gemini-3-Flash ). The last two rows match Table 1(b) and Table 2.
| Budget | Method | Overall SR (%) | GUI-Only SR (%) | User-Int. SR (%) | Avg. Steps |
|---|---|---|---|---|---|
| Minimal | MAPLE (Ours) | 22.36 | 21.37 | 25.00 | 15.24 |
| M3A Baseline | 29.19 | 29.91 | 27.27 | 17.05 | |
| High | MAPLE (Ours) | 39.13 | 35.04 | 50.00 | 19.13 |
| M3A Baseline | 31.68 | 31.62 | 31.82 | 21.11 |
Table 4: Ablation on VLM Thinking Budget using Gemini-3-Flash-Lite on the MobileWorld benchmark. MAPLE strictly requires a High thinking budget to process its dynamic memory, effectively unlocking performance scaling where the baseline plateaus.
| Model | Method | Overall (%) | GUI-Only (%) | User-Int. (%) | Avg. Steps |
|---|---|---|---|---|---|
| Claude-Opus-4.8 | MAPLE (Ours) | 63.35 | 64.96 | 59.09 | 22.20 |
| M3A Baseline | 63.35 | 64.10 | 61.36 | 23.24 | |
| Gemini-3.6-Flash | MAPLE (Ours) | 59.63 | 58.12 | 63.64 | 20.40 |
| M3A Baseline | 62.73 | 62.39 | 63.64 | 21.70 |
Table 5: Frontier Model Saturation. Evaluating MAPLE on state-of-the-art frontier models. Explicit playbook scaffolding causes performance to plateau or regress (the “scaffolding paradox”) by over-constraining the models’ native zero-shot exploratory capabilities, though trajectory lengths continue to strictly decrease.
| Model | Temperature | Top- | Top- | Max Tokens | Thinking Budget |
|---|---|---|---|---|---|
| Gemini 3.6 Flash | Default | Default | - | Default | Default (Medium) |
| Gemini 3 Flash | 0.0 | 0.95 | - | 8192 | Default (High) |
| Gemini 3 Flash-Lite | 0.0 | 0.95 | - | 8192 | High |
| Claude Opus 4.8 | Default | - | - | 8192 | Adaptive |
| Gemma 4-26B-A4B | 1.0 | 0.95 | 64 | 8192 | 6144 tokens (75%) |
| Qwen 3.6-35B | 0.6 (MM) / 0.7 (Text) | 0.95 | - | 8192 | 6144 tokens (75%) |
Table 6: Hyperparameter configurations across evaluated VLMs. Endpoint and special configuration details are elaborated in the text.
Figure 2: Set-of-Mark (SoM) visual grounding example in MobileWorld.
| Model | Method | Success Rate (%) | Avg. Steps |
|---|---|---|---|
| Claude-Opus-4.8 | MAPLE (Ours) | 81.90 | 11.47 |
| M3A Baseline | 77.60 | 11.94 | |
| Gemini-3.6-Flash | MAPLE (Ours) | 80.00 | 11.81 |
| M3A Baseline | 77.00 | 11.96 | |
| Gemini-3-Flash | MAPLE (Ours) | 75.90 | 12.75 |
| M3A Baseline | 71.60 | 13.11 |
Table 7: Cross-Architecture Generalization on AndroidWorld. Evaluating MAPLE across six distinct VLM families on the deterministic AndroidWorld benchmark. MAPLE acts as a universal enhancement, consistently delivering absolute Success Rate improvements while strictly reducing trajectory lengths across both lightweight and state-of-the-art frontier models.
| Budget Setting | Method | Success Rate (%) | Avg. Trajectory Length |
|---|---|---|---|
| 1X Budget (Standard) | MAPLE (Ours) | 75.90 | 12.75 |
| 3X Budget (Relaxed) | MAPLE (Ours) | 81.00 | 15.39 |
Table 8: Ablation on Step Budgets. Evaluating the MAPLE framework (using Gemini-3-Flash) under standard and relaxed step constraints on AndroidWorld. The 3X budget yields a state-of-the-art 81.00% Success Rate. The minimal increase in trajectory length proves the extra budget is utilized for deliberate error correction rather than stochastic guessing.
| Agent | Paradigm | Model | Method | Full SR | Subset SR |
| Baseline | Gradient-Free | Gemini-3.1-Flash | Minimal Guidance | 73.30% | 82.50% |
| MAPLE (Ours) | Gradient-Free | Gemini-3.1-Flash + Lite | Playbook + Context | 76.70% | 84.47% |
| MobileUse | Gradient-Free | Qwen2.5-VL-72B-Inst | Hierarchical Refl. | 62.90% | N/A |
| Agent-SAMA | Gradient-Free | GPT-4o + Qwen-VL+ | Dynamic FSM | 63.70% | N/A |
| Stable Planner | Gradient-Free | Qwen2.5-VL-72B | EFSM Traversal | 63.80% | N/A |
| Fine-Tuned | Qwen2.5-VL (72B+7B) | RL + Self-Evolve | 76.70% | 86.40% |
Table 9: Performance on the Full Benchmark vs. the Solvable Subset. Comparing Success Rates (SR) on the standard 116-task suite and the purified 103-task Solvable Subset (omitting missions with systemic validation/emulator flaws). Note: This audit evaluates a hybrid Gemini-3.1-Flash setup (pairing Flash-Lite for auxiliary memory modules in MAPLE), so Full SR differs slightly from the uniform single-backbone Gemini-3-Flash setup in Table 1 (a). Subset SR is reported only for baselines with publicly available trajectories.
Figure 3: A complete, 7-step successful trajectory in AndroidWorld for recording and saving an audio clip. The captions denote the exact JSON action generated by the agent at each respective step, demonstrating precise visual grounding and action formatting.
More in this topic
Mobile GUI Agents
Mobile GUI agents use visual and structured interface information to carry out tasks across smartphone apps.
See all 87 papers