Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
Authors: Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu
Organizations: Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Alibaba Token Hub, Alibaba Group · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.
Figures & tables
Figure 1: Overview of GMA . GMA provides a diverse and reproducible mobile ecosystem with seven custom applications, together with 300 tasks spanning four progressively more demanding tiers: Atomic, Compositional, Cross-Application, and Realistic. We further study the role of the agent harness, including context retention, explicit state tracking, and model-specific harness design. Evaluation on GMA reveals a substantial performance degradation as task complexity increases and highlights the growing importance of effective harness design for complex mobile workflows.
Figure 2: Screenshots of the seven custom applications developed for GMA . From left to right, the applications cover instant messaging, music library, lifestyle sharing, local discovery, food delivery, e-commerce, and travel planning, providing diverse interfaces, interaction patterns, and application-specific workflows for mobile agent evaluation.
Models
Overall
Atomic
Compositional
Cross-Application
Realistic
Success
Average
Success
Average
Success
Average
Success
Average
Success
Average
Doubao-Seed-2.1-pro
62.00
74.42
89.04
89.73
65.89
79.27
41.03
62.99
20.00
31.75
Qwen3.7-Plus
49.33
63.78
78.08
80.37
51.94
69.78
30.77
51.87
0 0.00
11.00
Qwen3.8-Max
67.33
80.35
93.15
93.84
70.54
85.17
52.56
70.57
10.00
38.22
Gemini 3.5 Flash
54.33
65.91
80.82
81.74
62.02
75.23
30.77
50.64
0 0.00
0 7.56
GPT-5.5
59.67
71.14
87.67
87.67
65.12
78.71
39.74
58.78
0 0.00
10.25
Table 1: Overall and difficulty-specific performance of evaluated models on GMA . For each setting, we report the strict task success rate ( Success ) and the average proportion of completed subtasks ( Average ). The best result is highlighted in bold, and the second-best result is underlined.
Figure 3: Performance across different applications, averaged over all evaluated models. Light and dark bars denote Success rate and Average scores, respectively. Orange denotes applications introduced in GMA , blue denotes applications inherited from MobileWorld, and green denotes Android system applications.
Images
Thought
Caption
Overall
Difficulty
Success
Average
Atomic
Comp.
Cross-App.
Realistic
1
✓
✗
43.33
57.20
72.83
61.67
47.30
0 9.97
2
✓
✗
49.33
63.78
80.37
69.78
51.87
11.00
2
✗
✗
41.67
55.77
80.37
57.13
43.03
0 6.89
2
✓
✓
51.33
65.13
78.31
70.65
51.50
34.60
4
✓
✗
51.00
65.53
80.37
71.89
54.64
12.86
Table 2: Effect of context retention on Qwen3.7-Plus. Images denotes the number of most recent screenshots retained in the context, Thought indicates whether previous reasoning traces are preserved, and Caption indicates whether older screenshots are retained as captions.
Figure 4: Overall performance with and without explicit state tracking under different numbers of retained screenshots.
Figure 5: Effect of state tracking across different difficulty tiers. The solid portion denotes the Average score without state tracking, while the lighter or hatched portion indicates the gain or loss after enabling state tracking. Numbers below each group denote the number of retained screenshots.
Model
State
Overall
Difficulty
Success
Average
Atomic
Comp.
Cross-App.
Realistic
Qwen
None
49.33
63.78
80.37
69.78
51.87
11.00
Free-form
50.33
64.73
85.16
71.27
49.37
0 7.89
Structured
54.33
68.41
86.30
74.07
53.46
24.92
GPT
None
62.67
75.86
86.99
81.60
64.21
43.64
Free-form
65.67
75.75
87.67
84.11
60.32
38.54
Table 3: Effect of different state-tracking designs across models. Free-form allows unconstrained state representation, while Structured uses predefined state fields.
Mobile planning requires a central agent to coordinate tools, persistent user context, reusable procedures, and specialized sub-agents while adapting to execution feedback. Evaluating these workflows requires distinguishing task completion from compliance with prescribed resource and procedural requirements. We introduce MobilePA-Bench, a benchmark comprising 1,705 tasks across eight query task domains and 212 mobile tools in a stateful simulated environment. It combines three features: (1) unified capability coverage, profiling Tool Use, Memory Usage, Skill Usage, and Sub-agent Collaboration under a common mobile protocol; (2) evidence-aligned verification, matching each task to a fixed tool-call, state-change, or interaction-behavior checker rather than imposing one completion criterion on all tasks; and (3) capability-gated evaluation, requiring task completion alongside prescribed memory checks and, in skill-only routing, complete route loading before business-tool use. With MobilePA-Bench, we systematically evaluate open- and closed-weight LLMs, revealing capability-specific weaknesses in complex mobile task execution. Fine-grained analyses distinguish failures in resource retrieval and procedural compliance from downstream execution errors, providing diagnostic insights beyond aggregate task-success rates. We further train a Qwen3.8-27B baseline with online reinforcement learning in the sandbox, whose improved performance provides initial evidence of the environment's utility for planner development.
Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.
Yunfei Feng, Xi Zhao, Cheng Zhang +5
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University · National Innovation Institute of High-end Smart Appliances
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.
Jintian Feng, Long Chen, Xiao Yu +7
Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI