Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Yiqi Zhu; Feiyu Gao; Jiaxing Fan; Jiahui Zeng; Minggang Wu; Chenliang Li; Haiyang Xu; Peng Li

cs.AIAug 21, 2026

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Authors: Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li,

Organizations: Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Alibaba Token Hub, Alibaba Group · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China

Abstract

Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.

Figures & tables

Figure 1: Overview of GMA . GMA provides a diverse and reproducible mobile ecosystem with seven custom applications, together with 300 tasks spanning four progressively more demanding tiers: Atomic, Compositional, Cross-Application, and Realistic. We further study the role of the agent harness, including context retention, explicit state tracking, and model-specific harness design. Evaluation on GMA reveals a substantial performance degradation as task complexity increases and highlights the growing importance of effective harness design for complex mobile workflows.

Figure 2: Screenshots of the seven custom applications developed for GMA . From left to right, the applications cover instant messaging, music library, lifestyle sharing, local discovery, food delivery, e-commerce, and travel planning, providing diverse interfaces, interaction patterns, and application-specific workflows for mobile agent evaluation.

Models	Overall		Atomic		Compositional		Cross-Application		Realistic
Models	Success	Average	Success	Average	Success	Average	Success	Average	Success	Average
Doubao-Seed-2.1-pro	62.00	74.42	89.04	89.73	65.89	79.27	41.03	62.99	20.00	31.75
Qwen3.7-Plus	49.33	63.78	78.08	80.37	51.94	69.78	30.77	51.87	0 0.00	11.00
Qwen3.8-Max	67.33	80.35	93.15	93.84	70.54	85.17	52.56	70.57	10.00	38.22
Gemini 3.5 Flash	54.33	65.91	80.82	81.74	62.02	75.23	30.77	50.64	0 0.00	0 7.56
GPT-5.5	59.67	71.14	87.67	87.67	65.12	78.71	39.74	58.78	0 0.00	10.25

Table 1: Overall and difficulty-specific performance of evaluated models on GMA . For each setting, we report the strict task success rate ( Success ) and the average proportion of completed subtasks ( Average ). The best result is highlighted in bold, and the second-best result is underlined.

Figure 3: Performance across different applications, averaged over all evaluated models. Light and dark bars denote Success rate and Average scores, respectively. Orange denotes applications introduced in GMA , blue denotes applications inherited from MobileWorld, and green denotes Android system applications.

Images	Thought	Caption	Overall		Difficulty
Images	Thought	Caption	Success	Average	Atomic	Comp.	Cross-App.	Realistic
1	✓	✗	43.33	57.20	72.83	61.67	47.30	0 9.97
2	✓	✗	49.33	63.78	80.37	69.78	51.87	11.00
2	✗	✗	41.67	55.77	80.37	57.13	43.03	0 6.89
2	✓	✓	51.33	65.13	78.31	70.65	51.50	34.60
4	✓	✗	51.00	65.53	80.37	71.89	54.64	12.86

Table 2: Effect of context retention on Qwen3.7-Plus. Images denotes the number of most recent screenshots retained in the context, Thought indicates whether previous reasoning traces are preserved, and Caption indicates whether older screenshots are retained as captions.

Figure 4: Overall performance with and without explicit state tracking under different numbers of retained screenshots.

Figure 5: Effect of state tracking across different difficulty tiers. The solid portion denotes the Average score without state tracking, while the lighter or hatched portion indicates the gain or loss after enabling state tracking. Numbers below each group denote the number of retained screenshots.

Model	State	Overall		Difficulty
Model	State	Success	Average	Atomic	Comp.	Cross-App.	Realistic
Qwen	None	49.33	63.78	80.37	69.78	51.87	11.00
	Free-form	50.33	64.73	85.16	71.27	49.37	0 7.89
	Structured	54.33	68.41	86.30	74.07	53.46	24.92
GPT	None	62.67	75.86	86.99	81.60	64.21	43.64
	Free-form	65.67	75.75	87.67	84.11	60.32	38.54

Table 3: Effect of different state-tracking designs across models. Free-form allows unconstrained state representation, while Structured uses predefined state fields.

Mobile Agent Benchmarks

See all 19 papers

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
Aug 24, 2026Yi Zhu, Xiongwei Wu, Qiyi Wang +8Mobile Agent Benchmarks LLM Agent Evaluation
MAI Team, Alibaba Token Hub, Alibaba Group
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
Feb 28, 2026Yunfei Feng, Xi Zhao, Cheng Zhang +5Computer-Use Agent Benchmarks Mobile Agent Benchmarks
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University · National Innovation Institute of High-end Smart Appliances
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Mobile GUI Automation GUI Agents
Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

Aug 24, 2026J/K move · Enter open · S save

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu+7

MAI Team, Alibaba Token Hub, Alibaba Group

Mobile Agent Benchmarks LLM Agent Evaluation Tool-Using Agents LLM Agent Planning Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning

13

functional domains and

212

realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: (1)~Sub-agent Collaboration---decomposing a complex task and delegating specialized work to capable sub-agents; (2)~Memory Usage---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and (3)~Skill Usage---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.

Abstract

Mobile Agent Benchmarks

Explore similar work

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents