MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

Yunfei Feng; Xi Zhao; Cheng Zhang; Dahu Feng; Daolin Cheng; Jianqi Yu; Yubin Xia; Erhu Feng

cs.AIFeb 28, 2026

MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

Authors: Yunfei Feng, Xi Zhao, Cheng Zhang, Dahu Feng, Daolin Cheng, Jianqi Yu, Yubin Xia, Erhu Feng

Organizations: Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University · National Innovation Institute of High-end Smart Appliances

Abstract

Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.

Figures & tables

Figure 2: Modeling agent-device interactions with state transition graph . Executing actions triggers state transitions. Completing a task corresponds to reaching a terminal state.

Benchmark	#APP	#Task	Interactive	Real App	No Dependencies	Multi-Solution	Evaluation
Mobile-Bench-v2 ( Xu et al., 2025a )	49	12,856	✗	✓	✓	✓	Trajectory-based
AndroidControl ( Leung et al., 2025 )	-	14,548	✗	✓	✓	✗	Trajectory-based
MobileAgentBench ( Wang et al., 2024 )	100	10	✓	✗	✗	✓	Result-based
AndroidWorld ( Rawles et al., 2024 )	20	116	✓	✗	✗	✓	Result-based
AndroidLab ( Xu et al., 2025a )	9	138	✓	✗	✗	✓	Result-based
SPA-Bench ( Chen et al., 2024 )	66	340	✓	✓	✗	✓	Result-based

Table 1: Comparison of different datasets and environments for benchmarking Mobile GUI agents. Column definitions: #Task. (number of tasks), #Apps (number of applications), Interactive (Support environmental interaction), Real App (Including any third-party apps), No Dependencies (Does not need additional software dependencies, such as an Android emulator), Multi-Solution (supports solving tasks in multiple ways), Evaluation (Evaluation Strategy)

Figure 3: Comparison of the Distribution of Interactive Element Counts Across Different Modeling Approaches . The left figure presents the app-based modeling approach, while the right figure illustrates the task-based modeling approach.

Figure 4: Algorithm workflow illustration: For states with identical labels (or equivalently, similar transition structures), we merge them and allow the merged state to share their transition conditions, thereby compressing the state complexity.

Figure 5: Special scenarios including instruction following, instruction interference, application interference, and open exploration (to observe whether the model can deviate from erroneous paths).

Metric	Formula	Notation Explanation	Description
Success Rate (SR)	$\frac{1}{N}\sum_{i=1}^{N}\text{suc}(i)$	$N$ : number of tasks; $\text{suc}(i)=1$ if $g_{i}$ is completed, else $0$ .	Indicates whether a task is fully completed, focusing only on the final outcome.
Completion Rate (CR)	$\max\!\left\{\frac{d(s_{\text{visited}},s_{\text{goal}})}{d(s_{\text{start}},s_{\text{goal}})}\right\}$	$s_{\text{start}},s_{\text{goal}}$ : initial and goal states; $d(\cdot,\cdot)$ : shortest-path distance.	Measures the maximum progress achieved toward the goal state.
Coverage Rate (CVR)	$\frac{\sum_{i}\|\mathcal{S}_{\text{visited}}\|_{i}}{\sum_{i}\|\mathcal{S}\|_{i}}$	$\|\mathcal{S}_{\text{visited}}\|_{i}$ : visited states in task $i$ ; $\|\mathcal{S}\|_{i}$ : total states.	Evaluates exploration by the proportion of states observed.
Action Match Rate(AMR)	$\frac{1}{N}\sum_{i}\frac{\sum_{j}am[i][j]}{\|IF_{i}\|}$	$\|IF_{i}\|$ : number of instructions; $am[i][j]=1$ if instruction $j$ is satisfied.	Measures instruction-following ability and safety.
Time to Action (TTA)	$\frac{1}{N}\sum_{i}\frac{\sum_{j}t_{ij}}{\|\tau_{i}\|}$	$\|\tau_{i}\|$ : number of interactions; $t_{ij}$ : time to generate action $j$ .	Captures execution efficiency via average action latency.

Table 2: Evaluation metrics defined on the state-transition graph.

Figure 6: Task statistics for MobiFlow , covering the type, complexity, and application domain of basic-scenario tasks, as well as the quantity of special-scenario tasks.

			Base Scenario			Following		Noisy Scenario		Open Exploration
Model	Size(B)	TTA(s) $\downarrow$	SR(%) $\uparrow$	CR(%) $\uparrow$	CVR(%)	AMR(%) $\uparrow$	SR(%) $\uparrow$	CR1(%) $\uparrow$	CR2(%) $\uparrow$	CR(%) $\uparrow$	CVR(%)
General Model
Gemini-2.5-Flash	-	17.03	30.7	41.7	17.4	35.5	33.3	35.5	37.7	53.2	21.1
Gemini-2.5-Pro	-	30.12	38.7	52.1	26.9	55.3	41.6	56.4	51.7	59.7	28.1
Gemini-3.0-Flash	-	33.31	40.2	59.7	30.2	56.4	41.6	64.7	58.4	65	34.3
Grok4	-	142.21	8.3	12.3	8.41	4.1	8.3	8.1	10.0	17.8	10.2

Table 3: Main Results on MobiFlow .Includes general-purpose and specialized models. TTA (Time to Action), SR (Success Rate), CR (Completion Rate), AMR (Action Matching Rate), CVR (Coverage Rate). CR1 (with instruction noise), CR2 (under APP interference).

Figure 7: Model execution efficiency. GUI models use the same inference framework and operate under identical network request environments.

Model	No Instruction Noise AMR	Instruction Noise AMR
General Model
Gemini-2.5-Flash	35.55	38.55 +3.0
Gemini-2.5-Pro	55.34	56.48 +1.1
Claude-sonnet-4.5	66.38	67.38 +1.0
GPT-5	71.38	71.07 -0.3
GUI specialized Model

Table 4: Robustness to Instruction Noise : A Comparison of General Models vs. GUI Models

Figure 8: Correlation Analysis . Correlation between General Models’ Performance in Specific Scenarios and Different Benchmark Capabilities.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Efficient trajectory collection tools . A trajectory collection framework based on real devices and actual applications.

Action	Parameter	Description
GUI Operation
click	x,y	Click at the specified coordinates
input	string	Type text into the focused field
swipe	start_x,start_y,end_x,end_y	Swipe from start to end coordinates
wait	–	Wait for update
double_click	x,y	Double Click at the specified coordinates

Appendix

Table 5: Action types and their parameters in the task execution framework.

Figure 10: This figure illustrates the statistical distribution of transition action counts on individual interfaces across different mobile applications(Due to space limitations, we present only a subset of the statistical results).

Figure 11: Visualization Examples of Task State Transition Graphs , including Different Tasks from Ele.me, AutoNavi, and NetEase Cloud Music Apps.

Figure 12: Execution Example of General Models : General models leverage icon recognition models for enhancement.

Figure 13: The tension between generalization capability and determinism . The upper figure presents the performance scores of a general-purpose model across multiple sampling trials on randomly selected tasks. The lower figure shows the sampling results of a GUI-specialized model on the same set of tasks.

Figure 14: Completion rate of different models under varying resolution scaling factors . We randomly sample representative examples and evaluate them at different scaling levels based on a resolution of 1080×2400.

Computer-Use Agent Benchmarks

Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.

See all 119 papers

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
Aug 21, 2026Yiqi Zhu, Feiyu Gao, Jiaxing Fan +7Mobile Agent Benchmarks AI Agent Evaluation
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Alibaba Token Hub, Alibaba Group · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Mobile GUI Automation GUI Agents
Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI
OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools
Oct 9, 2026Chenyang Yan, Yingying Zhang, Kanzhi Cheng +21Mobile Agent Benchmarks Mobile GUI Agents
Nanjing University · SenseTime · The University of Hong Kong +3

Aug 21, 2026J/K move · Enter open · S save

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng+6

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Alibaba Token Hub, Alibaba Group · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China

Mobile Agent Benchmarks AI Agent Evaluation Mobile GUI Agents Agent Harness Optimization

Open paper AskPDF arXiv

Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.

Abstract

Computer-Use Agent Benchmarks

Explore similar work

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools