From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks

Yuchen Sun; Chenglin Cai; Gongjie Zhang; Tianyu Xia; Quyu Kong; Panrong Tong; Zhengwen Zeng; Long Chen

cs.AISep 25, 2026

From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks

Authors: Yuchen Sun, Chenglin Cai, Gongjie Zhang, Tianyu Xia, Quyu Kong, Panrong Tong, Zhengwen Zeng, Long Chen,

Organizations: Shanghai Jiao Tong University · Tongyi Lab, Alibaba Group

Abstract

Mobile GUI agents complete tasks using GUI actions like taps and swipes. These actions are broadly applicable across applications, but reaching a navigation interface. A single deeplink call can replace a sequence of screen-by-screen GUI actions. We therefore introduce hybrid interaction, using deeplinks for direct navigation and GUI actions for other on-screen operations and fallback. To enable this, we discover candidate deeplinks through static analysis, validate them on real devices, and describe their observed landing screens. This process creates a verified and grounded deeplink catalog that pairs each working deeplink with a description of its landing screen. Using this catalog, we introduce GUI-Hopper, a improves task success in commercial applications on real devices, further demonstrating the benefits of hybrid interaction.

Figures & tables

Figure 1: From GUI-only to hybrid interaction. For a long-horizon mobile task spanning multiple applications, a GUI-only agent navigates screen by screen, requiring 15+ GUI steps (left). GUI-Hopper uses verified deeplinks to reach task-relevant screens directly and GUI actions for the remaining interaction (right), shortening the trajectory and reducing opportunities for error.

Figure 2: Overview of our approach. (a) A mining pipeline verifies deeplinks on real devices and builds a grounded catalog. (b) Hybrid trajectory synthesis and online reinforcement learning train GUI-Hopper in the hybrid action space. (c) At inference, the harness retrieves task-specific deeplink candidates and checks whether each call reaches its expected Activity, with GUI fallback on failure.

General-purpose VLMs
Model	Eval. with deeplink	SR (%) $\uparrow$	Avg. length (steps) $\downarrow$
Model	Eval. with deeplink	SR (%) $\uparrow$	Success-only	All
Claude Opus 4.6 ( Anthropic, 2026 )	$\times$	44.4	18.2	28.5
Gemini 3.1 Pro ( Google DeepMind, 2026 )	$\times$	58.1	21.1	25.6
GPT-5.6-Sol ( OpenAI, 2026 )	$\times$	70.1	21.9	26.5
Seed-2.0-Pro ( ByteDance Seed, 2026a )	$\times$	63.3	21.3	26.9

Table 1: Experiment results on MobileWorld. GUI-Hopper improves success and efficiency over GUI-only training across three model scales. Its success-rate gains persist under GUI-only evaluation.

General-purpose VLMs
Model	Eval. with deeplink	SR (%) $\uparrow$	Avg. length (steps) $\downarrow$
Model	Eval. with deeplink	SR (%) $\uparrow$	Success-only	All
Claude Opus 4.6 ( Anthropic, 2026 )	$\times$	75.0	21.8	25.9
Gemini 3.1 Pro ( Google DeepMind, 2026 )	$\times$	96.9	14.9	15.6
GPT-5.6-Sol ( OpenAI, 2026 )	$\times$	96.6	21.2	21.7
Kimi K2.6 ( Moonshot AI, 2026 )	$\times$	68.9	19.8	28.6

Table 2: Experiment results on MobileWorld-Real, a real-device mobile GUI agent benchmark. GUI-Hopper improves success and efficiency over GUI-only training on physical devices.

Stage	SR (%) $\uparrow$	Length $\downarrow$
Base (GUI)	15.7	32.9
+ Hybrid SFT	57.8	22.5
+ Online RL	60.1	20.9

Table 3: Training-stage ablation on Qwen3.5-4B. Online RL further improves success and efficiency after hybrid SFT.

Supply	Qwen3.7-plus		GUI-Hopper (9B)
Supply	SR (%) $\uparrow$	Length $\downarrow$	SR (%) $\uparrow$	Length $\downarrow$
NA	61.5	25.6	56.7	23.1
GT	62.4	21.4	63.8	21.7
Ten	59.5	21.4	63.5	21.8
Full	53.0	24.1	62.8	21.7

Table 4: Deeplink supply on MobileWorld. Task-matched deeplinks improve general VLM performance even without hybrid training. Our trained GUI-Hopper gains more from deeplinks and retains these gains when irrelevant candidates are included.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Application	Entries	Application	Entries
Bilibili	330	Huaxiaozhu	15
Xiaohongshu	177	Baidu Maps	11
Weibo	148	Meituan Dache	11
NetEase Cloud Music	128	NetEase News	9
Douyin	91	Tencent News	5
iQIYI	75	Baidu	4

Appendix

Table 5: Per-application counts in the verified deeplink catalog for MobileWorld-Real. The catalog contains 1,251 callable entries across 26 applications.

Figure 3: An AMap driving-route entry connecting model-facing descriptions with harness-side execution and landing verification. The deeplink opens route results for the specified origin and destination. Starting navigation remains a GUI action when required by the task.

Figure 4: Successful-trajectory length distributions under GUI-only evaluation on MobileWorld.

Task group	Model / Evaluation	SR (%) $\uparrow$	Avg. length (steps) $\downarrow$
			Success-only	All
Single-app	GUI-only / GUI	53.3	16.8	23.7
Single-app	GUI-Hopper / Hybrid	68.5	16.5	21.0
Multi-app	GUI-only / GUI	55.4	21.7	27.4
Multi-app	GUI-Hopper / Hybrid	59.1	18.5	22.4

Appendix

Table 6: Qwen3.5-9B performance by task structure on MobileWorld. Success gains are larger for single-app tasks, while trajectory reductions are larger for multi-app tasks.

Model size	GUI-only (s) $\downarrow$	GUI-Hopper (s) $\downarrow$
4B	346.5	181.7
9B	395.8	283.4
35B-A3B	493.9	212.9

Appendix

Table 7: Recorded task durations on MobileWorld. GUI-Hopper has lower recorded times at all three scales, although the two models use different timing methods.

Model size	GUI-only (s) $\downarrow$	GUI-Hopper (s) $\downarrow$
4B	424.8	378.0
9B	273.0	265.2
35B-A3B	271.2	245.4

Appendix

Table 8: Execution time on MobileWorld-Real. GUI-Hopper reduces mean elapsed time by 2.9–11.0% relative to the GUI-only trained model.

Figure 5: Complete successful trajectories for an interview-invitation task on MobileWorld. GUI-Hopper takes five steps versus 26 for the GUI-only baseline. Direct navigation and SMS prefilling shorten the workflow, while GUI actions remain necessary to read the resume and send the message. Screenshots show pre-action observations, and step counts include the final answer.

Figure 6: GUI-only trajectories and model reasoning for an interview-email task on MobileWorld. Both 35B-A3B models succeed without deeplinks. GUI-Hopper obtains the recipient’s address from the resume earlier, completing the task in 14 steps compared with 32 for the GUI-only baseline. Screenshots show observations before each action, and step counts include the final answer.

Figure 7: GUI recovery after incorrect deeplink use on a MobileWorld calendar task. The call reaches the expected Activity but prefills the wrong date, showing why the model must also check task-relevant screen contents. GUI-Hopper later recreates the event through GUI actions, taking 19 steps versus 18 for the GUI-only baseline. Removal of the earlier wrong-date event is not shown.

Mobile GUI Agents

Mobile GUI agents use visual and structured interface information to carry out tasks across smartphone apps.

See all 84 papers

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
Sep 28, 2026Yangqin Jiang, Lingrui Xu, Chao HuangMobile GUI Automation Mobile GUI Agents
The University of Hong Kong
UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
May 28, 2026Yuxiang Chai, Han Xiao, Xinyu Fu +3Mobile GUI Automation GUI Agents
1CUHK MMLab · 2Huawei Research · 3Shenzhen Loop Area Institute +1
AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents
Apr 22, 2026Jeonghyeon Kim, Byeongjun Joung, Junwon Lee +3Mobile GUI Automation Computer-Use Agents
Department of Computer Science and Engineering Sungkyunkwan University (SKKU) Suwon, Republic of Korea

Sep 28, 2026J/K move · Enter open · S save

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

Yangqin Jiang, Lingrui Xu, Chao Huang

The University of Hong Kong

Mobile GUI Automation Mobile GUI Agents

Open paper AskPDF arXiv

Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.

Abstract

Mobile GUI Agents

Explore similar work

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents

AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents