Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

Xiaokai Yan; Jingtao Ding; Yong Li; Zhiwen Yu

cs.HCAug 14, 2026

Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

Authors: Xiaokai Yan, Jingtao Ding, Yong Li, Zhiwen Yu

Organizations: Northwestern Polytechnical University, China · Tsinghua University, China

Abstract

Mobile GUI Agents powered by multimodal large language models (MLLMs) show promise in human-computer intelligence. However, current research primarily focuses on reactive task execution while lacking a comprehensive understanding-prediction-execution process for user intentions, which are the core requirements of active agents. In this paper, we propose the Act2Intention framework that builds an active mobile agent by integrating understanding, predicting user intentions, and executing decisions. First, we construct the Act2Intention Bench through data collection and validated generation, comprising 72,511 intentions and over 700,000 actions across 52 apps, thereby establishing the first benchmark for evaluating proactive agents via continuous intention-action trajectories. We further develop the Act2Intention Agent, achieving proactive services through Proactive-oriented Intention Understanding, Personalized Proactive Intention Prediction, and Experience-guided Intention Execution. Experimental results show that supervised fine-tuning on Act2Intention Bench yields absolute improvements of +32.0 Acc-S, +10.25 Acc-S, and +6.9 SSR points over non-fine-tuned counterparts under the same agent framework for intention understanding, prediction, and execution, respectively. This success underscores the necessity and value of the Act2Intention Bench, which establishes a standardized platform for developing and evaluating proactive agents and consequently paves the way for research on intention-driven human-computer interaction.

Figures & tables

Figure 1. The Act2Intention Framework illustrates a shift from reactive to active agents . (1) Reactive Agent can only passively assist until users send instructions. (2) Active agent understands, predicts the user’s intention through GUI actions, and thereby provides active services (Understanding $\rightarrow$ Prediction $\rightarrow$ Execution). A toy example of Act2Intention Bench is shown on the right.

Dataset	Episodes/Events	Platform	Avg.Steps	Continuity?
AndroidControl	15283	Mobile	5.5	no
AITW	715,142	Mobile	6.5	no
AITZ	2,504	Mobile	7.5	no
AMEX	8,000	Mobile	12.8	no
GUI-Odyssey	7,735	Mobile	15.4	no
ProactiveBench	6,790	Windows	-	yes

Table 1. Dataset comparison.

Notation	Meaning	Notation	Meaning
$a_{t}$	Atomic GUI action at step $t$	$o_{t}$	GUI observation (screenshot + context) at step $t$
$d_{t}$	Natural-language description of $a_{t}$	$I_{i}$	The $i$ -th user intention
$\tau_{i}$	Action trajectory for intention $I_{i}$	$\hat{I}_{t+1}$	Predicted next intention
$\tau_{i}^{des}$	Action description trajectory for $I_{i}$	$p_{i},\,q_{i}$	Start/end indices of the $i$ -th intention
$T$	Current timestamp	$m$	Total number of intentions in a session
$P$	User persona	$f_{\phi},f_{\theta},f_{\pi}$	Models for understanding, prediction, execution

Table 2. Summary of key notations used in this paper.

Figure 2. Overview of bench construction . We generate trajectory data using both real and synthesized personas with LLMs and GUI Agents, followed by dual verification.

Stage	Example
Persona	A weather-conscious and socially active individual who frequently checks weather conditions through MyObservatory across multiple cities. This user maintains social connections via WhatsApp and Discord, shops online at Etsy, and navigates with Google Maps. Usage is concentrated in the afternoon and morning hours, suggesting an outdoor enthusiast who plans activities around weather conditions.
Scenario	Weekday schedule involving weather monitoring (MyObservatory), online shopping (Etsy), social messaging (WhatsApp), and nearby restaurant exploration (Google Maps).
Generated intentions	(1) 09:07, MyObservatory: Ask the Dr. Tin chatbot about the expected sunset time in New York.
	(2) 17:10, Etsy: Search “coffee mugs” priced $5–$ 20, sort by highest price, and add the third item to cart.
	(3) 17:40, Google Maps: Explore nearby restaurants rated over 4 stars and get walking directions to the nearest one.
Action trajectories	For the Etsy intention, the agent executes the following steps:

Table 3. An example of persona-conditioned trajectory generation.

Stage	Generated	Retained	Filtered	Retention
Intention filter	73640	68210	5430	92.63%
Action filter	68210	59362	8848	87.03%

Table 4. Synthetic-data filtering.

Subset	Personas	Intents	P.C.Intents	Actions
Act2Intention	360	72511	161.14	705366
Act2Intention-RR	90	13149	146.10	134542
Act2Intention-RG	90	17678	196.42	181131
Act2Intention-GG	270	41684	154.39	389693

Table 5. Statistics of Act2Intention Bench.

Subset	Realism $\uparrow$	Achievability $\uparrow$	Persona Consistency $\uparrow$
Act2Intention-RR	4.19	4.62	4.30
Act2Intention-RG	3.91	4.00	4.06
Act2Intention-GG	3.55	3.79	4.01

Table 6. Human realism assessment of real and synthetic trajectories.

Panel of Figure 3 — Figure 3. Dataset statistics and distribution.

Figure 4. Overview of Act2Intention Agent: Understanding Intentions from Actions, Predicting Needs Proactively, and Executing Tasks with Experience.

Models	Understanding			Prediction
Models	Acc-G	Acc-S	BLEU-4	Acc-C	Acc-S	BLEU-4
Closed-source models
GPT-3.5-turbo	16.31	0.17	1.91	42.50	0.29	17.13
GPT-4o-mini	20.66	0.21	2.31	38.50	0.31	18.36
GPT-4o	20.08	0.25	5.32	34.50	0.32	16.74
Claude-3.5-haiku	24.28	0.24	4.12	28.50	0.17	14.27

Table 7. Evaluation results of Intention Understanding and Intention Prediction on the ID test set. All of the fine-tuned models achieve significant performance improvements on all metrics.

Table 12

Table 13

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Overview of the data collection process.

Action	Description
CLICK[x, y]	Click on the position with coordinates [x, y]
LONG_CLICK[x, y]	Long-click at the position with coord. [x, y]
TYPE[text]	Input the content of “text” in the input box
SWIPE[direction]	Swipe in the specified direction
PRESS[KEY]	Press a key in “HOME”,“ENTER”,“BACK”

Appendix

Table 12. Action space.

Panel of Figure 6 — Figure 6. Error analysis of semantic accuracy under correct segmentation.

Figure 7. Distribution of confidence scores. The dashed green line denotes the threshold $\theta=0.60$ .

Setting	threshold	Push Rate (%)	Acc-S ${}_{\text{push}}$	SR ${}_{\text{push}}$
w/o fallback	–	100.0	0.31	22.7
w/ confidence fallback	0.80	14.0	0.44	34.6
w/ confidence fallback	0.60	40.0	0.39	29.4

Appendix

Table 13. Confidence-based fallback with Qwen-2.5-7B-SFT and UI-TARS-7B-SFT. The threshold

\theta=0.60

is the default decision threshold, while

\theta=0.80

is a stricter test setting.

Method	Understanding				Prediction
	ID		OOD		ID		OOD
	Acc-G	Acc-S	Acc-G	Acc-S	Acc-C	Acc-S	Acc-C	Acc-S
Gemini-1.5-pro	68.63	0.32	68.78 [+0.15]	0.31 [-0.01]	34.50	0.36	35.00 [+0.50]	0.36 [-0.00]
Llama-3.1-8B	6.03	0.11	6.13 [+0.10]	0.28 [+0.17]	37.80	0.33	37.50 [-0.30]	0.32 [-0.01]
Llama-3.1-8B-SFT $\dagger$	97.31	0.47	84.58 [-12.73]	0.32 [-0.15]	54.20	0.39	43.55 [-10.65]	0.34 [-0.05]
Deepseek-7B	6.12	0.19	5.87 [-0.25]	0.19 [+0.00]	46.89	0.34	40.93 [-5.96]	0.28 [-0.06]

Appendix

Table 14. Generalization evaluation. Red indicates the performance difference between the ID and OOD test sets.

Panel of Figure 8 — Figure 8. Performance comparison of different trajectory sequence lengths. (a) represents the process of understanding segmentation accuracy and semantic accuracy; (b) represents the process of prediction category accuracy and semantic accuracy.

Panel of Figure 9 — Figure 9. Performance of different training sets. We fine-tuned Qwen-2.5-7B; (a) shows the results on intention understanding, and (b) shows the results on intention prediction. The combination “RR+RG+GG” achieves the highest overall performance in this experiment, suggesting that real and generated data are complementary for improving model generalization.

Panel of Figure 10 — Figure 10. Effect of Persona on Intention Prediction.

Figure 11. A case of Intention Understanding and Prediction.

Explore similar work

ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices
Feb 25, 2026Dezhi Kong, Zhengzhao Feng, Qiliang Liang +12AI Agent Benchmarks Agentic AI
HyperAI Team, Xiaomi Corporation · Zhejiang University · Peking University +1
PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents
Mar 9, 2026Yuxiang Chai, Shunye Tang, Han Xiao +2GUI Agents AI Agent Benchmarks
Nankai University · Huawei Research
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Apr 6, 2026Rongqian Chen, Yu Li, Zeyu Fang +3Reward Modeling Computer-Use Agents
George Washington University, Washington D.C., USA

Feb 25, 2026J/K move · Enter open · S save

ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices

Dezhi Kong, Zhengzhao Feng, Qiliang Liang, Hao Wang+11

HyperAI Team, Xiaomi Corporation · Zhejiang University · Peking University+1

AI Agent Benchmarks Agentic AI Multimodal Agents

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have made significant progress in mobile agent development, yet their capabilities are predominantly confined to a reactive paradigm, where they merely execute explicit user commands. The emerging paradigm of proactive intelligence, where agents autonomously anticipate needs and initiate actions, represents the next frontier for mobile agents. However, its development is critically bottlenecked by the lack of benchmarks that can address real-world complexity and enable objective, executable evaluation. To overcome these challenges, we introduce ProactiveMobile, a comprehensive benchmark designed to systematically advance research in this domain. ProactiveMobile formalizes the proactive task as inferring latent user intent across four dimensions of on-device contextual signals and generating an executable function sequence from a comprehensive function pool of 63 APIs. The benchmark features over 3,660 instances of 14 scenarios that embrace real-world complexity through multi-answer annotations. To ensure quality, a team of 30 experts conducts a final audit of the benchmark, verifying factual accuracy, logical consistency, and action feasibility, and correcting any non-compliant entries. Extensive experiments demonstrate that our fine-tuned Qwen2.5-VL-7B-Instruct achieves a success rate of 19.15%, outperforming o1 (15.71%) and GPT-5 (7.39%). This result indicates that proactivity is a critical competency widely lacking in current MLLMs, yet it is learnable, emphasizing the importance of the proposed benchmark for proactivity evaluation.

Abstract

Explore similar work

ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices

PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents

IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents