ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices

Dezhi Kong; Zhengzhao Feng; Qiliang Liang; Hao Wang; Haofei Sun; Changpeng Yang; Yang Li; Peng Zhou

cs.AIFeb 25, 2026

ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices

Authors: Dezhi Kong, Zhengzhao Feng, Qiliang Liang, Hao Wang, Haofei Sun, Changpeng Yang, Yang Li, Peng Zhou, +7 more

Organizations: HyperAI Team, Xiaomi Corporation · Zhejiang University · Peking University · Northeastern University

Abstract

Multimodal large language models (MLLMs) have made significant progress in mobile agent development, yet their capabilities are predominantly confined to a reactive paradigm, where they merely execute explicit user commands. The emerging paradigm of proactive intelligence, where agents autonomously anticipate needs and initiate actions, represents the next frontier for mobile agents. However, its development is critically bottlenecked by the lack of benchmarks that can address real-world complexity and enable objective, executable evaluation. To overcome these challenges, we introduce ProactiveMobile, a comprehensive benchmark designed to systematically advance research in this domain. ProactiveMobile formalizes the proactive task as inferring latent user intent across four dimensions of on-device contextual signals and generating an executable function sequence from a comprehensive function pool of 63 APIs. The benchmark features over 3,660 instances of 14 scenarios that embrace real-world complexity through multi-answer annotations. To ensure quality, a team of 30 experts conducts a final audit of the benchmark, verifying factual accuracy, logical consistency, and action feasibility, and correcting any non-compliant entries. Extensive experiments demonstrate that our fine-tuned Qwen2.5-VL-7B-Instruct achieves a success rate of 19.15%, outperforming o1 (15.71%) and GPT-5 (7.39%). This result indicates that proactivity is a critical competency widely lacking in current MLLMs, yet it is learnable, emphasizing the importance of the proposed benchmark for proactivity evaluation.

Figures & tables

Figure 1: A comparison of proactive and reactive paradigms in mobile agents.

Figure 2: The overview of data generation.

Dataset	Description	Usage
GUI-Odyssey [ 25 ]	A dataset for cross-app mobile GUI navigation	21,669
AITZ [ 50 ]	Largest dataset in the Android GUI navigation field	9,413
CAGUI [ 53 ]	A real-world Chinese Android GUI benchmark	3,196
MobileAgentBench	Self-collected GUI data from mobile devices	12,481

Table 1: Summary of GUI datasets.

Figure 3: Distribution of the 14 primary user intent categories, demonstrating the benchmark’s broad scenario coverage (e.g., Personal Management, Office Work, Food & Dining).

Split	Data Type	Scenarios	Items	Intents	Images	Ave. Image	Functions	Ave. Functions	L1	L2	L3
Train	Multimodal	12	4,438	8,977	32,418	7.30	9,964	1.11	308	1,376	2,754
Train	Text	12	4,438	4,438	-	-	8,259	1.86	372	1,208	2,858
Test	Multimodal	14	1,832	3,711	14,341	7.83	4,173	1.24	118	613	1,101
Test	Text	14	1,828	2,676	-	-	2,266	0.85	259	711	858

Table 2: Statistics of the ProactiveMobile dataset, broken down by Train and Test splits and data modality. The table details the composition of our benchmark, including the number of scenarios, items, intents, functions, and distribution of different difficulties. Notably, the test set includes two additional scenarios (14 vs. 12) not present in the training set, forming our dedicated out-of-distribution evaluation split.

Model	L1						L2
	Multimodal		Text		All		Multimodal		Text		All
	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓
GPT-5	12.71	75.00	24.32	25.57	20.69	30.45	7.50	69.44	17.42	30.24	12.81	37.79
GPT-4o	11.86	100.00	13.13	47.06	12.73	51.79	5.71	95.60	10.58	56.29	8.32	62.87
o1	22.03	64.29	30.50	2.73	27.85	9.68	11.91	37.76	24.90	6.23	18.88	13.09
Gemini-2.5-Pro	16.10	69.57	13.51	59.09	14.32	60.18	12.07	70.53	6.61	84.74	9.14	81.90

Table 3: Overall performance comparison of our fine-tuned model ( +Proactive ) against baselines on the ProactiveMobile test set. We report two key metrics: SR ↑ , where higher is better, and FTR ↓ , where lower is better. The comparison is broken down by task difficulty (L1-L3) and data modality. For each metric, the best result is in bold and the second-best is underlined . All scores are in percentage (%).

Model	SR ↑	FTR ↓
GPT-5	14.55	36.84
GPT-4o	5.26	47.37
o1	15.63	13.33
Gemini-2.5-Pro	9.38	45.46
Qwen2.5-VL-7B-Instruct	4.69	42.86
MiMo-VL-7B-SFT-2508	0.00	100.00

Table 4: Performance on the OOD test set. This set comprises 64 instances from two scenarios (Logistics Delivery and Smart Home) that are entirely absent from the training data.

Training Strategy	Multimodal		Text		All
Training Strategy	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓
Func.	12.87	95.07	5.69	85.19	9.18	93.16
Rec.+Func.(ours)	15.61	23.91	26.04	8.51	20.82	13.76
Think+Func.	8.81	95.07	4.04	85.04	6.36	93.06
Think+Rec.+Func.	5.53	2.87	10.50	1.53	8.02	2.06

Table 5: Ablation study on the impact of different output formats. We compare our primary Text Recommendation + Function strategy against variants that only output the Function , or include an additional reasoning ( Think ) step.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Detailed dataset statistics and distribution.

Figure 5: Exemplary implementation of the book_transport API.

Difficulty	Model	Multimodal				Text				All
Difficulty	Model	Type-Acc ↑	F1 ↑	P ↑	R ↑	Type-Acc ↑	F1 ↑	P ↑	R ↑	Type-Acc ↑	F1 ↑	P ↑	R ↑
L1	GPT-5	33.05	58.50	56.64	65.25	67.57	70.91	71.04	71.36	56.76	67.03	66.53	69.45
	GPT-4o	30.51	49.70	46.85	57.91	45.17	48.82	48.46	49.81	40.58	49.09	47.95	52.34
	o1	51.70	58.48	59.75	58.05	91.12	92.41	92.47	92.47	78.78	81.79	82.23	81.70
	Gemini-2.5-Pro	55.08	63.70	65.25	63.14	40.93	43.50	44.02	43.44	45.36	49.82	50.66	49.60
	Qwen2.5-VL-7B-Instruct	3.39	3.39	3.39	3.39	7.72	7.72	7.72	7.72	6.37	6.37	6.37	6.37

Appendix

Table 6: Detailed performance comparison using granular, set-based metrics. We report on function name sequence accuracy (Type-Acc ↑ ), F1-Score ↑ , Precision (P ↑ ), and Recall (R ↑ ) across different difficulties and modalities. Best results are in bold , and second-best are underlined . All scores are in percentage (%). Qwen2.5-VL-7B +Proactive represents Qwen2.5-VL-7B-Instruct + Proactive, and MiMo-VL-7B-SFT +Proactive represents MiMo-VL-7B-SFT-2508 + Proactive.

Input	Model	Multimodal		Text		All
Input	Model	SR ↑	FTR ↓	SR ↑	FTR ↓	SR ↑	FTR ↓
w/o Profile	GPT-5	8.41	40.30	14.00	27.76	11.20	31.87
	GPT-4o	5.19	89.42	11.43	45.35	8.31	57.78
	o1	12.12	17.28	20.02	5.18	16.07	9.43
	Gemini-2.5-Pro	10.70	60.10	8.59	77.42	9.65	71.55
	Qwen2.5-VL-7B-Instruct	1.91	55.86	2.52	60.16	2.21	58.60

Appendix

Table 7: Ablation study on the impact of contextual information dimensions. We evaluate model performance after systematically removing one of the four key dimensions: User Profile, Device Status, World Information, and Behavioral Trajectories. The “All Info” row represents the full model performance with no information removed, serving as the baseline. Metrics are Success Rate (SR ↑ ) and False Trigger Rate (FTR ↓ ). Best results are in bold , and second-best are underlined . All scores are in percentage (%). Note that the ‘w/o Trajectories‘ experiment removes the distinction between Multimodal and Text data.

User Profile	Based on the user profile and historical behavior analysis, the user is 25 years old and a photography enthusiast who frequently browses digital camera review websites. Recently, the user has repeatedly searched for the “Sony Alpha 7 IV.” The user also has a habit of purchasing products from U.S. e-commerce platforms via cross-border shopping (“Haitao”). On the phone, the user has installed international remittance apps such as Wise and has a record of using them, showing a preference for payment channels with lower transaction fees. Recently, the user has also browsed content related to travel and geography, such as searching for “the capital of China.”
Device Status	The current device time is 9:45 AM on Tuesday, October 11. The system’s primary language is set to English (Australia). The device is located in Sydney, Australia. The phone battery level is sufficient at 91%, and it is connected to a stable Wi-Fi network. Installed applications include Amazon, Wise, Gmail, Chrome, and YouTube. Recent notifications include a shopping cart reminder from Amazon: “Items in your cart are still waiting for you!”
World Information	Macroeconomic information indicates that the global financial market is currently in an active trading period. According to real-time data, the current USD/AUD interbank exchange rate is approximately 1.58, although most payment channels typically charge additional fees on top of this rate. A financial news report notes that cross-border payment services provided by emerging fintech companies often offer more favorable exchange rates than traditional banks. An international news summary also reports that the European Space Agency is preparing a new Mars exploration program.
Behavioral Trajectories
Thinking	The system detects that the user has pending items in their Amazon cart (notification) and is currently checking the USD/AUD exchange rate (interaction trajectory), suggesting a cross-border payment intent. Given the user’s preference for using Wise due to its lower fees (user profile), the system proactively recommends comparing exchange rates through Wise and completing the payment there, potentially helping the user save money.
Text Intent Label	It has been detected that you have items waiting for checkout in your Amazon shopping cart and that you are currently checking the USD to AUD exchange rate. Considering that you have a habit of using Wise for cross-border payments—and that Wise may offer more competitive exchange rates—would you like to open the Wise app to check the rate and complete the payment?

Appendix

Table 8: Case study.

User Profile	根据用户画像和历史行为分析，该用户年龄为25岁，是一位摄影爱好者，频繁浏览数码相机评测网站，近期多次搜索’索尼Alpha 7 IV’。有从美国电商网站’海淘’商品的习惯。该用户手机上安装了’Wise’等国际汇款应用，并有使用记录，偏好使用低手续费的支付渠道。近期有浏览关于旅游和地理知识的内容，例如查询过’中国的首都’。
Device Status	设备当前时间为上午9点45分，日期为10月11日，星期二。系统主要语言设定为’英语(澳大利亚)’。地理位置在澳大利亚悉尼。手机电量充足，为91%，并已连接至稳定的Wi-Fi网络。设备上安装了’亚马逊’、’Wise’、’Gmail’、’Chrome’和’YouTube’等应用。最近的通知消息包括一条来自’亚马逊’的购物车提醒：’您购物车中的商品仍在等待您！’
World Information	宏观信息显示，全球金融市场正处于活跃交易时段。根据实时数据，当前美元兑澳元(USD/AUD)的银行间汇率约为’1.58’，但多数支付渠道会在此基础上收取额外费用。一条财经新闻提到，新型金融科技公司提供的跨境支付服务通常比传统银行更具汇率优势。一条国际新闻摘要显示，欧洲航天局正在筹备新的火星探测计划。
Behavioral Trajectories
Thinking	检测到用户有亚马逊待支付商品（通知），并正在查询美元/澳元汇率（操作轨迹），表明其有跨境支付意图。结合用户偏好使用低费率Wise（画像）的习惯，主动推荐使用Wise比较汇率并完成支付，以满足其潜在的省钱需求。
Text Intent Label	检测到您在亚马逊购物车有商品等待结算，并且您正在查询美元兑换澳元的汇率。考虑到您有使用Wise进行跨境支付的习惯，并且Wise可能提供更优惠的汇率，是否前往Wise App查看汇率并完成支付？

Appendix

Table 9: Case study.

Model	Prediction
Label	⬇ [ { " name ": " bank_service_assistance ", " parameters ": { " bank_name ": " Wise ", " service_type ": " foreign exchange service ", " location_constraint ": "", " tasks ": " check exchange rate " } } ]
GPT-5	No recommendation.
o1	No recommendation.
Qwen2.5-VL-7B-Instruct + Proactive	⬇ [ { " name ": " bank_service_assistance ", " parameters ": { " bank_name ": " Wise ", " service_type ": " foreign exchange service ", " location_constraint ": " global " } } ]

Appendix

Table 10: Case study.

Explore similar work

Aug 14, 2026

Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

Mobile GUI Agents powered by multimodal large language models (MLLMs) show promise in human-computer intelligence. However, current research primarily focuses on reactive task execution while lacking a comprehensive understanding-prediction-execution process for user intentions, which are the core requirements of active agents. In this paper, we propose the Act2Intention framework that builds an active mobile agent by integrating understanding, predicting user intentions, and executing decisions. First, we construct the Act2Intention Bench through data collection and validated generation, comprising 72,511 intentions and over 700,000 actions across 52 apps, thereby establishing the first benchmark for evaluating proactive agents via continuous intention-action trajectories. We further develop the Act2Intention Agent, achieving proactive services through Proactive-oriented Intention Understanding, Personalized Proactive Intention Prediction, and Experience-guided Intention Execution. Experimental results show that supervised fine-tuning on Act2Intention Bench yields absolute improvements of +32.0 Acc-S, +10.25 Acc-S, and +6.9 SSR points over non-fine-tuned counterparts under the same agent framework for intention understanding, prediction, and execution, respectively. This success underscores the necessity and value of the Act2Intention Bench, which establishes a standardized platform for developing and evaluating proactive agents and consequently paves the way for research on intention-driven human-computer interaction.

Xiaokai Yan, Jingtao Ding, Yong Li +1

Northwestern Polytechnical University, China · Tsinghua University, China

Mobile GUI Agents Agentic AI AI Agent Benchmarks

Ask PDF

Apr 16, 2026

From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench

Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.

Ke Xu, Yuhao Wang, Yu Wang

Shanghai Jiao Tong University

LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks Multimodal Model Evaluation

Ask PDF

Oct 22, 2025

Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously. However, evaluating proactivity is challenging; current benchmarks are constrained to localized context, limiting their ability to test reasoning across sources and longer time horizons. To address this gap, we present PROBE (Proactive Resolution Of BottlEnecks). PROBE decomposes proactivity as a pipeline of three core capabilities: (1) searching for unspecified issues, (2) identifying specific bottlenecks, and (3) executing appropriate resolutions. We apply PROBE to evaluate leading LLMs and popular agentic frameworks, showing that even state-of-the-art models struggle to solve this benchmark. Computing our consistent measurements across frontier LLMs and agents, we find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1. Additionally, we demonstrate the relative capabilities of each model and analyze mutual failure modes. Our results highlight the current limitations of autonomous action in agentic systems, and expose promising future research directions.

Gil Pasternak, Dheeraj Rajagopal, Julia White +4

LLM Agent Evaluation AI Agent Benchmarks Proactive Assistance

Ask PDF