PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent

Hongyi Nie; Xunyuan Liu; Yudong Bai; Yaqing Wang; Yang Liu; Quanming Yao; Zhen Wang

cs.AIMar 31, 2026

PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent

Authors: Hongyi Nie, Xunyuan Liu, Yudong Bai, Yaqing Wang, Yang Liu, Quanming Yao, Zhen Wang

Organizations: Northwestern Polytechnical University · Tsinghua University · Peking University · Beijing Institute of Mathematical Sciences and Applications

Abstract

Smartphone GUI agents execute tasks by operating directly on app interfaces, offering a path to broad capability without deep system integration. However, real-world smartphone use is highly personalized: users adopt diverse workflows and preferences, challenging agents to deliver customized assistance rather than generic solutions. Existing GUI agent benchmarks cannot adequately capture this personalization dimension due to sparse user-specific data and the lack of fine-grained evaluation metrics. To address this gap, we present PSPA-Bench, the benchmark dedicated to evaluating personalization in smartphone GUI agents. PSPA-Bench comprises over 12,855 personalized instructions aligned with real-world user behaviors across 10 representative daily-use scenarios and 22 mobile apps, and introduces a structure-aware process evaluation method that measures agents' personalized capabilities at a fine-grained level. Through PSPA-Bench, we benchmark 11 state-of-the-art GUI agents. Results reveal that current methods perform poorly under personalized settings, with even the strongest agent achieving limited success. Our analysis further highlights three directions for advancing personalized GUI agents: (1) reasoning-oriented models consistently outperform general LLMs, (2) perception remains a simple yet critical capability, and (3) reflection and long-term memory mechanisms are key to improving adaptation. Together, these findings establish PSPA-Bench as a foundation for systematic study and future progress in personalized GUI agents.

Figures & tables

Figure 1: GUI agents are shifting from general-purpose, one-time task execution to personalized, long-term service. (i) Given the same instruction, an agent should adapt its execution path to user preferences; (ii) For the same user, the agent uses historical execution experience to provide long-term support; experience up to time $t$ informs a refined execution at $t+1$ for similar instructions.

Figure 2: Benchmark framework of PSPA-Bench . (a) Task decomposition graph : the task is decomposed into a directed acyclic graph of unit instructions, where fixed nodes denote universal steps and flexible nodes denote user-specific requirements. (b) Template-driven personalized instruction generation : the TDG is used to construct task templates, which are then instantiated with user preferences to generate personalized instructions. (c) Fine-grained personalized evaluation : The trace-graph alignment is used to compute APR (A-Progress Ratio) and PPR (P-Progress Ratio) for fine-grained evaluation of both immediate and long-term performance.

Figure 3: Evaluation metrics taxonomy of PSPA-Bench .

Method		Perception		Reasoning		Memory		Evolution		Collaboration
Type	Name	Text	Vision	Planning	Reflection	Temporary	Persistent	Rules	Toolkits	Single	Multiple
LLM	Base		✓			✓				✓
LLM	+A11y	✓				✓				✓
	+A11y+SoM	✓	✓			✓				✓
	+A11y+Thinking	✓		✓		✓				✓
General Agent Framework	ReAct	✓	✓	✓		✓				✓

Table 1: Comparison of evaluated methods in PSPA-Bench . The table presents baseline LLMs, general agent frameworks, and GUI agent frameworks, assessed across five key dimensions: Perception, Reasoning, Memory, Evolution, and Collaboration.

Method	Setting: Clarity fixed at middle, varying complexity
	Complexity: low				Complexity: middle				Complexity: high
	APR $\uparrow$	PPR $\uparrow$	CT $\downarrow$	CPT $\downarrow$	APR $\uparrow$	PPR $\uparrow$	CT $\downarrow$	CPT $\downarrow$	APR $\uparrow$	PPR $\uparrow$	CT $\downarrow$	CPT $\downarrow$
LLM-Base	0.002	0	236.7	0.612	0.001	0	256.6	0.501	0.002	0	293.4	0.816
+A11y	0.625	0.586	176.7	0.056	0.431	0.449	189.0	0.038	0.364	0.356	229.3	0.062
+A11y+SoM	0.643	0.596	243.6	0.671	0.463	0.461	254.3	0.525	0.373	0.372	293.3	0.930
+A11y+Thinking	0.674	0.624	197.7	0.673	0.466	0.464	233.0	0.726	0.382	0.378	274.8	0.884

Table 2: Mean values of evaluation metrics (APR, PPR, CT, CPT) across five scenarios under high clarity conditions, evaluated at three levels of task complexity (low, middle, high). The results reflect each method’s capability in achieving the immediate objective of personalized GUI tasks, i.e., accurately and efficiently executing the current instruction

Method	Setting (i): Clarity fixed at high, varying complexity
	Complexity: low				Complexity: middle				Complexity: high
	$\Delta$ APR $\uparrow$	$\Delta$ PPR $\uparrow$	$\Delta$ CT $\downarrow$	$\Delta$ CPT $\downarrow$	$\Delta$ APR $\uparrow$	$\Delta$ PPR $\uparrow$	$\Delta$ CT $\downarrow$	$\Delta$ CPT $\downarrow$	$\Delta$ APR $\uparrow$	$\Delta$ PPR $\uparrow$	$\Delta$ CT $\downarrow$	$\Delta$ CPT $\downarrow$
LLM-Base	0	0	9.28	0.20	0	0	-17.68	0.73	0	0	-19.18	0.94
+A11y	0.66	-0.58	-12.73	-0.63	-0.39	0.05	-2.72	0.42	0.22	-0.72	-8.31	-0.27
+A11y+SoM	-0.09	0.57	12.01	-0.03	0.18	-0.91	4.30	0.66	-0.87	0.90	-18.63	0.62
+A11y+Thinking	-0.39	-0.80	7.37	-0.12	-0.76	-0.01	-18.62	0.82	-0.48	0.33	-7.53	0.04

Table 3: Mean values of evaluation metrics (

\Delta

APR,

\Delta

PPR,

\Delta

CT,

\Delta

CPT) across five scenarios, reported under two settings: (i) high clarity with varying task complexity (low, middle, high), and (ii) middle complexity with varying task clarity (low, middle, high). Before evaluation, each method performs personalized tasks with middle complexity and middle clarity to accumulate initial experience. The results reflect each method’s capability in achieving the long-term objective of personalized GUI tasks, i.e., adapting to user preferences over long-term use. (Note: For presentation convenience,

\Delta

APR,

\Delta

PPR and

\Delta

CPT values are

\times 100

Figure 4: Radar plots showing the comparative ranks of different methods under two objectives. Left: Immediate objective, evaluated with APR, PPR, CT, and CPT. Right: Long-term objective, evaluated with $\Delta$ APR, $\Delta$ PPR, $\Delta$ CT, and $\Delta$ CPT. Each axis indicates the rank of a method on the corresponding metric, with lower values representing better performance.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Dataset & Benchmark	Environment with physical phones	Third-part apps	User profiles	User preferences	Task instructions	Difficulty dimensions	Fine-grained progress evaluation	Incremental evaluation
General GUI benchmark	AndroidArena	✗	✗	✗	✗	221	✗	✓	✗
	LlamaTouch	✗	✓	✗	✗	495	Step length	✗	✗
	MobileAgentBench	✗	✗	✗	✗	100	Step length	✗	✗
	AndroidWorld	✗	✗	✗	✗	$\infty$	Step length	✗	✗
	SPA-bench	✓	✓	✗	✗	340	Step length	✗	✗
Personalized GUI benchmark	LearnGUI	✗	✗	✗	✗	2353	✗	✗	✗

Appendix

Table 4: Comparison of General and Personalized GUI Benchmarks

Figure 5: The task decomposition graphs of 4 personalized scenarios: shopping, dining, navigation, and travel. The nodes with green color refer to fixed type, while nodes with blue color refers to personalized type.

Figure 6: (a) Preference Weights Over Time : The blue curve represents the long-term preference weight, which gradually increases with time, reflecting the accumulation of stable needs. The red curve depicts the short-term preference weight, which oscillates periodically, reflecting dynamic, context-dependent needs. (b) Stacked Sampling Probabilities : The stacked area plot shows the normalized sampling probabilities over time. The blue region corresponds to the probability of sampling the long-term preference, while the orange region corresponds to the probability of sampling the short-term preference. The relative areas of these regions change over time, depending on the weights of each preference.

Comparison (Pairwise)	Win-rate Ours $\uparrow$	Win-rate Baseline	Tie (%)
Ours vs. Static	72.5%	24.1%	3.4%
Ours vs. Random	81.3%	16.2%	2.5%
Static vs. Random	59.6%	38.7%	1.7%

Appendix

Table 5: Pairwise win-rates of different methods in human evaluation.

Dimension	Immediate Objective	Long-term Objective
Performance	APR: $\tfrac{1}{N}\sum\tfrac{\text{completed unit inst.}}{\text{total unit inst.}}$	$\Delta$ APR: $\text{APR}_{T_{k}}-\text{APR}_{T_{1}}$
Performance	PPR: $\tfrac{1}{N}\sum\tfrac{\text{completed flexible unit inst.}}{\text{total flexible unit inst.}}$	$\Delta$ PPR: $\text{PPR}_{T_{k}}-\text{PPR}_{T_{1}}$
Efficiency	CT: $\tfrac{1}{N}\sum t_{i}$	$\Delta$ CT: $\text{CT}_{T_{k}}-\text{CT}_{T_{1}}$
Efficiency	CPT: $\tfrac{1}{N}\sum\text{Cost}_{i}$	$\Delta$ CPT: $\text{CPT}_{T_{k}}-\text{CPT}_{T_{1}}$

Appendix

Table 6: Evaluation metrics for the Personalized GUI Agent Task. Short-term metrics measure absolute performance on individual tasks, while long-term metrics capture improvements across instruction sequences. N denotes the number of tasks.

Figure 7: Comparison of evaluation metrics across tasks sorted by human ratings. Each point represents one task. Human ratings (pink dots) are shown with an acceptable tolerance band (±0.05, gray area). A-process Ratio (APR, brown squares) closely follows human ratings with minimal deviation, demonstrating strong alignment ( $\rho$ =0.95, MSE=0.005). In contrast, Task Completion Ratio (TCR, green triangles) shows weaker consistency with human ratings ( $\rho$ =0.47, MSE=0.062), while Success Signal (blue crosses) provides only binary outcomes, lacking granularity.

Instruction Type	Acc.	Prec.	Rec.	F1	$\kappa$
Fixed Instructions	0.94	0.93	0.96	0.94	0.88
Flexible Instructions	0.91	0.90	0.93	0.91	0.83
Overall	0.93	0.92	0.95	0.93	0.86

Appendix

Table 7: Validation of LLM-based alignment against human annotations. We report Accuracy, Precision, Recall, F1-score, and Cohen’s

\kappa

for agreement with human ground truth.

Clarity level	Complexity level	Template Examples (English)
High clarity	Low	1. I want to search for {brand} {category} on {APP}, with prices between {price range} yuan. 2. Help me find a {brand} {category} on {APP}, with prices in the range of {price range} yuan. 3. Check {APP} for a {brand} {category}, keeping the price within {price range} yuan.
	Medium	1. I want to buy a {brand} {category} on {APP}, with prices between {price range} yuan, and stop at the product details page. 2. Help me find a {brand} {category} on {APP}, with prices in the range of {price range} yuan, and stay on the product details page. 3. Check {APP} for a {brand} {category}, with prices around {price range} yuan, and stay on the product details page.
	High	1. I want to buy a {brand} {category} on {APP}, with prices between {price range} yuan, add the item to the cart, and finally stay on the cart page. 2. Help me find a {brand} {category} on {APP}, with prices around {price range} yuan, add it to the cart, and finally stay on the cart page. 3. Check {APP} for a {brand} {category}, within {price range} yuan, add it to the cart, and finally stay on the cart page.
Medium clarity	Low	1. Search {brand} {category} on {APP}, price about {price range}. 2. Help me find {brand} {category} in {APP}, budget {price range}. 3. Check {APP} for {brand} {category}, price around {price range}.
	Medium	1. Want to buy {brand} {category} on {APP}, price {price range}, just open the details page. 2. {APP}, find {brand} {category}, around {price range}, go to the details page. 3. Help me search {brand} {category} on {APP}, price {price range}, open the details page.
	High	1. Buy {brand} {category} on {APP}, budget {price range}, add to cart and stay there. 2. {APP}, find {brand} {category}, price about {price range}, add to cart. 3. Help me add {brand} {category} ({price range}) in {APP} to the cart, and stay on the cart page.

Appendix

Table 8: Instruction templates of shopping scenario.

Clarity level	Complexity level	Template Examples (English)
High clarity	Low	1. I want to search for {singer} on {APP} and go to the singer’s homepage. 2. Help me search for {singer} on {APP} and enter the singer’s homepage. 3. Check {APP} for {singer} and click to open the singer’s homepage.
	Medium	1. I want to search for {singer} on {APP}, go to the singer’s homepage, and view the albums. 2. Help me find {singer} on {APP}, enter the singer’s homepage, and check the albums. 3. On {APP}, search for {singer}, open the singer’s homepage, then view the albums.
	High	1. I want to search for {singer} on {APP}, open the singer’s homepage, check the albums, and select a song to play. 2. Help me find {singer} on {APP}, enter the singer’s homepage, view the albums, and then play a song. 3. On {APP}, search for {singer}, open the homepage, view the albums, and finally play a song.
Medium clarity	Low	1. Search {singer} on {APP}, go to homepage. 2. Help me find {singer} in {APP}, check homepage. 3. {APP} search {singer}, open homepage.
	Medium	1. Want to find {singer} on {APP}, check homepage and albums. 2. {APP} search {singer}, enter homepage, view albums. 3. Help me search for {singer} on {APP}, view albums in homepage.
	High	1. Find {singer} on {APP}, view albums, select a song to play. 2. {APP} search {singer}, choose a song from album to listen. 3. Help me find {singer} on {APP}, select a song from album and play.

Appendix

Table 9: Instruction templates for music scenario.

Clarity level	Complexity level	Template Examples (English)
High clarity	Low	1. I want to search for {cuisine type} restaurants on {APP}, with an average spending between {price range} yuan. 2. Help me find a {cuisine type} restaurant on {APP}, with average spending in the range of {price range} yuan. 3. Check {APP} for {cuisine type} restaurants, priced between {price range} yuan.
	Medium	1. I want to search for {cuisine type} restaurants on {APP}, with an average spending between {price range} yuan, and stay on the restaurant details page. 2. Help me find a {cuisine type} restaurant on {APP}, with average spending in the range of {price range} yuan, and remain on the details page. 3. Check {APP} for {cuisine type} restaurants, priced between {price range} yuan, and stay on the details page.
	High	1. I want to search for {cuisine type} restaurants on {APP}, with an average spending between {price range} yuan, enter the details page, and view all recommended dishes. 2. Help me find a {cuisine type} restaurant on {APP}, with average spending in the range of {price range} yuan, go to the details page, and view all recommended dishes. 3. Check {APP} for {cuisine type} restaurants, priced between {price range} yuan, enter the details page, and view all recommended dishes.
Medium clarity	Low	1. Search {cuisine type} on {APP}, around {price range}. 2. Help me find {cuisine type} in {APP}, budget {price range}. 3. Check {APP} for {cuisine type}, price about {price range}.
	Medium	1. Want to find {cuisine type} on {APP}, price between {price range}, open details page. 2. {APP} search {cuisine type}, in range {price range}, open details. 3. Help me search {cuisine type} on {APP}, price {price range}, view details.
	High	1. Find {cuisine type} on {APP}, budget {price range}, check recommended dishes. 2. {APP} search {cuisine type}, price {price range}, view recommended dishes. 3. Help me find {cuisine type} ({price range}) on {APP}, view recommended dishes.

Appendix

Table 10: Instruction templates for dining scenario.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 172 papers

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Apr 9, 2026Tongbo Chen, Zhengxi Lu, Zhan Xu +13Mobile Agent Benchmarks AI Agent Benchmarks
Zhejiang University · Apple · Tencent
Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces
Oct 6, 2026Yeji Park, Jaeyun Shim, Taesik GongMobile GUI Agents AI Agent Reliability
UNIST
Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Apr 13, 2026Zhixin Lin, Jungang Li, Dongliang Xu +5Mobile GUI Agents Personalized Language Model Alignment
Shandong University · The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology +4

Apr 9, 2026J/K move · Enter open · S save

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao+12

Zhejiang University · Apple · Tencent

Mobile Agent Benchmarks AI Agent Benchmarks Proactive Assistance Mobile GUI Agents Preference Elicitation User Preference Modeling

Open paper AskPDF arXiv

Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.

Abstract

GUI Agents

Explore similar work

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces

Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization