iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

Yuting Ning; Ruijia Cheng; Sushobhan Ghosh; Peitong Duan; HungShi Lin; Abhishek Sundararajan; Shreyash Pandey; Huan Sun

cs.CLOct 26, 2026

iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

Authors: Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun,

Organizations: The Ohio State University · Apple

Abstract

An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified. To act correctly for the particular user, an agent must infer some missing information from the user's activity history and ask for other information that only the user can provide, without making unsupported assumptions or burdening the user with unnecessary questions. We present iPrefBench, a benchmark for evaluating the infer and ask capabilities of personal mobile-use agents in an executable iOS environment with an interactive simulated user. Grounded in a coherent persona with 3,051 past activity events, iPrefBench comprises 130 underspecified user tasks across 26 apps. Each omitted user-dependent requirement is, by construction, either history-inferable or ask-only. iPrefBench evaluates personalized mobile agents on both whether agents complete tasks correctly for the user (i.e., personalized task completion) and whether they seek user input when necessary (i.e., user-agent interaction). Evaluation of 7 frontier agents reveals a substantial gap between execution and personalization. Agents achieve high action completion scores (up to 91.6%) but low full task success (up to 47.7%) because personalization lags behind. They especially struggle with ask-only information, failing to request enough necessary information. Our findings highlight a challenge for personalized agents in knowing what to infer and when to ask. We hope that iPrefBench can serve as a valuable testbed for building agents that know their user and work with them.

Figures & tables

Figure 1: iPrefBench evaluates personalized mobile-use agents on underspecified user tasks that require both inferring user-specific information from grounded activity history and asking the user for information unavailable from history.

Agent	Personalized Task Completion					User-Agent Interaction			#Steps
	SR	Action	Personalization			Recall	Precision	#Ask
	SR	Action	Overall	Infer	Ask	Recall	Precision	#Ask
Claude Opus 5	$47.7$	$91.6$	$69.4$	$76.3$	$51.3$	$44.2$	$65.8$	$0.35$	$22.5$
Claude Sonnet 5	$39.2$	$83.1$	$64.8$	$74.2$	$41.5$	$25.7$	$60.4$	$0.24$	$22.0$
Gemini 3.1 Pro	$36.2$	$78.6$	$60.9$	$69.2$	$40.1$	$23.9$	$77.1$	$0.20$	$22.6$
Gemini 3.7 Flash	$30.0$	$79.6$	$55.1$	$67.5$	$21.1$	$10.6$	$70.6$	$0.08$	$25.6$

Table 1: Main results on iPrefBench. Personalized task completion is measured by success rate (SR), action completion and personalization score. We further divide the personalization score by information resolution source (i.e., inferable and ask-only). For user-agent interaction, we report Recall, Precision, and the average number of ask_user calls per task (#Ask). #Steps denotes the average number of actions per task. All scores are percentages except #Ask and #Steps.

Figure 2: Performance on the three perturbation subsets of iPrefBench. Left : action completion and personalization differences, computed for each agent as its subset score minus its overall score on the full benchmark. Bars are means across agents and markers are individual agents. Right : average number of ask_user actions per task.

Figure 3: Distribution of personalization failures along two dimensions: whether the unresolved information is ask-only or inferable, and whether the agent recognizes the corresponding decision. A trajectory may contain multiple errors.

Agent	Clarification	Personalized Task Completion					User-Agent Interaction			#Steps
		SR	Action	Personalization			Recall	Precision	#Ask
		SR	Action	Overall	Infer	Ask	Recall	Precision	#Ask
Claude Sonnet 5	on-demand	$39.2$	$83.1$	$64.8$	$74.2$	$41.5$	$25.7$	$60.4$	$0.24$	$22.0$
Claude Sonnet 5	+ pre-execution	$53.8$	$82.0$	$77.8$	$73.9$	$82.0$	$85.0$	$42.5$	$1.37$	$24.5$
Qwen3.8-27B	on-demand	$26.2$	$77.5$	$54.4$	$62.1$	$29.4$	$21.2$	$70.6$	$0.21$	$20.7$
Qwen3.8-27B	+ pre-execution	$35.4$	$76.6$	$67.8$	$66.0$	$65.0$	$69.0$	$60.9$	$0.84$	$21.5$

Table 2: Effect of an explicit pre-execution clarification phase on Claude Sonnet 5 and Qwen3.8-27B. All scores are percentages except #Ask and #Steps.

Figure 4: Effect of different amounts of user history exposed in the agent’s context. (a) Personalization score split by information source. (b) User-agent interaction measured by recall and precision.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Environment	Inferable Info.	Ask-only Info.	Necessity-Aware
Benchmark	Environment	Inferable Info.	Ask-only Info.	Interaction Eval.
General computer-use and GUI agent benchmarks
Mind2Web ( Deng et al., 2023 )	Web	✗	✗	✗
VisualWebArena ( Koh et al., 2024 )	Web	✗	✗	✗
OSWorld ( Xie et al., 2024 )	Desktop	✗	✗	✗
AndroidWorld ( Rawles et al., 2025 )	Mobile	✗	✗	✗

Appendix

Table A.1: Comparison of iPrefBench with existing benchmarks. Inferable Info. : tasks omit information that the agent can recover from the user’s behavioral history (

\ast

: user data exists in the environment but is not exposed to the agent as context). Ask-only Info. : tasks omit information that can only be obtained by asking the user. Necessity-Aware Interaction Eval. : the user-agent interaction is evaluated against what genuinely needs to be asked.

Category	App	Description
Food & dining	DineSpot	Restaurant search and table reservations
	FreshCart	Grocery shopping and delivery
	QuickBite	Food ordering and delivery from restaurants
	TasteRank	Restaurant reviews and ratings
Work & productivity	LockedIn	Professional networking profile, connections, and job postings
	Notes	Personal notes and lists

Appendix

Table B.1: The

26

mock applications in the iPrefBench environment, grouped by category.

Action	Parameters	Description
tap_xy	x , y : integers in $[0,1000]$	Tap a specified screen position using normalized coordinates.
type	text : string	Enter text into the currently focused input field. The agent must focus the field before typing.
swipe	direction : up, down, left, or right; optional x , y	Swipe in the specified direction, optionally from a given starting position.
home	None	Return to the device’s home screen.
wait	duration : number (seconds; default: 2.0)	Pause execution to allow content loading, animations, or other interface updates.
stop	answer : string	Terminate the task and return the agent’s final response.

Appendix

Table B.2 : Action space supported by iPrefBench. The first six actions provide standard mobile GUI interaction and task control, while ask_user is introduced to enable interactive clarification with the simulated user.

Dimension	Representative Information
Identity	Name, occupation, education, and skills.
Digital	Devices, system settings, and default applications.
Home and Household	Home and work locations, roommate, and household information.
People	Family, friends, colleagues, and social relationships.
Finances	Accounts, payment methods, recurring bills, and spending habits.
Health and Fitness	Nutrition goals, exercise habits, and fitness preferences.

Appendix

Table C.1 : The 13 dimensions of the structured user profile in iPrefBench, with representative information covered by each dimension.

Statistic	Value
User persona
Activity events	$3,051$
Mock apps with seeded user data	26
Profile dimensions / entries	$13$ / $92$
Tasks
Tasks	$130$

Appendix

Table D.1: Statistics of iPrefBench.

Figure D.1 : Distribution of apps used by iPrefBench tasks. Bars count the tasks that require the app, grouped by app category and sorted within each category. Some tasks span multiple apps.

Model	Total #Ask	Overall Quality	Specificity	Answerability	Option Provision
Claude Opus 5	$46$	$98.55$	$100.00$	$100.00$	$93.75$
Claude Sonnet 5	$31$	$100.00$	$100.00$	$100.00$	$100.00$
Gemini 3.1 Pro	$26$	$100.00$	$100.00$	$100.00$	$100.00$
Gemini 3.7 Flash	$11$	$93.94$	$100.00$	$100.00$	$71.43$
Kimi K3	$20$	$100.00$	$100.00$	$100.00$	$100.00$
Qwen3.8-27B	$27$	$100.00$	$100.00$	$100.00$	$100.00$

Appendix

Table E.1 : Message quality evaluation, which is the message-level average over the three criteria. Each criterion is scored only on the messages where it applies (Option Provision is only applicable to questions with a small closed set of answers).

Figure F.1 : An example of Silent Default . When asked to book a round trip for “a few days coming up” without specific dates, the agent directly uses the app’s default departure and return dates as if they were the user’s, never treating travel dates as an open detail that needs to be resolved.

Figure F.2 : An example of Missed Inference . While the user sets a seat preference on every restaurant booking, and selects Bar Seating for an izakaya, chef’s-counter or omakase venue, the agent never notices this user preference, so seating is absent from its reasoning entirely. When booking at an izakaya, the agent leaves the reservation with “No Preference”.

Figure F.3 : An example of Silent Guess . The user instruction explicitly mentions that the ride is “with friends” without the specific group size, and only the user knows the actual group size (i.e., $5$ ) in the current situation. However, the agent directly assumes “with friends” as a group of at most four and books a ride that seats one person too few.

Figure F.4 : An example of Wrong Inference . The agent grounds “my usual morning workout” in a real pattern in $\mathcal{H}$ , but in the pattern the user has already moved away from.

Figure F.5 : An example of keeping on trying when the preferred option is unavailable. The agent keeps attempting with different search keywords until the task hits the step limit with an empty cart.

Figure F.6 : An example of substituting silently when the preferred option is unavailable. The agent directly places the order with a different color and fails to disclose the substitution in its final answer to the user.

Figure F.7 : An example of directly omitting the unavailable item. The agent skips the unavailable item without asking the user and completes the task without any potential substitution of the unavailable item.

Personal AI Agents

Personal AI agents use persistent user context and digital tools to carry out personalized, proactive tasks across apps, devices, and services.

See all 64 papers

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Apr 9, 2026Tongbo Chen, Zhengxi Lu, Zhan Xu +13Mobile Agent Benchmarks AI Agent Benchmarks
Zhejiang University · Apple · Tencent
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Jun 8, 2026Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom +3Personal AI Agents Mobile GUI Agents
Carnegie Mellon University
PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent
Mar 31, 2026Hongyi Nie, Xunyuan Liu, Yudong Bai +4GUI Agents Mobile GUI Agents
Northwestern Polytechnical University · Tsinghua University · Peking University +1

Apr 9, 2026J/K move · Enter open · S save

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao+12

Zhejiang University · Apple · Tencent

Mobile Agent Benchmarks AI Agent Benchmarks Proactive Assistance Mobile GUI Agents Preference Elicitation User Preference Modeling

Open paper AskPDF arXiv

Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.

Abstract

Personal AI Agents

Explore similar work

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent