An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified. To act correctly for the particular user, an agent must infer some missing information from the user's activity history and ask for other information that only the user can provide, without making unsupported assumptions or burdening the user with unnecessary questions. We present iPrefBench, a benchmark for evaluating the infer and ask capabilities of personal mobile-use agents in an executable iOS environment with an interactive simulated user. Grounded in a coherent persona with 3,051 past activity events, iPrefBench comprises 130 underspecified user tasks across 26 apps. Each omitted user-dependent requirement is, by construction, either history-inferable or ask-only. iPrefBench evaluates personalized mobile agents on both whether agents complete tasks correctly for the user (i.e., personalized task completion) and whether they seek user input when necessary (i.e., user-agent interaction). Evaluation of 7 frontier agents reveals a substantial gap between execution and personalization. Agents achieve high action completion scores (up to 91.6%) but low full task success (up to 47.7%) because personalization lags behind. They especially struggle with ask-only information, failing to request enough necessary information. Our findings highlight a challenge for personalized agents in knowing what to infer and when to ask. We hope that iPrefBench can serve as a valuable testbed for building agents that know their user and work with them.
Figures & tables
Figure 1: iPrefBench evaluates personalized mobile-use agents on underspecified user tasks that require both inferring user-specific information from grounded activity history and asking the user for information unavailable from history.
Agent
Personalized Task Completion
User-Agent Interaction
#Steps
SR
Action
Personalization
Recall
Precision
#Ask
Overall
Infer
Ask
Claude Opus 5
47.7
91.6
69.4
76.3
51.3
44.2
65.8
0.35
22.5
Claude Sonnet 5
39.2
83.1
64.8
74.2
41.5
25.7
60.4
0.24
22.0
Gemini 3.1 Pro
36.2
78.6
60.9
69.2
40.1
23.9
77.1
0.20
22.6
Gemini 3.7 Flash
30.0
79.6
55.1
67.5
21.1
10.6
70.6
0.08
25.6
Table 1: Main results on iPrefBench. Personalized task completion is measured by success rate (SR), action completion and personalization score. We further divide the personalization score by information resolution source (i.e., inferable and ask-only). For user-agent interaction, we report Recall, Precision, and the average number of ask_user calls per task (#Ask). #Steps denotes the average number of actions per task. All scores are percentages except #Ask and #Steps.
Figure 2: Performance on the three perturbation subsets of iPrefBench. Left : action completion and personalization differences, computed for each agent as its subset score minus its overall score on the full benchmark. Bars are means across agents and markers are individual agents. Right : average number of ask_user actions per task.
Figure 3: Distribution of personalization failures along two dimensions: whether the unresolved information is ask-only or inferable, and whether the agent recognizes the corresponding decision. A trajectory may contain multiple errors.
Agent
Clarification
Personalized Task Completion
User-Agent Interaction
#Steps
SR
Action
Personalization
Recall
Precision
#Ask
Overall
Infer
Ask
Claude Sonnet 5
on-demand
39.2
83.1
64.8
74.2
41.5
25.7
60.4
0.24
22.0
+ pre-execution
53.8
82.0
77.8
73.9
82.0
85.0
42.5
1.37
24.5
Qwen3.8-27B
on-demand
26.2
77.5
54.4
62.1
29.4
21.2
70.6
0.21
20.7
+ pre-execution
35.4
76.6
67.8
66.0
65.0
69.0
60.9
0.84
21.5
Table 2: Effect of an explicit pre-execution clarification phase on Claude Sonnet 5 and Qwen3.8-27B. All scores are percentages except #Ask and #Steps.
Figure 4: Effect of different amounts of user history exposed in the agent’s context. (a) Personalization score split by information source. (b) User-agent interaction measured by recall and precision.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Environment
Inferable Info.
Ask-only Info.
Necessity-Aware
Interaction Eval.
General computer-use and GUI agent benchmarks
Mind2Web ( Deng et al., 2023 )
Web
✗
✗
✗
VisualWebArena ( Koh et al., 2024 )
Web
✗
✗
✗
OSWorld ( Xie et al., 2024 )
Desktop
✗
✗
✗
AndroidWorld ( Rawles et al., 2025 )
Mobile
✗
✗
✗
Appendix
Table A.1: Comparison of iPrefBench with existing benchmarks. Inferable Info. : tasks omit information that the agent can recover from the user’s behavioral history ( ∗ : user data exists in the environment but is not exposed to the agent as context). Ask-only Info. : tasks omit information that can only be obtained by asking the user. Necessity-Aware Interaction Eval. : the user-agent interaction is evaluated against what genuinely needs to be asked.
Category
App
Description
Food & dining
DineSpot
Restaurant search and table reservations
FreshCart
Grocery shopping and delivery
QuickBite
Food ordering and delivery from restaurants
TasteRank
Restaurant reviews and ratings
Work & productivity
LockedIn
Professional networking profile, connections, and job postings
Notes
Personal notes and lists
Appendix
Table B.1: The 26 mock applications in the iPrefBench environment, grouped by category.
Action
Parameters
Description
tap_xy
x , y : integers in [0,1000]
Tap a specified screen position using normalized coordinates.
type
text : string
Enter text into the currently focused input field. The agent must focus the field before typing.
swipe
direction : up, down, left, or right; optional x , y
Swipe in the specified direction, optionally from a given starting position.
home
None
Return to the device’s home screen.
wait
duration : number (seconds; default: 2.0)
Pause execution to allow content loading, animations, or other interface updates.
stop
answer : string
Terminate the task and return the agent’s final response.
Appendix
Table B.2 : Action space supported by iPrefBench. The first six actions provide standard mobile GUI interaction and task control, while ask_user is introduced to enable interactive clarification with the simulated user.
Dimension
Representative Information
Identity
Name, occupation, education, and skills.
Digital
Devices, system settings, and default applications.
Home and Household
Home and work locations, roommate, and household information.
People
Family, friends, colleagues, and social relationships.
Finances
Accounts, payment methods, recurring bills, and spending habits.
Health and Fitness
Nutrition goals, exercise habits, and fitness preferences.
Appendix
Table C.1 : The 13 dimensions of the structured user profile in iPrefBench, with representative information covered by each dimension.
Statistic
Value
User persona
Activity events
3,051
Mock apps with seeded user data
26
Profile dimensions / entries
13 / 92
Tasks
Tasks
130
Appendix
Table D.1: Statistics of iPrefBench.
Figure D.1 : Distribution of apps used by iPrefBench tasks. Bars count the tasks that require the app, grouped by app category and sorted within each category. Some tasks span multiple apps.
Model
Total #Ask
Overall Quality
Specificity
Answerability
Option Provision
Claude Opus 5
46
98.55
100.00
100.00
93.75
Claude Sonnet 5
31
100.00
100.00
100.00
100.00
Gemini 3.1 Pro
26
100.00
100.00
100.00
100.00
Gemini 3.7 Flash
11
93.94
100.00
100.00
71.43
Kimi K3
20
100.00
100.00
100.00
100.00
Qwen3.8-27B
27
100.00
100.00
100.00
100.00
Appendix
Table E.1 : Message quality evaluation, which is the message-level average over the three criteria. Each criterion is scored only on the messages where it applies (Option Provision is only applicable to questions with a small closed set of answers).
Figure F.1 : An example of Silent Default . When asked to book a round trip for “a few days coming up” without specific dates, the agent directly uses the app’s default departure and return dates as if they were the user’s, never treating travel dates as an open detail that needs to be resolved.
Figure F.2 : An example of Missed Inference . While the user sets a seat preference on every restaurant booking, and selects Bar Seating for an izakaya, chef’s-counter or omakase venue, the agent never notices this user preference, so seating is absent from its reasoning entirely. When booking at an izakaya, the agent leaves the reservation with “No Preference”.
Figure F.3 : An example of Silent Guess . The user instruction explicitly mentions that the ride is “with friends” without the specific group size, and only the user knows the actual group size (i.e., 5 ) in the current situation. However, the agent directly assumes “with friends” as a group of at most four and books a ride that seats one person too few.
Figure F.4 : An example of Wrong Inference . The agent grounds “my usual morning workout” in a real pattern in H , but in the pattern the user has already moved away from.
Figure F.5 : An example of keeping on trying when the preferred option is unavailable. The agent keeps attempting with different search keywords until the task hits the step limit with an empty cart.
Figure F.6 : An example of substituting silently when the preferred option is unavailable. The agent directly places the order with a different color and fails to disclose the substitution in its final answer to the user.
Figure F.7 : An example of directly omitting the unavailable item. The agent skips the unavailable item without asking the user and completes the task without any potential substitution of the unavailable item.
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existing mobile agent benchmarks lack this kind of personalization. We introduce iOSWorld, the first interactive native iOS simulator benchmark built around a persistent user identity spanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-source computer-use models in both vision-only and privileged vision+XML settings. The best configuration reaches 52% overall but only 37% on multi-app tasks. Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code.
Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom +3
Smartphone GUI agents execute tasks by operating directly on app interfaces, offering a path to broad capability without deep system integration. However, real-world smartphone use is highly personalized: users adopt diverse workflows and preferences, challenging agents to deliver customized assistance rather than generic solutions. Existing GUI agent benchmarks cannot adequately capture this personalization dimension due to sparse user-specific data and the lack of fine-grained evaluation metrics. To address this gap, we present PSPA-Bench, the benchmark dedicated to evaluating personalization in smartphone GUI agents. PSPA-Bench comprises over 12,855 personalized instructions aligned with real-world user behaviors across 10 representative daily-use scenarios and 22 mobile apps, and introduces a structure-aware process evaluation method that measures agents' personalized capabilities at a fine-grained level. Through PSPA-Bench, we benchmark 11 state-of-the-art GUI agents. Results reveal that current methods perform poorly under personalized settings, with even the strongest agent achieving limited success. Our analysis further highlights three directions for advancing personalized GUI agents: (1) reasoning-oriented models consistently outperform general LLMs, (2) perception remains a simple yet critical capability, and (3) reflection and long-term memory mechanisms are key to improving adaptation. Together, these findings establish PSPA-Bench as a foundation for systematic study and future progress in personalized GUI agents.
Hongyi Nie, Xunyuan Liu, Yudong Bai +4
Northwestern Polytechnical University · Tsinghua University · Peking University +1