Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
Figures & tables
Figure 1: Left: Model performance drops substantially from clear to vague instructions. Right: Key components of KnowU-Bench.
Benchmark or Dataset
Capability Dimensions
Evaluation Method
Task Target
Vague Instr.
Proactive Exec.
User Sim.
User Logs
User Model.
GUI Execution Benchmarks
AITW ( Rawles et al., 2023 )
✗
✗
✗
✗
✗
Action Matching
GUI Execution
AndroidControl ( Li et al., 2024 )
✗
✗
✗
✗
✗
Action Matching
GUI Execution
SPA-Bench ( Chen et al., 2025 )
✗
✗
✗
✗
✗
LLM as Judge
GUI Execution
AndroidWorld ( Rawles et al., 2024 )
✗
✗
✗
✗
✗
Rule-based
GUI Execution
Table 1: Comparison of KnowU-Bench with existing GUI benchmarks and datasets. ✓ fully incorporated; ✓ partially incorporated; ✗ not incorporated.
Figure 2: Overview of the KnowU-Bench framework. The benchmark couples a reproducible environment module, a GUI agent, an online user simulator grounded in user profiles and logs, and a hybrid evaluation pipeline combining rule based checks with LLM-as-a-judge scoring.
Model
Overall SR
General
Personalized
Proactive
easy
hard
easy
hard
easy
hard
SR
SR
SR
Score
SR
Score
SR
SR
Open-source models
UI-Venus-1.5-8B
26.0
72.2
25.0
18.6
0.48
7.0
0.40
34.4
31.3
Qwen3-VL-8B
21.9
72.2
4.2
7.0
0.27
7.0
0.25
46.9
21.9
GUI-Owl-1.5-8B
22.4
77.8
33.3
9.3
0.42
2.4
0.34
28.1
21.9
Table 2: Main results on KnowU-Bench under the noisy full-history memory setting ( Full Log, Noisy ), where each agent receives the complete user logs together with irrelevant history. Each task type is split into easy and hard subsets, and Overall SR is computed over all tasks. General and Proactive columns report Success Rate (SR), while Personalized additionally reports Average Score. Best and second-best denote the top two values in each column.
Figure 3: Visualization analyses. (a) Average score across four user roles: Developer (Dev.), Grandma (Grand.), Student (Stud.), and Researcher (Res.). (b) Personalized interaction metrics, including Efficiency (defined as 50/Avg. Steps ), Average Queries, and Interaction Efficiency (IE). (c) Proactive safety rates, including Act, Silent, and Stop.
Figure 4: Judge sensitivity against human ratings. Task-level scatter plots comparing two automatic evaluators against the mean score of four human experts on 26 shared trajectories. Each point denotes one task, the dashed diagonal indicates perfect agreement, and the inset reports mean absolute error. The hybrid judge (LLM-as-a-judge combined with rule-based scoring) exhibits tighter clustering around the diagonal and lower error than the pure rule-based variant, confirming stronger alignment with human judgment.
Model
Full Log
RAG Log
Clean
Noisy
Clean
Noisy
MAI-UI-8B
11.1
13.6
12.3
9.3
Qwen3-VL-8B
13.6
17.2
20.4
19.8
UI-Venus-1.5-8B
15.6
20.3
13.7
19.6
Table 3: Overall success rate under four memory settings, computed over personalized and proactive tasks only.
Figure 5: Failure mode breakdown. (a) Personalized failures are categorized into Clarify (insufficient clarification), Partial (partial preference satisfaction), Preference (preference misidentification), and GUI (GUI navigation failure). Most failures come from Clarify and Partial. (b) Proactive failures are categorized into Intervention (unwarranted intervention), Passive (false passivity), GUI (GUI navigation failure), and Rejection (post-rejection violation).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Additional view of the KnowU-Bench pipeline, showing task initialization, agent interaction, user simulation, and hybrid evaluation.
Action
Parameters
Description
click
x, y
Tap at the specified coordinates
double_tap
x, y
Double-tap at the specified coordinates
long_press
x, y
Long-press at the specified coordinates
drag
start_x, start_y, end_x, end_y
Drag from start to end coordinates
input_text
text
Type text into the focused field
scroll
direction
Scroll in the specified direction (up/down/left/right)
Appendix
Table 4: Action Space
App
Description
Comparable Commercial App
#Tasks
jingdian
E-commerce shopping platform
JD.com
35
Taodian
E-commerce shopping platform
Taobao
35
Messages
SMS and chat messaging
-
26
Mattermost
Team collaboration and messaging
Slack
25
Settings
System configuration
-
20
Calendar
Manage events and schedules
Google Calendar
18
Appendix
Table 5: App coverage of KnowU-Bench. #Tasks counts app level participations rather than unique episodes; each cross app task is counted for every involved app.
Field
Type
Function
identity
dict
Basic identity attributes such as name, age, occupation, employer, and optional contact or authentication metadata.
locations
dict
Task relevant physical places such as home and work, optionally with addresses, coordinates, labels, and delivery instructions.
digital_context
dict
The user’s digital environment, including device usage, system language, time zone, theme, and security preferences.
habits
dict
Recurrent behavior patterns encoded as trigger and action rules, functioning as a library of routine policies.
preferences
dict
Stable personal preferences such as food choices, shopping platforms, travel options, app choices, and communication style.
decision_criteria
dict
High level priorities, tradeoffs, and pain points used to resolve conflicts between competing actions or options.
Appendix
Table 6: Top level schema of KnowU-Bench user profiles.
Field
Type
Function
time
str
Event timestamp, typically represented in ISO 8601 format.
location
str
Free form location description indicating where the behavior took place.
action
str
Natural language description of the user behavior, which serves as the main semantic carrier for downstream reasoning.
label
str
Record label used to distinguish preference relevant or routine relevant signal from injected noise.
category
str
Behavior category indicating the thematic source of the record, such as commute, food preference, or morning reading routine.
Appendix
Table 7: Schema of KnowU-Bench user log entries.
Figure 7: General task success. The agent opens Contacts, selects Son (Qiang) , and places the call.
Figure 8: Instruction: “Help me post a status on Mastodon about finally beating a game boss that has troubled me for three days.”
Figure 9: A representative proactive success case. The agent notices a suspicious SMS notification, opens the message thread, selects the risky conversation, and proactively blocks and reports the sender as spam.
Figure 10: Instruction: “Post about completing a zero downtime production K8s rolling upgrade.”
Figure 11: Instruction: “Please remove from my shopping cart the clothes that I do not like.”
Figure 12: Instruction: “Please help me remove from my shopping cart the clothes that I think are too expensive.”
Figure 13: Instruction: “Help me buy a case of my favorite cola and send it to my work location.”
Figure 15: Unwarranted intervention. The agent wrongly opens Taodian and starts a shopping flow without asking for permission.
Figure 16: A representative post rejection violation case in ContactSaverTask under the developer role. After seeing a plausible contact update message (“Hi, this is Bob, my new number”), the agent asks whether it should act, receives an explicit rejection, then overrides both the role prior and the user response, labels the sender as spam, and blocks the number.
Figure 17: A representative proactive GUI navigation failure in GalleryCleanupTask . The agent enters Gallery and reaches the screenshots view, but the trajectory is derailed by preview and pop up pages, so the target screenshots are not deleted.
The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce π-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, π-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
Haoran Zhang, Luxin Xu, Zhilin Wang +11
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +7
Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.
Yuxin Chen, Yi Zhang, Zhengzhou Cai +11
1National University of Singapore · 2Meituan · University of Science and Technology of China +2
Smartphone GUI agents execute tasks by operating directly on app interfaces, offering a path to broad capability without deep system integration. However, real-world smartphone use is highly personalized: users adopt diverse workflows and preferences, challenging agents to deliver customized assistance rather than generic solutions. Existing GUI agent benchmarks cannot adequately capture this personalization dimension due to sparse user-specific data and the lack of fine-grained evaluation metrics. To address this gap, we present PSPA-Bench, the benchmark dedicated to evaluating personalization in smartphone GUI agents. PSPA-Bench comprises over 12,855 personalized instructions aligned with real-world user behaviors across 10 representative daily-use scenarios and 22 mobile apps, and introduces a structure-aware process evaluation method that measures agents' personalized capabilities at a fine-grained level. Through PSPA-Bench, we benchmark 11 state-of-the-art GUI agents. Results reveal that current methods perform poorly under personalized settings, with even the strongest agent achieving limited success. Our analysis further highlights three directions for advancing personalized GUI agents: (1) reasoning-oriented models consistently outperform general LLMs, (2) perception remains a simple yet critical capability, and (3) reflection and long-term memory mechanisms are key to improving adaptation. Together, these findings establish PSPA-Bench as a foundation for systematic study and future progress in personalized GUI agents.
Hongyi Nie, Xunyuan Liu, Yudong Bai +4
Northwestern Polytechnical University · Tsinghua University · Peking University +1