Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
Figures & tables
Figure 1: Left: Model performance drops substantially from clear to vague instructions. Right: Key components of KnowU-Bench.
Benchmark or Dataset
Capability Dimensions
Evaluation Method
Task Target
Vague Instr.
Proactive Exec.
User Sim.
User Logs
User Model.
GUI Execution Benchmarks
AITW ( Rawles et al., 2023 )
✗
✗
✗
✗
✗
Action Matching
GUI Execution
AndroidControl ( Li et al., 2024 )
✗
✗
✗
✗
✗
Action Matching
GUI Execution
SPA-Bench ( Chen et al., 2025 )
✗
✗
✗
✗
✗
LLM as Judge
GUI Execution
AndroidWorld ( Rawles et al., 2024 )
✗
✗
✗
✗
✗
Rule-based
GUI Execution
Table 1: Comparison of KnowU-Bench with existing GUI benchmarks and datasets. ✓ fully incorporated; ✓ partially incorporated; ✗ not incorporated.
Figure 2: Overview of the KnowU-Bench framework. The benchmark couples a reproducible environment module, a GUI agent, an online user simulator grounded in user profiles and logs, and a hybrid evaluation pipeline combining rule based checks with LLM-as-a-judge scoring.
Model
Overall SR
General
Personalized
Proactive
easy
hard
easy
hard
easy
hard
SR
SR
SR
Score
SR
Score
SR
SR
Open-source models
UI-Venus-1.5-8B
26.0
72.2
25.0
18.6
0.48
7.0
0.40
34.4
31.3
Qwen3-VL-8B
21.9
72.2
4.2
7.0
0.27
7.0
0.25
46.9
21.9
GUI-Owl-1.5-8B
22.4
77.8
33.3
9.3
0.42
2.4
0.34
28.1
21.9
Table 2: Main results on KnowU-Bench under the noisy full-history memory setting ( Full Log, Noisy ), where each agent receives the complete user logs together with irrelevant history. Each task type is split into easy and hard subsets, and Overall SR is computed over all tasks. General and Proactive columns report Success Rate (SR), while Personalized additionally reports Average Score. Best and second-best denote the top two values in each column.
Figure 3: Visualization analyses. (a) Average score across four user roles: Developer (Dev.), Grandma (Grand.), Student (Stud.), and Researcher (Res.). (b) Personalized interaction metrics, including Efficiency (defined as 50/Avg. Steps ), Average Queries, and Interaction Efficiency (IE). (c) Proactive safety rates, including Act, Silent, and Stop.
Figure 4: Judge sensitivity against human ratings. Task-level scatter plots comparing two automatic evaluators against the mean score of four human experts on 26 shared trajectories. Each point denotes one task, the dashed diagonal indicates perfect agreement, and the inset reports mean absolute error. The hybrid judge (LLM-as-a-judge combined with rule-based scoring) exhibits tighter clustering around the diagonal and lower error than the pure rule-based variant, confirming stronger alignment with human judgment.
Model
Full Log
RAG Log
Clean
Noisy
Clean
Noisy
MAI-UI-8B
11.1
13.6
12.3
9.3
Qwen3-VL-8B
13.6
17.2
20.4
19.8
UI-Venus-1.5-8B
15.6
20.3
13.7
19.6
Table 3: Overall success rate under four memory settings, computed over personalized and proactive tasks only.
Figure 5: Failure mode breakdown. (a) Personalized failures are categorized into Clarify (insufficient clarification), Partial (partial preference satisfaction), Preference (preference misidentification), and GUI (GUI navigation failure). Most failures come from Clarify and Partial. (b) Proactive failures are categorized into Intervention (unwarranted intervention), Passive (false passivity), GUI (GUI navigation failure), and Rejection (post-rejection violation).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Additional view of the KnowU-Bench pipeline, showing task initialization, agent interaction, user simulation, and hybrid evaluation.
Action
Parameters
Description
click
x, y
Tap at the specified coordinates
double_tap
x, y
Double-tap at the specified coordinates
long_press
x, y
Long-press at the specified coordinates
drag
start_x, start_y, end_x, end_y
Drag from start to end coordinates
input_text
text
Type text into the focused field
scroll
direction
Scroll in the specified direction (up/down/left/right)
Appendix
Table 4: Action Space
App
Description
Comparable Commercial App
#Tasks
jingdian
E-commerce shopping platform
JD.com
35
Taodian
E-commerce shopping platform
Taobao
35
Messages
SMS and chat messaging
-
26
Mattermost
Team collaboration and messaging
Slack
25
Settings
System configuration
-
20
Calendar
Manage events and schedules
Google Calendar
18
Appendix
Table 5: App coverage of KnowU-Bench. #Tasks counts app level participations rather than unique episodes; each cross app task is counted for every involved app.
Field
Type
Function
identity
dict
Basic identity attributes such as name, age, occupation, employer, and optional contact or authentication metadata.
locations
dict
Task relevant physical places such as home and work, optionally with addresses, coordinates, labels, and delivery instructions.
digital_context
dict
The user’s digital environment, including device usage, system language, time zone, theme, and security preferences.
habits
dict
Recurrent behavior patterns encoded as trigger and action rules, functioning as a library of routine policies.
preferences
dict
Stable personal preferences such as food choices, shopping platforms, travel options, app choices, and communication style.
decision_criteria
dict
High level priorities, tradeoffs, and pain points used to resolve conflicts between competing actions or options.
Appendix
Table 6: Top level schema of KnowU-Bench user profiles.
Field
Type
Function
time
str
Event timestamp, typically represented in ISO 8601 format.
location
str
Free form location description indicating where the behavior took place.
action
str
Natural language description of the user behavior, which serves as the main semantic carrier for downstream reasoning.
label
str
Record label used to distinguish preference relevant or routine relevant signal from injected noise.
category
str
Behavior category indicating the thematic source of the record, such as commute, food preference, or morning reading routine.
Appendix
Table 7: Schema of KnowU-Bench user log entries.
Figure 7: General task success. The agent opens Contacts, selects Son (Qiang) , and places the call.
Figure 8: Instruction: “Help me post a status on Mastodon about finally beating a game boss that has troubled me for three days.”
Figure 9: A representative proactive success case. The agent notices a suspicious SMS notification, opens the message thread, selects the risky conversation, and proactively blocks and reports the sender as spam.
Figure 10: Instruction: “Post about completing a zero downtime production K8s rolling upgrade.”
Figure 11: Instruction: “Please remove from my shopping cart the clothes that I do not like.”
Figure 12: Instruction: “Please help me remove from my shopping cart the clothes that I think are too expensive.”
Figure 13: Instruction: “Help me buy a case of my favorite cola and send it to my work location.”
Figure 15: Unwarranted intervention. The agent wrongly opens Taodian and starts a shopping flow without asking for permission.
Figure 16: A representative post rejection violation case in ContactSaverTask under the developer role. After seeing a plausible contact update message (“Hi, this is Bob, my new number”), the agent asks whether it should act, receives an explicit rejection, then overrides both the role prior and the user response, labels the sender as spam, and blocks the number.
Figure 17: A representative proactive GUI navigation failure in GalleryCleanupTask . The agent enters Gallery and reaches the screenshots view, but the trajectory is derailed by preview and pop up pages, so the target screenshots are not deleted.