Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

Fuyao Zhang; Xuan Wang; Zherui Li; Jiaming Zhang; Longtao Huang; Wei Yang Bryan Lim

cs.CVSep 27, 2026

Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

Authors: Fuyao Zhang, Xuan Wang, Zherui Li, Jiaming Zhang, Longtao Huang, Wei Yang Bryan Lim

Organizations: NTU & Alibaba

Abstract

Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy's context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining 90% of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that relevance does not imply applicability}: relevance is determined at the task level, whereas applicability depends on the situation at decision time. We therefore recast personalization as experience activation and introduce ExpActivator, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone's latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28% on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately 2.3 $\times$ the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.

Figures & tables

Figure 1: From retrieval to activation. Top: personalized execution. Bottom: proactive suggestion. (a) Retrieval supplies whole trajectories without indicating the matching step. (b) ExpActivator activates the matching step. (c) Retrieval returns related records without indicating whether a routine is due. (d) ExpActivator activates the routine matching the time and scenario, then suggests or abstains.

Figure 2: Overview of ExpActivator. Step-level activation (middle) matches each screen to recorded states of task-relevant trajectories and supplies the best-matching action as a reference. Task-level activation (bottom) scores recurring intents by scenario and temporal support and supplies a representative intent, or abstains. A frozen backbone outputs an action or suggestion.

		Personalized			Proactive
Backbone	Method	Type $\uparrow$	SSR $\uparrow$	CSR $\uparrow$	Sem. $\uparrow$	P $\uparrow$	BA $\uparrow$	MCC $\uparrow$
Qwen3-VL-8B	Base	47.6	19.4	23.7	–	–	–	–
	Recent	50.6	20.6	29.2	48.6	68.4	50.3	2.3
	Retrieve	51.6	22.6	41.4	48.4	67.8	48.9	$-5.5$
	LLM-UM	51.0	22.0	39.2	29.1	68.9	51.5	10.5
	HIM-Agent	51.3	22.6	40.2	50.8	72.7	58.5	20.7

Table 1: Personalized execution and proactive suggestion on AndroidIntent. Within each block all methods share the same history and frozen policy. QwenVL-Max’s BA and MCC are derived from its published recall and false-alarm rate, and Always suggest suggests on every state. Values are percentages, and bold marks the best result within each block.

Method	History	Total	All	Opening	Within-traj.	$-$ Base [95% CI]
	Prompt tokens/step		Step success rate			Within-trajectory
Base	0.0	1011.4	19.39	28.26	18.43	–
Recent	756.6	1768.0	20.57	40.90	18.36	$-0.07$ [ $-0.87$ , $+0.85$ ]
Retrieve	730.6	1742.0	22.63	62.06	18.35	$-0.08$ [ $-1.16$ , $+0.99$ ]
LLM-UM	843.7	1855.1	22.01	62.32	17.63	$-0.80$ [ $-1.83$ , $+0.12$ ]
HIM-Agent	417.8	1429.2	22.58	62.32	18.26	$-0.17$ [ $-0.89$ , $+0.57$ ]

Table 2: History-token cost and the two phases of an episode on Qwen3-VL-8B. History tokens are prompt tokens added over the no-history condition. Opening is each episode’s first action and within-trajectory covers the 7,140 actions after it. The last column is the within-trajectory difference from Base with paired 95% user-cluster intervals. Rates are percentages.

Figure 3: Step success by user history size on Qwen3-VL-8B. Users are grouped into quartiles by the records in their memory split, and all six conditions are scored on the same steps. Text above each group gives ExpActivator minus the strongest baseline there, with paired 95% user-cluster intervals.

Control	Control score	ExpActivator	Gain [95% CI]
Step-level activation: policy SSR, $K_{\xi}=1$
Whole trajectory	18.47	23.84	5.36 [3.74, 7.06]
Step-level activation: policy SSR, $K_{\xi}=8$
Previous screen	22.63	26.67	4.03 [3.22, 4.84]
Update every four steps	23.17	26.67	3.50 [2.59, 4.47]
Task-level activation: balanced accuracy

Table 3: Ablations with paired 95% user-cluster intervals. Step-level rows report policy SSR with one or eight retrieved trajectories held fixed; task-level rows report balanced accuracy at the declared operating point.

Figure 4: How step-level activation draws on past trajectories. Left: number of distinct past trajectories that supply the references of one episode. Right: share of steps whose reference comes from a different trajectory than at the previous step. Bars and the line give the mean over the four backbones; whiskers and the shaded band show their range.

Figure 5: Activation on real records. Left: step-level activation places each current screen of one NUS NextBus task beside the recorded state it activates. Right: task-level activation for one user at home, showing four routines; each curve is a routine’s contextual support over the day, dark segments mark when it is activated, gray bands mark abstention, and ticks mark its past occurrences. Intents are translated from Chinese.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{M}_{u},\ \mathcal{I}_{u}$	interaction history of user $u$ and its record indices
$r_{i}=(g_{i},p_{i},c_{i},\tau_{i},\xi_{i})$	record: goal, application, scenario, timestamp, trajectory
$\xi_{i}=\{(s_{i,j},a_{i,j})\}_{j<T_{i}}$	recorded screens and the actions taken from them
$q,\ s_{t},\ \mathcal{H}_{t}$	instruction, current screen, preceding actions
$(c_{x},\tau_{x})$	scenario and timestamp of a proactive query
$\mathcal{B}_{u}^{(d)}(x),\ \kappa_{d},\ \Gamma_{d}$	candidates, compatibility, and readout at granularity $d$

Appendix

Table 4: Notation.

Component	Setting	Value
Step-level	goal encoder $f$	paraphrase-multilingual-MiniLM-L12-v2
	candidate trajectories $K_{\xi}$	8
	screen layout $B$	4 centered quadrants; MAI-UI adds 1 global block
	reference coordinate frame	normalized to $[0,999]$
Task-level	intent encoder $\psi$	paraphrase-multilingual-MiniLM-L12-v2
	routine threshold $\eta$	0.94

Appendix

Table 5: Hyperparameters and inference settings.

History and execution		Action type share (%)
Users	82	click	41.1
History records	15,966	scroll	17.9
Records per user (median)	196.5	wait	13.6
Records per user (min, max)	39, 414	open app	9.8
Execution episodes	775	finish	9.8
Execution steps	7,915	type	5.8

Appendix

Table 6: Statistics of the history and evaluation splits. Action shares are over the 7,915 execution steps.

Backbone	Method	Sem.	P	R	FA	F1
Qwen3-VL-8B	Recent	48.6	68.4	98.6	98.0	80.8
	Retrieve	48.4	67.8	95.8	98.0	79.4
	LLM-UM	29.1	68.9	99.1	96.0	81.3
	HIM-Agent	50.8	72.7	87.9	71.0	79.6
	ExpActivator	52.1	82.6	79.5	36.0	81.0
MAI-UI-8B	Recent	44.5	68.3	100.0	100.0	81.1

Appendix

Table 7: Complete proactive metrics on 215 positive and 100 negative states. All values are percentages; lower is better for FA.

Action reference	Policy SSR
Step-level activation, $K_{\xi}=1$
Whole trajectory	18.47
Selected reference	23.84
Step-level activation, $K_{\xi}=8$
Previous historical action	23.32
Selected reference (ExpActivator)	26.67

Appendix

Table 8: Qwen3-VL-8B controls on the 7,140 within-trajectory actions. Whole trajectory supplies the entire record. Previous historical action replaces the selected action with the action immediately before it in the historical trajectory.

Action reference	Policy SSR	Difference [95% CI]
Previous screen	22.63	4.03 [3.22, 4.84]
Update every four steps	23.17	3.50 [2.59, 4.47]
Current screen (ExpActivator)	26.67	–

Appendix

Table 9: Step-level activation with eight fixed historical trajectories. All conditions run the frozen Qwen3-VL-8B policy on the same current inputs and 7,140 within-trajectory actions. Rates are percentages; differences are Current screen minus control in percentage points with 95% paired user-cluster intervals.

Condition	AUROC	BA	Recall	FA	Semantic
ExpActivator	80.37	72.66	82.33	37.00	55.35
No time weighting	74.10	69.33	84.65	46.00	50.61
No scenario condition	79.95	64.58	91.16	62.00	52.67
No frequency normalization	80.37	72.66	82.33	37.00	53.24

Appendix

Table 10: Task-level activation before suggestion generation on 215 positive and 100 negative states. BA is balanced accuracy and FA is false-alarm rate. Semantic is the official score on emitted positive suggestions. All conditions use the same activation threshold. All values are percentages.

Figure 6: Task-level activation ROC curves and paired AUROC differences. Dots mark the original threshold. Error bars are 95% user-cluster bootstrap intervals. No frequency normalization has the same trigger decisions and curve as ExpActivator. The curves include the standard ROC endpoints described above.

Figure 7: Sensitivity of task-level activation. Left: activation AUROC over the routine threshold $\eta$ and the minimum total support $n_{\min}$ , with the declared setting outlined. Right: the share of the 315 states for which no routine satisfies the support requirements, at $\eta=0.94$ . The AUROC decline beyond $n_{\min}=12$ tracks this loss of coverage.

Temporal bandwidth $\sigma$ (hours)						Same-scenario support $n_{\mathrm{ctx}}$
$\sigma$	AUROC	BA	Recall	FA	No routine	$n_{\mathrm{ctx}}$	AUROC	BA	Recall	FA	No routine
0.25	79.92	73.05	72.09	26.00	27.6	1	79.82	69.59	84.19	45.00	21.6
0.5	79.96	73.07	78.14	32.00	27.6	2	79.77	69.90	82.79	43.00	24.4
0.75	80.20	72.73	80.47	35.00	27.6	3	80.07	71.66	82.33	39.00	27.0
1 †	80.37	72.66	82.33	37.00	27.6	4 †	80.37	72.66	82.33	37.00	27.6
1.5	80.51	71.40	82.79	40.00	27.6	5	79.74	71.50	80.00	37.00	28.9

Appendix

Table 11: One-at-a-time sensitivity of task-level activation on 215 positive and 100 negative states, with the other constants held at their declared values. BA is balanced accuracy, FA is the false-alarm rate, and No routine is the share of states for which no routine meets the support requirements. Declared settings are marked † . All values are percentages except the swept constant.

Figure 8: Step success against history tokens per step on Qwen3-VL-8B. The shaded region holds every point that costs more and scores less than ExpActivator. Left: all steps. Right: actions within the trajectory, where the four memory baselines stay at the no-history level.

Backbone	Base	Recent	Retrieve	LLM-UM	HIM-Agent	ExpActivator
Qwen3-VL-8B	18.43	$-0.07$ [ $-0.87$ , $+0.85$ ]	$-0.08$ [ $-1.16$ , $+0.99$ ]	$-0.80$ [ $-1.83$ , $+0.12$ ]	$-0.17$ [ $-0.89$ , $+0.57$ ]	$+8.24$ [ $+6.51$ , $+10.16$ ]
MAI-UI-8B	17.41	$+0.77$ [ $-0.07$ , $+1.63$ ]	$+2.34$ [ $+1.32$ , $+3.36$ ]	$+0.41$ [ $-0.39$ , $+1.27$ ]	$+0.98$ [ $+0.21$ , $+1.75$ ]	$+5.67$ [ $+4.73$ , $+6.69$ ]
Qwen3.5-9B	16.57	$-0.20$ [ $-1.04$ , $+0.73$ ]	$+0.88$ [ $-0.27$ , $+2.11$ ]	$-0.10$ [ $-1.05$ , $+0.86$ ]	$+0.56$ [ $-0.37$ , $+1.50$ ]	$+3.74$ [ $+2.79$ , $+4.76$ ]
GUI-Owl-1.5-8B	22.30	$+0.48$ [ $-0.59$ , $+1.58$ ]	$+1.72$ [ $+0.75$ , $+2.72$ ]	$+0.63$ [ $-0.17$ , $+1.42$ ]	$+1.92$ [ $+1.24$ , $+2.64$ ]	$+2.38$ [ $+0.53$ , $+4.41$ ]

Appendix

Table 12: Within-trajectory step success over the 7,140 actions after each episode’s first decision. Base is the no-history rate in percent; other columns give the difference from Base in percentage points with paired 95% user-cluster bootstrap intervals (20,000 draws).

Figure 9: Personalized execution by history size on Qwen3-VL-8B. Users are grouped into quartiles by the number of records in their memory split, and all six conditions are scored on the same steps inside each quartile. Top: all steps. Bottom: within-trajectory execution, where the four memory baselines stay at the no-history level in every quartile. Blue text gives ExpActivator minus the strongest baseline in that group, in percentage points with paired 95% user-cluster bootstrap intervals; the strongest baseline is selected per group and is named, so the comparator changes across groups.

Group	Users	Records	Steps	Base	Recent	Retrieve	LLM-UM	HIM-Agent	ExpActivator	Strongest	Gain [95% CI]
All steps
Q1	21	39–111	782	19.31	18.67	22.25	21.74	22.63	30.95	HIM-Agent	$+8.31$ [ $+5.32$ , $+12.50$ ]
Q2	20	118–196	1535	19.09	21.43	22.87	22.54	23.26	29.71	HIM-Agent	$+6.45$ [ $+3.29$ , $+8.74$ ]
Q3	20	197–245	2424	17.08	19.22	21.08	20.21	20.42	28.22	Retrieve	$+7.14$ [ $+3.90$ , $+10.86$ ]
Q4	21	258–414	3174	21.33	21.64	23.79	23.19	23.88	33.55	HIM-Agent	$+9.67$ [ $+6.48$ , $+13.25$ ]
All	82	39–414	7915	19.39	20.57	22.63	22.01	22.58	30.92	Retrieve	$+8.29$ [ $+6.37$ , $+10.40$ ]

Appendix

Table 13: Personalized execution by user history size on Qwen3-VL-8B. Users are grouped into quartiles by records in their memory split. Rates are step success percentages on the steps belonging to each group. Gain is ExpActivator minus the strongest baseline within that group, in percentage points with paired 95% user-cluster bootstrap intervals.

Figure 10: Step-level activation on five steps of one NUS NextBus task. Each current screen activates the most similar recorded state among eight task-relevant trajectories, so the reference moves from one past trip to another as the screen changes; circles mark the frozen policy’s taps. The task instruction is translated from Chinese.

Figure 11: Task-level activation for two more users, in the format of Figure 5 (right). Intents are translated from Chinese.

Figure 12: Task-level activation for three more users, in the format of Figure 5 (right). Top: one commuter on the street and on a vehicle on the same morning. Intents are translated from Chinese.

Explore similar work

Sep 30, 2026cs.LG

Learning Reliable GUI Agents under Imperfect Priors

GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.

Bo Han, Qianyi Wang, Shuai Liu +9

Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.

Graphical User Interface Agents Reliable Prior Knowledge Integration Noise-Aware

Ask PDF

Sep 15, 2026cs.LG

Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents

GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose \textbf{EvoSkill-GUI}, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \textbf{\emph{reflect-revise-reuse}} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of

+16.2\%

+6.0\%

, and

+10.5\%

respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.

Bofan Chen, Boxuan Zhang, Fei Tang +7

Zhejiang University · UESTC

Graphical User Interface Agents Graphical User Interface Skills Reuse

Ask PDF

May 12, 2026cs.AI

Executable Agentic Memory for GUI Agent

Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to

19.6\%

on AndroidWorld, while reducing token costs

6\times

relative to GPT-4o. With a

2.8

s average latency, EAM enables reliable, quick, and long-horizon GUI automation.

Zerui Qin, Sheng Yue, Xingyuan Hua +2

1Tsinghua University, China · 2Sun Yat-sen University, China

Graphical User Interface Agents Agentic Memory Knowledge Graphs Efficient Planning Value Functions

Ask PDF