Selective Knowledge Control for Continual GUI Agent Learning over Application Streams

Zirui Shang; Xin Shu; Yang Liu; Zhi Gao; Xinxiao Wu; Lifeng Fan

cs.CVSep 6, 2026

Selective Knowledge Control for Continual GUI Agent Learning over Application Streams

Authors: Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan

Organizations: Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University

Abstract

Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.

Figures & tables

Figure 1: Example of shared and application-specific knowledge across GUI tasks. Exporting a spreadsheet in LibreOffice Calc and an image in GIMP shares the procedure of opening the File menu, selecting Export , and interacting with an export dialog, while the subsequent steps are application-specific.

Figure 2: Overview of the proposed activation-conditioned selective knowledge control framework. Within each training stage, forward activations partition protected historical neurons into shared and application-specific subsets, and backward gradient surgery applies different update rules to these subsets. After the stage ends, the historical knowledge state is updated with newly identified important neurons and their historical direction subspaces.

Method	Stage	VLC	Thunderbird	Writer	Impress	Calc	GIMP	VS Code	Chrome	Overall
Baseline	Stage 1	37.3	–	–	–	–	–	–	–	37.3
	Stage 2	33.3	42.2	–	–	–	–	–	–	37.5
	Stage 3	29.4	33.3	47.6	–	–	–	–	–	37.7
	Stage 4	33.3	42.2	46.0	37.9	–	–	–	–	39.5
	Stage 5	25.5	48.9	39.7	42.4	15.1	–	–	–	31.9
	Stage 6	23.5	53.3	33.3	45.5	13.5	57.7	–	–	35.5

Table 1: Average success rates (%) over three runs on all seen applications after each training stage. Parentheses show signed percentage-point differences between ours and the baseline. Column-best results are bolded, and “–” denotes unavailable results.

Stage	Method	RP (%)	CP (%)	RAH (%)
Stage 2	Baseline	89.3	42.2	57.3
Stage 2	Ours	100.0 (+10.7)	51.1 (+8.9)	67.6 (+10.3)
Stage 3	Baseline	78.9	47.6	59.4
Stage 3	Ours	97.7 (+18.9)	57.1 (+9.5)	72.1 (+12.7)
Stage 4	Baseline	95.6	37.9	54.3
Stage 4	Ours	96.5 (+0.9)	47.3 (+9.4)	63.5 (+9.2)

Table 2: Retained performance (RP), current-application performance (CP), and retention–adaptation harmonic score (RAH) during Stages 2–8. Parentheses report the signed percentage-point difference (Ours

-

Baseline).

Method	Stage	VLC	Thunderbird	Writer	Overall
Baseline	Stage 1	37.3	–	–	37.3
	Stage 2	33.3	42.2	–	37.5
	Stage 3	29.4	33.3	47.6	37.7
Replay	Stage 1	37.3	–	–	37.3
	Stage 2	39.5	40.5	–	40.0
	Stage 3	35.3	51.1	46.0	44.0

Table 3: Comparison under the same training configuration. The table reports stage-wise success rates and overall performance for the baseline, replay, EWC, LoRA, and the proposed method.

Setting	Stage 2 (%)	Stage 3 (%)	Mean (%)
Baseline	37.5	37.7	37.6
Static Projection	42.7	40.3	41.5
Static Freezing	41.7	38.4	40.1
Ours	43.8	50.3	47.1

Table 4: Ablation studies on selective control with four update-control settings.

Selection Ratio	VLC	Thunderbird	Writer	Overall
$\rho=1\%$	27.5	51.1	44.4	40.9
$\rho=3\%$	33.3	48.9	48.3	44.2
Ours ( $\rho=2\%$ )	35.3	57.8	57.1	50.3

Table 5: Hyperparameter analysis of the selection ratio

\rho

Figure 3: Shared procedural knowledge reuse on a Chrome task.

SVD Rank	$r_{\mathrm{svd}}=2$	$r_{\mathrm{svd}}=4$	$r_{\mathrm{svd}}=8$
Overall Success Rate (%)	38.5	42.7	38.7

Table 6: Hyperparameter analysis of the truncated SVD rank

r_{\mathrm{svd}}

, reporting the overall success rate (%) at Stage 8 under different numbers of retained historical update directions.

Method	Time (s/step)	Tokens/s	TFLOPs
Baseline	4279.23	400.07	289.79
Ours	4324.00	386.73	284.30

Table 7: Training overhead evaluation. We report per-step training time, token throughput (Tokens/s), and compute throughput (TFLOPs) under the same training configuration.

Method	Stage	VLC	VS Code	GIMP	Overall
Baseline	Stage 1	37.3	–	–	37.3
	Stage 2	39.2	65.2	–	54.2
	Stage 3	41.2	59.4	66.2	57.4
Ours	Stage 1	37.3	–	–	37.3
	Stage 2	39.2	69.6	–	56.7
	Stage 3	37.3	63.8	75.6	61.6

Table 8: Application-stream order ablations. The table reports stage-wise success rates and overall performance of the baseline and the proposed method under a shuffled application-stream order.

Figure 4: Shared scope-aware formatting knowledge transferred from Stage 3 Writer to Stage 4 Impress.

Figure 5: Shared destination-aware procedural knowledge transferred from Stage 1 VLC to Stage 8 Chrome.

Figure 6: Layer-wise comparison of stage-wise MLP parameter offsets between ours and the baseline. Rows correspond to the 28 decoder layers, indexed from 0 to 27. (a) Direction similarity $D_{\ell,t}$ , where zero denotes orthogonal directions. (b) Log displacement ratio $\log_{2}R_{\ell,t}$ , where zero denotes equal update magnitudes. S2–S8 correspond to Thunderbird, Writer, Impress, Calc, GIMP, VS Code, and Chrome, respectively.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning
Mar 3, 2026Zhenquan Yao, Zitong Huang, Yihan Zeng +5Continual Learning for LLM Agents RL for GUI Agents
Harbin Institute of Technology · Huawei Noah’s Ark Lab · University College Dublin +1
GUI-AC: Enhancing Continual Learning in GUI Agents
Jun 9, 2026Can Lin, Tao Feng, Hangjie Yuan +3Non-Stationary RL RL for GUI Agents
Beijing University of Posts and Telecommunications · Tsinghua University · Zhejiang University +1
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
Aug 19, 2026Borui Kang, Jinrui Gu, Junhan Lv +3Continual Learning for LLM Agents Continual Learning
State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Mar 3, 2026J/K move · Enter open · S save

CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning

Zhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han+4

Harbin Institute of Technology · Huawei Noah’s Ark Lab · University College Dublin+1

Continual Learning for LLM Agents RL for GUI Agents RL Fine-Tuning

Open paper AskPDF arXiv

Graphical User Interface (GUI) Agents, benefiting from recent advances in multimodal large language models (MLLM), have achieved significant development. However, due to the frequent updates of GUI applications, adapting to new tasks without forgetting old tasks in GUI continual learning remains an open problem. In this work, we reveal that while Supervised Fine-Tuning (SFT) facilitates fast adaptation, it often triggers knowledge overwriting, whereas Reinforcement Learning (RL) demonstrates an inherent resilience that shields prior interaction logic from erasure. Based on this insight, we propose a Continual GUI Learning (CGL) framework that dynamically balances adaptation efficiency and skill retention by enhancing the synergy between SFT and RL. Specifically, we introduce an SFT proportion adjustment mechanism guided by policy entropy to dynamically control the weight allocation between the SFT and RL training phases. To resolve explicit gradient interference, we further develop a specialized gradient surgery strategy. By projecting exploratory SFT gradients onto GRPO-based anchor gradients, our method explicitly clips the components of SFT gradients that conflict with GRPO. On top of that, we establish an AndroidControl-CL benchmark, which divides GUI applications into distinct task groups to effectively simulate and evaluate the performance of continual GUI learning. Experimental results demonstrate the effectiveness of our proposed CGL framework across continual learning scenarios. The benchmark, code, and model will be made publicly available.

Abstract

Continual Learning

Explore similar work

CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning

GUI-AC: Enhancing Continual Learning in GUI Agents

Harness Continual Learning: Continual Adaptation Beyond Model Parameters