Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

Ruilin Li; Heming Zou; Xiufeng Yan; Zheming Liang; Jie Yang; Chenliang Li; Xue Yang

cs.LGMar 19, 2026

Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

Authors: Ruilin Li, Heming Zou, Xiufeng Yan, Zheming Liang, Jie Yang, Chenliang Li, Xue Yang

Organizations: Wuhan University · Shanghai Innovation Institute · Tsinghua University · China University of Mining and Technology ·

Abstract

Recent paradigms in Random Projection Layer (RPL)-based continual representation learning have demonstrated superior performance when building upon a pre-trained model (PTM). These methods insert a randomly initialized RPL after a PTM to enhance feature representation in the initial stage. Subsequently, a linear classification head is used for analytic updates in the continual learning stage. However, under severe domain gaps between pre-trained representations and target domains, a randomly initialized RPL exhibits limited expressivity under large domain shifts. While largely scaling up the RPL dimension can improve expressivity, it also induces an ill-conditioned feature matrix, thereby destabilizing the recursive analytic updates of the linear head. To this end, we propose the Stochastic Continual Learner with MemoryGuard Supervisory Mechanism (SCL-MGSM). Unlike random initialization, MGSM constructs the projection layer via a principled, data-guided mechanism that progressively selects target-aligned random bases to adapt the PTM representation to downstream tasks. This facilitates the construction of a compact yet expressive RPL while improving the numerical stability of analytic updates. Extensive experiments on multiple exemplar-free Class Incremental Learning (CIL) benchmarks demonstrate that SCL-MGSM achieves superior performance compared to state-of-the-art methods.

Figures & tables

Figure 1: Overview of the prior RPL-based CIL paradigm and comparison of initial-stage RPL construction. (a) Prior Methods: After first stage adaptation, frozen PTM extracts features $\bm{Z}_{\text{init}}$ , which are projected through a randomly initialized RPL ( $\bm{W}_{\text{RPL}}$ ) to obtain high-dimensional features $\bm{H}_{\text{init}}$ , followed by computing classifier weights $\bm{W}_{\beta}$ . During incremental learning, new features $\bm{Z}_{t}$ pass through the same frozen $\bm{W}_{\text{RPL}}$ to $\bm{H}_{t}$ , and only $\bm{W}_{\beta}$ is updated to $\bm{W}_{\beta}^{(t)}$ via recursive ridge regression. (b) Our Method: We leverage the initial task and PTM to inform MGSM-guided RPL construction. During incremental learning, $\bm{W}_{\beta}$ is updated recursively as in (a).

Figure 2: Overview of MGSM-guided RPL construction in SCL-MGSM. Data from any stage can serve as the initialization set to build the RPL from scratch. Random hidden units are progressively sampled, evaluated by MGSM, and appended to the RPL only if they satisfy the supervisory criterion. The construction terminates once the residual converges below a predefined threshold $\varepsilon$ . See Appendix B for details.

Method	ImageNet-R (B-0)				ImageNet-A (B-0)
Joint linear probe	67.23 ±0.05				50.98 ±0.65
Joint fine-tuning	83.67 ±0.14				60.99 ±0.32
	Inc-5		Inc-10		Inc-5		Inc-10
	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$
CODA-Prompt	57.52 ±0.68	65.71 ±0.00	68.03 ±0.42	74.57 ±0.74	28.81 ±0.60	41.45 ±0.47	39.21 ±0.83	50.17 ±0.97
SD-LoRA	66.29 ±1.10	73.67 ±1.19	72.25 ±0.72	78.14 ±0.80	32.92 ±1.07	46.00 ±1.32	45.95 ±0.53	58.41 ±1.01

Table 1: Comparison of CIL Performance Across Methods and Benchmarks. Bold and underline mark the best and second-best results.

A_{\text{last}}

and

A_{\text{avg}}

denote final and average accuracy (%). B-

m

Inc-

n

(

m

: initial classes,

n

: incremental classes;

m{=}0

: equal split). Joint linear probe trains only the classifier head on all tasks, while Joint fine-tuning updates the entire PTM on all tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Illustration of the MGSM-driven RPL construction process.

Dataset	Avg. Final Nodes ( $L^{*}$ )	Std. Dev.
ImageNet-R	14650	± 106
ImageNet-A	15950	± 313
ObjectNet	15250	± 176
OmniBenchmark	15450	± 248

Appendix

Table 4: Average final hidden sizes (

L^{*}

) of SCL-MGSM across datasets.

Figure 14: Impact of first-session adaptation on representative RPL-based methods. We report $\Delta A_{\text{last}}$ relative to the corresponding Joint Linear Probe baseline on four datasets. Dark bars indicate variants with FSA, and light bars indicate variants without FSA. Larger positive values mean that the incremental learner exceeds the frozen-backbone joint linear probe by a wider margin.

Method	ImageNet-R (B-0)				ImageNet-A (B-0)
Joint Linear Probe	82.47 $\pm$ 0.07				69.67 $\pm$ 0.37
	Inc-5		Inc-10		Inc-5		Inc-10
	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$
SimpleCIL	75.35 $\pm$ 0.00	80.82 $\pm$ 0.94	75.35 $\pm$ 0.00	80.52 $\pm$ 0.91	68.93 $\pm$ 0.00	77.58 $\pm$ 1.13	68.93 $\pm$ 0.00	77.13 $\pm$ 1.07
KLDA	75.78 $\pm$ 0.17	80.76 $\pm$ 1.15	75.78 $\pm$ 0.16	80.79 $\pm$ 0.78	66.91 $\pm$ 0.63	70.13 $\pm$ 1.76	66.91 $\pm$ 0.63	70.15 $\pm$ 1.49
LoRanPAC	84.06 $\pm$ 0.10	88.34 $\pm$ 0.55	84.97 $\pm$ 0.21	89.11 $\pm$ 0.55	66.69 $\pm$ 0.65	73.60 $\pm$ 1.08	69.23 $\pm$ 0.44	75.28 $\pm$ 1.12

Appendix

Table 5: Comparison of CIL performance across methods with DINO-v2 as the PTM. B-

m

Inc-

n

denotes

m

initial classes and

n

incremental classes (

m{=}0

: equal split).

A_{\text{last}}

and

A_{\text{avg}}

denote final and average accuracy (%), respectively. Bold and underline mark the best and second-best incremental results under each protocol.

Method	ImageNet-R (B-0)				ImageNet-A (B-0)
	Inc-5		Inc-10		Inc-5		Inc-10
	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$	$A_{\text{last}}$	$A_{\text{avg}}$
SimpleCIL	37.28 $\pm$ 0.00	45.72 $\pm$ 1.24	37.28 $\pm$ 0.00	45.13 $\pm$ 1.07	13.63 $\pm$ 0.00	23.89 $\pm$ 0.11	13.63 $\pm$ 0.00	23.14 $\pm$ 0.33
KLDA	48.96 $\pm$ 0.18	55.82 $\pm$ 0.99	48.89 $\pm$ 0.40	55.08 $\pm$ 1.21	15.87 $\pm$ 0.40	21.75 $\pm$ 0.62	15.80 $\pm$ 0.37	20.99 $\pm$ 1.40
LoRanPAC	53.38 $\pm$ 0.21	62.18 $\pm$ 0.84	53.48 $\pm$ 0.25	61.75 $\pm$ 0.68	22.03 $\pm$ 0.45	30.81 $\pm$ 1.01	21.94 $\pm$ 0.60	30.15 $\pm$ 1.24
AnaCP	52.98 $\pm$ 0.41	61.63 $\pm$ 1.00	53.03 $\pm$ 0.43	61.19 $\pm$ 0.39	17.86 $\pm$ 0.17	28.49 $\pm$ 0.73	17.95 $\pm$ 1.16	27.65 $\pm$ 0.89

Appendix

Table 6: Comparison of CIL performance across methods with ResNet101 as the PTM. B-

m

Inc-

n

denotes

m

initial classes and

n

incremental classes (

m{=}0

: equal split).

A_{\text{last}}

and

A_{\text{avg}}

denote final and average accuracy (%), respectively. Bold and underline mark the best and second-best results under each protocol.

Method	ImageNet-R (B-0)		ImageNet-A (B-0)
	Inc-5	Inc-10	Inc-5	Inc-10
SimpleCIL	8.06 ±0.46	7.73 ±0.26	11.75 ±0.51	11.59 ±0.33
KLDA	6.89 ±0.75	6.52 ±0.65	10.15 ±0.49	11.67 ±0.21
RanPAC	7.11 ±0.49	6.34 ±0.38	24.83 ±6.31	10.53 ±0.44
G-ACIL	7.63 ±0.35	7.21 ±0.43	13.41 ±0.27	12.36 ±0.07
LoRanPAC	6.66 ±0.17	6.03 ±0.09	10.98 ±0.32	10.48 ±0.52

Appendix

Table 7: Comparison of Average Forgetting. We report

F_{\text{avg}}

(in %) for representative incremental methods; lower is better. Bold and underline mark the lowest and second-lowest forgetting under each protocol.

Figure 15: Top-1 accuracy during sequential continual learning using ViT-B/16-IN21K.

Figure 16: Eigenvalue and condition number analysis of $\bm{P}_{t}$ . Left: maximum eigenvalue. Center: minimum eigenvalue. Right: condition number of $\bm{P}_{t}$ .

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 159 papers

Continual Learning of Domain-Invariant Representations
May 15, 2026Pascal Janetzky, Tobias Schlagenhauf, Stefan FeuerriegelOOD Generalization Cross-Domain Generalization
LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany +1
MePo++: Unifying Representation Refinement and Reconciliation for General Continual Learning
Sep 4, 2026Guanglong Sun, Kanglei Zhou, Liyuan Wang +6Continual Learning Online Continual Learning
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China +1
Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning
Aug 17, 2026Zhiming Xu, Huiyu Yi, Zhen-Hao Xie +4Incremental Learning
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Electronic Science and Engineering, Nanjing University, China +1

May 15, 2026J/K move · Enter open · S save

Continual Learning of Domain-Invariant Representations

Pascal Janetzky, Tobias Schlagenhauf, Stefan Feuerriegel

LMU Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany+1

OOD Generalization Cross-Domain Generalization Invariant Representation Learning

Open paper AskPDF arXiv

Continual learning (CL) aims to train models sequentially over multiple domains without forgetting previously learned knowledge. However, existing CL methods optimize for in-domain performance and are therefore prone to learning spurious, domain-specific cues (``shortcut learning''), which limits generalization to unseen domains after deployment. In this paper, we address this limitation through continual learning of domain-invariant representation. We introduce a broad class of CL methods that sequentially learn representations capturing invariant structures across domains. Our methods are motivated by the observation that such invariant structures often preserve the underlying causal mechanisms, which can reduce the risk of overfitting to domain-specific cues and thus offer better out-of-domain generalization. Our proposed CL methods combine replay-based training with a tailored sequential invariance alignment to learn -- and preserve -- invariant structures over time. We evaluate our methods under a deployment-oriented protocol that measures performance on unseen target domains. Across six benchmark and real-world datasets spanning vision, medicine, manufacturing, and ecology, our methods consistently outperform existing CL baselines in terms of generalization to unseen target domains. As an ablation, we further show that naïve extensions of sequential training with existing domain-invariant representation learning (DIRL) methods provide only limited benefits. To the best of our knowledge, this is the first work to develop domain-invariant representation methods for CL.

Abstract

Class-Incremental Learning

Explore similar work

Continual Learning of Domain-Invariant Representations

MePo++: Unifying Representation Refinement and Reconciliation for General Continual Learning

Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning