Organizations: Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland · National University of Singapore, Singapore, Singapore · School of Computer Science and Technology, Dalian University of Technology, Dalian, China
The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams. While pre-trained models have shown powerful performance in continual learning, they still require finetuning to adapt effectively to downstream tasks. However, prevailing Parameter-Efficient Fine-Tuning (PEFT) methods operate through empirical, black-box optimization at the weight level. These approaches lack explicit control over representation drift, leading to sensitivity to domain shifts and catastrophic forgetting in continual learning scenarios. In this work, we introduce Continual Representation Learning (CoRe), a novel framework that for the first time shifts the finetuning paradigm from weight space to representation space. Unlike conventional methods, CoRe performs task-specific interventions within a low-rank linear subspace of hidden representations, adopting a learning process with explicit objectives, which ensures stability for past tasks while maintaining plasticity for new ones. By constraining updates to a low-rank subspace, CoRe achieves exceptional parameter efficiency. Extensive experiments across multiple continual learning benchmarks demonstrate that CoRe not only preserves parameter efficiency but also significantly outperforms existing state-of-the-art methods. Our work introduces representation finetuning as a new, more effective and interpretable paradigm for continual learning.
Figures & tables
Fig. 1 : The overall structure of the proposed method. (a) illustrates a standard ViT block and (b) depicts the implementation of ReFT. Unlike previous finetuning approaches, ReFT directly intervenes in the model by modifying its intermediate features. Specifically, the features are projected into a low-rank subspace via learnable parameters R, w, and b, and then mapped back to the original dimension. Gray areas indicate frozen parameters, and green areas denote trainable components.
Method
Aircraft
Clathch101
CIFAR
DTD
EuroSAT
Flowers
Food
MNIST
OxfordPet
StanfordCars
SUN397
Avg
ViT [ 7 ]
64.50
99.01
95.14
92.57
96.34
99.86
96.23
96.49
97.09
72.15
93.25
Finetune
67.94
99.36
95.03
92.39
89.60
99.81
94.12
96.38
97.01
70.66
93.11
Prompt [ 10 ]
69.33
99.29
98.01
90.04
89.84
99.79
96.61
94.60
97.56
79.92
93.44
SSF [ 26 ]
66.63
99.31
97.52
93.85
94.39
99.81
95.96
97.36
96.98
75.11
93.14
Adapter [ 9 ]
64.55
99.08
97.98
92.57
95.89
99.86
96.40
99.48
96.97
72.22
93.19
CoRe(Ours)
75.09
99.43
98.15
92.71
96.43
99.87
96.77
99.60
97.00
79.41
93.48
TABLE I : Performance comparison of different methods under the Task Incremental Learning scenario. All experiments are based on the ViT-B/16-IN21K. Each task consists of 5 classes for DTD and OxfordPet, 2 classes for EuroSAT and MNIST, 20 classes for StandCars and SUN397, and 10 classes for the remaining datasets. The best results are highlighted in bold.
OfficeHome Inc65
CORe50 Inc50
CDDB Inc2
DomainNet Inc345
Method
Avg
Last
Avg
Last
Avg
Last
Avg
Last
ViT [ 7 ]
74.22
64.99
77.55
61.49
76.30
72.41
50.73
40.37
Finetune
69.06
58.15
75.86
58.22
71.98
66.64
55.31
42.54
Prompt [ 10 ]
75.30
65.95
76.20
57.20
76.51
72.42
55.32
44.09
SSF [ 26 ]
75.48
65.80
76.42
59.02
75.39
69.34
53.57
42.50
Adapter [ 9 ]
74.67
65.83
76.50
58.53
76.64
72.07
55.51
44.23
TABLE II : Performance comparison of different methods under the Domain Incremental Learning scenario. All experiments are conducted using ViT-B/16-IN21K. For all datasets, each task contains the same number of classes. The best results are highlighted in bold.
CIFAR Inc5
CUB Inc10
IN-R Inc5
IN-A Inc10
ObjNet Inc10
Omni Inc30
VTAB Inc10
Method
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
ViT [ 7 ]
87.57
81.26
92.20
86.73
62.58
54.55
60.5
49.44
65.45
53.59
79.34
73.15
85.99
84.38
Finetune
87.67
81.27
91.82
86.39
70.51
62.42
61.57
50.76
61.41
48.34
73.02
65.03
87.47
80.44
Prompt [ 10 ]
88.46
82.17
91.02
84.99
68.79
60.48
60.59
48.72
67.83
54.65
81.05
74.47
86.59
83.06
SSF [ 26 ]
87.78
81.98
91.72
86.13
68.94
60.60
62.81
51.48
69.15
56.64
80.53
74.00
85.66
81.92
Adapter [ 9 ]
90.65
85.15
92.21
86.73
72.35
64.33
60.53
49.57
67.18
55.24
80.75
74.37
85.95
84.35
TABLE III : Performance comparison of various finetuning methods under Class Incremental Learning scenario. All experiments are based on ViT-B/16-IN21K. Here, ’IN-R’ denotes ImageNet-R datasets, ’IN-A’ refers to ImageNet-A datasets, ’ObjNet’ represents the ObjectNet dataset, and ’Omni’ stands for the OmniBenchmark dataset. For all datasets, each task contains an equal number of classes. The best results are highlighted in bold.
Fig. 2 : Comparison of trainable parameters and Avg accuracy of finetuning methods, the experiments are conducted on CIFAR Inc10 with ViT-B/16-IN21K.
Fig. 3 : Average value of Last accuracy of methods under different random seeds. All the results are conducted on CIFAR Inc10 with ViT-B/16-IN21K.
α= 1
α= 0.5
α= 0.1
α= 0.05
α= 0.01
Method
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
ViT [ 7 ]
87.13
81.26
87.07
81.14
86.70
80.88
86.51
80.47
84.70
78.07
Finetune
88.56
82.80
88.06
82.52
88.40
83.11
88.06
82.24
85.70
79.68
Prompt [ 10 ]
90.59
85.27
90.51
85.12
88.43
84.99
86.83
80.45
76.61
68.52
SSF [ 26 ]
90.71
85.21
90.55
85.09
90.17
84.88
89.71
84.38
87.03
80.96
Adapter [ 9 ]
92.27
87.50
91.94
87.14
89.51
84.15
88.29
82.69
85.25
79.09
TABLE IV : Performance comparison between CoRe and other finetune methods under different imbalance factors α . All experiments are conducted on CIFAR Inc10 with ViT-B/16-IN21K, the best results are highlighted in bold.
Fig. 4 : Last accuracy of various finetuning methods under Class Incremental Learning scenario using ViT-B/16-IN1K as the backbone. Each task contains the same number of classes across all datasets. Core consistently outperforms other finetuning approaches even with the ViT-B/16-IN1K architecture.
Reft Rank
Avg
Last
4
92.28
87.62
8
92.34
87.63
16
92.27
87.51
32
92.02
87.29
64
92.01
87.08
TABLE V : Ablation studies on ReFT rank, where ”Reft Rank” indicates the rank value of every ReFT in model. All experiments are conducted on CIFAR Inc10 with ViT-B/16-IN21K.
Layer
Avg
Last
1
91.95
87.25
3
91.59
86.61
6
92.16
87.40
9
92.17
87.59
12
92.24
87.63
TABLE VI : Ablation studies on the number of block layer inserted with ReFT. The ViT-B/16-IN21K architecture consists of 12 block layers. Here, ”1” indicates that ReFT is inserted into only one block layer, and ”12” denotes that ReFT is applied to every block layer. All experiments are conducted on CIFAR Inc10 with ViT-B/16-IN21K.
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Electronic Science and Engineering, Nanjing University, China +1
Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · City University of Hong Kong, Hong Kong, China · School of Artificial Intelligence, Jilin University, Changchun, China +3