Organizations: College of Computer Science and Technology, Zhejiang University, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi · School of Computer Science and Technology, Zhejiang University of Science and Technology, China · School of Computer and Computing Science, Hangzhou City University, China
Recent pre-trained vision-language models often encounter a Multi-Domain Task-Incremental Learning (MTIL) scenario in practice, where a set of classes of multi-modal tasks arrive incrementally. Due to privacy concerns and memory constraints, MTIL with pre-trained models encounters forgetting of knowledge from old tasks, degradation of zero-shot transfer capability, and underfitting of new-task knowledge. To overcome these challenges, previous MTIL methods attempt to learn a discriminative cross-task identification (CTI) module and an effective new-task adaptation (NTA) module. However, current CTI modules suffer from severe task confusion between seen and unseen tasks, and NTA modules cannot adaptively balance the performance and parameter cost while incorporating task-specific knowledge. To alleviate the above dilemmas, we propose an effective and efficient TSP-DLoRA method for MTIL, which consists of Task Subspace Projection (TSP) and Dynamic Low Rank Adapter (DLoRA) modules. Specifically, our TSP module includes a task identifier classifier based on task-specific subspaces and a feature projection strategy that can determine the identifier associated with samples from both seen and unseen tasks. Our DLoRA improves the knowledge adaptation from new tasks by dynamically assigning Low Rank Adapter (LoRA) across transformer layers based on the task distributions. Experimental evaluations across 11 datasets, using three performance metrics, demonstrate the effectiveness of our proposed method. Code is available at https://github.com/FerdinandZJU/TSP-DLoRA.
Figures & tables
Figure 1: (a) Comparison with existing methods. Unlike ZSCL, our method eliminates the need for additional storage and improves parameter efficiency. Compared to MoE-Adapter, our approach incurs significantly lower computational overhead with fewer trainable parameters. In contrast to DIKI, ours not only achieves higher accuracy but also precisely distinguishes whether samples belong to previously seen tasks (Precise Task ID). (b) Mechanism comparison. Existing training-free CTI applies the same NTA operation to both seen and unseen tasks, restricted by a weight “ W ”. Conversely, our TSP module explicitly identifies the boundary between tasks, directing unseen samples to the original CLIP while processing seen samples with DLoRA. (c) Performance trade-off. Our method achieves a superior trade-off between average accuracy (both seen and unseen tasks) and parameter efficiency compared to state-of-the-art methods.
Figure 2: The training process of the DLoRA module in our method. The DLoRA module integrates LoRA into the first Lt transformer layers of both the image and text encoders. The gating mechanism before the LoRA modules in the image encoder determines whether to activate the LoRA based on the feature Fv′(x) derived from the original CLIP. Only the LoRA and the gating mechanisms are trained, while the remaining parts are kept frozen.
Figure 3: The workflow of the TSP module of our method in training and inference processes. During training phase, for image dataset of every task, the TSP module decomposes the image features via SVD and selects top r singular vectors from the right singular vector matrix to be the corresponding task subspace. During inference phase, for every test sample, the TSP module computes the cosine similarity between the raw feature extracted from original CLIP and the subspaces associated with each seen task. The result is compared against a threshold, denoted as “thres.”. The sample is classified using the corresponding DLoRA module if the cosine similarity exceeds the “thres.”. Otherwise, the original CLIP model is utilized to classify the sample.
Method
Extra.
Param.
Aircraft Maji et al. (2013)
Caltech101 Fei-Fei et al. (2004)
CIFAR100 Krizhevsky et al. (2009)
DTD Cimpoi et al. (2014)
EuroSAT Helber et al. (2019)
Flowers Nilsback and Zisserman (2008)
Food Bossard et al. (2014)
MNIST Deng (2012)
OxfordPet Parkhi et al. (2012)
StanfordCars Krause et al. (2013)
SUN397 Xiao et al. (2010)
Average
CLIP
Zero-shot
✗
-
24.3
88.4
68.2
44.6
54.9
71.0
88.5
59.4
89.0
64.7
65.2
65.3
Full Fine-tune
✗
211M
62.0
95.1
89.6
79.5
98.9
97.5
92.7
99.6
94.7
89.6
81.8
89.2
Transfer
Continual-FT
✓
211M
-
67.1
46.0
32.1
35.6
35.0
57.7
44.1
60.8
20.5
46.6
44.6
LwF Li and Hoiem (2017)
✓
211M
-
74.5
56.9
39.1
51.1
52.6
72.8
60.6
75.1
30.3
55.9
58.9
iCaRL Rebuffi et al. (2017)
✓
211M
-
56.6
44.6
32.7
39.3
46.6
68.0
46.0
77.4
31.9
60.5
50.4
LwF-VR Ding et al. (2022)
✓
211M
-
77.1
61.0
40.5
45.3
54.4
74.6
47.9
76.7
36.3
58.6
57.2
Table 1: Comparison with SOTA on MTIL benchmark in terms of “Transfer”, “Average”, and “Last” metrics (%). “Ours” denotes our method. The presented results are derived from the Order- I , for Order- II results, please refer to the Table 2 .
Method
Extra.
Param.
Cars Krause et al. (2013)
Food Bossard et al. (2014)
MNIST Deng (2012)
OxfordPet Parkhi et al. (2012)
Flowers Nilsback and Zisserman (2008)
SUN397 Xiao et al. (2010)
Aircraft Maji et al. (2013)
Caltech101 Fei-Fei et al. (2004)
DTD Cimpoi et al. (2014)
EuroSAT Helber et al. (2019)
CIFAR100 Krizhevsky et al. (2009)
Average
CLIP
Zero-shot
✗
-
64.7
88.5
59.4
89.0
71.0
65.2
24.3
88.4
44.6
54.9
68.2
65.3
Full Fine-tune
✗
211M
89.6
92.7
99.6
94.7
97.5
81.8
62.0
95.1
79.5
98.9
89.6
89.2
Transfer
Continual-FT
✓
211M
89.5
59.6
57.9
40.0
46.7
11.1
70.0
30.5
26.6
37.7
46.6
LwF Li and Hoiem (2017)
✓
211M
-
87.8
58.5
71.9
46.6
57.3
12.8
81.4
34.5
34.5
46.8
53.2
iCaRL Rebuffi et al. (2017)
✓
211M
-
86.1
51.8
67.6
50.4
57.9
11.0
72.3
31.2
32.7
48.1
50.9
LwF-VR Ding et al. (2022)
✓
211M
-
88.2
57.0
71.4
50.0
58.0
13.0
82.0
34.4
29.3
47.6
53.1
Table 2: Comparison with SOTA on MTIL benchmark in terms of “Transfer”, “Average”, and “Last” metrics (%). “Ours” denotes our method. The presented results are derived from the Order- II , for Order- I results, please refer to the Table 1 .
Method
Extra.
Param.
Aircraft
Caltech101
CIFAR100
DTD
Flowers
Food
StanfordCars
SUN397
Average
CLIP
Zero-shot
✗
-
24.8
92.9
68.4
43.8
71.4
85.8
65.8
62.6
64.4
Full Fine-tune
✗
211M
62.0
96.2
89.6
79.5
97.5
92.7
89.6
81.8
86.1
Transfer
ZSCL Zheng et al. (2023)
✓
211M
87.3
67.7
45.4
67.8
86.6
59.7
63.4
68.3
L2P Wang et al. (2022c)
✗
0.5M
66.7
54.3
30.6
47.3
71.5
54.6
52.4
53.9
DualPrompt Wang et al. (2022b)
✗
1.8M
78.8
64.4
32.0
51.7
77.5
49.4
51.3
57.9
S-Prompts Wang et al. (2022a)
✗
0.5M
70.3
52.7
31.5
54.8
74.0
55.4
50.0
55.5
Table 3: Comparison of SOTA methods on 16-shot MTIL-FS benchmark.
Dataset
Classes
Train
Test
Aircraft Maji et al. (2013)
100
3334
3333
Caltech101 Fei-Fei et al. (2004)
101
6212
2465
CIFAR100 Krizhevsky et al. (2009)
100
50000
10000
DTD Cimpoi et al. (2014)
47
2068
1692
EuroSAT Helber et al. (2019)
10
18800
8100
Flowers Nilsback and Zisserman (2008)
102
4706
2463
Table 4: Details of 11 datasets.
Method
Trans .
Avg.
Lst.
DIKI*
67.4
75.7
85.1
DIKI*+TSP
68.9
76.3
85.3
LoRA+TSP
69.3
76.5
85.4
DLoRA+TSP
69.4
77.5
86.5
Table 5: The ablation experiments for the DLoRA and TSP modules in our method. The asterisk (*) denotes the empirical results obtained from our experiments.
Method
Time
GPU
Train
Infer.
MoE-Adapter
5.3h
48 GB
1.58s/it
0.63s/it
DIKI
2.6h
24 GB
0.36s/it
0.33s/it
Ours
2.4h
24 GB
0.38s/it
0.23s/it
Table 6: Computational cost of three PEFT methods. “GPU” denotes the GPU memory requirement, “Train” and “Infer.” represent the training and inference speeds, respectively.
Figure 4: The distribution of similarity scores for the TSP module across different tasks. The red line denotes a threshold set at 0.96. The majority of samples exhibit similarity scores above this threshold. The dense distribution of similarity scores also indicates the TSP module’s capability to effectively capture and extract critical information from each task.
Figure 5: Impact of learning rate and temperature on the DLoRA module. We vary one parameter while keeping the other fixed. The results demonstrate that the DLoRA module is robust to variations in these hyperparameters. Furthermore, adjustments to the DLoRA module do not affect the TSP module, yielding consistent performance on the “Transfer” metric.
Figure 6: Visualization of six task-specific distributions generated by the TSP module. We randomly selected 1,000 samples from each dataset. The majority of samples are correctly classified, as shown by their corresponding colors. A small number of samples are misclassified as unseen tasks, while an even smaller number are incorrectly assigned to other seen tasks. Overall, the TSP module achieves an accuracy exceeding 90% for each task.
Threshold
0.89
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
Energy
0.89
0.889
0.912
0.921
0.909
0.867
0.800
0.699
0.589
0.498
0.462
0.90
0.869
0.897
0.918
0.923
0.899
0.850
0.762
0.646
0.525
0.466
0.91
0.846
0.878
0.908
0.926
0.920
0.884
0.811
0.699
0.559
0.474
0.92
0.815
0.850
0.885
0.914
0.930
0.915
0.862
0.767
0.614
0.492
0.93
0.779
0.815
0.853
0.888
0.921
0.930
0.900
0.827
0.682
0.520
Table 7: TSP module performance under different energy and threshold setting.
Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to parameter explosion. Conversely, recent similarity-driven sharing mechanisms falsely equate superficial visual similarity with underlying alignment consistency. This fundamental mismatch triggers severe negative transfer between visually similar but logically distinct tasks and fails to exploit alignment reuse across visually diverse ones. We argue thatalignment sharing is fundamentally a geometric problem of overlapping optimization trajectories within shared low-rank subspaces. Grounded in this insight, we propose iGSP, a novel framework that achieves efficient adaptation via implicit gradient subspace projection. Leveraging the early convergence of MoE routers to establish the subspace basis, iGSP bifurcates the adaptation process into two phases. First, the Subspace Identification phase introduces candidate experts via basis pre-expansion, applies a novel subspace-constrained regularization to implicitly project new task gradients onto the historical subspace, and precisely prunes redundant dimensions by treating routing probabilities as gradient flow indicators, ultimately to maximize knowledge reuse. Second, the Orthogonal Subspace Fine-Tuning phase fixes this structural basis and removes the regularization to rapidly fit the task-specific residual loss. Extensive experiments on the MTIL benchmark demonstrate that iGSP achieves state-of-the-art accuracy while significantly improving training efficiency, reducing the average trainable parameters by 42.7% compared to current SOTA methods, and decreasing the final total parameters by 86.9% relative to counterparts. The source code is available at https://github.com/GeoX-Lab/iGSP.
Xuezhi Cui, Dongbo Zhou, Wang Guo +8
School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · Faculty of Artifi cial Intelligence in Education in Central China Normal University, Wuhan 430079, China · School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology, Xiangtan 411201, China
Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.
Mao-Lin Luo, Yi-Lin Zhang, Zi-Hao Zhou +5
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education, China · Huawei Noah’s Ark Lab
Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma. Most existing methods rely on fixed architectures with statically allocated parameters, which limits adaptation to new domains and aggravates catastrophic forgetting. To address these challenges, we propose DIMoE-Adapters, a Dynamic Incremental Mixture-of-Experts Adapters framework that introduces a dynamic expert evolution paradigm to balance stability and plasticity. This paradigm is implemented through two collaborative components: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE constructs and evolves a sparse expert pool through expert optimization dynamics, improving plasticity while reducing redundant capacity. PGES controls expert utilization based on the pool shaped by SCEE, improving stability across both previously encountered and unseen tasks. Extensive experiments show that DIMoE-Adapters outperforms previous state-of-the-art methods across various settings.
Mengxin Qin, Xiang Zhang, Xi Wang +3
School of Electronic Engineering, Xidian University Xi’an 710071, China