A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

Haihua Luo; Xuming Ran; Jiangrong Shen; Timo Hämäläinen; Zhonghua Chen; Qi Xu; Fengyu Cong

cs.CVMar 11, 2026

A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

Authors: Haihua Luo, Xuming Ran, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong

Organizations: Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland · Equal contribution · National University of Singapore, Singapore, Singapore · Lab of Brain-Machine Intelligence, Zhejiang University, Zhejiang, China ·

Abstract

Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a vision-language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).

Figures & tables

Fig. 1 : The overall framework of SimE. The green represents trainable and the grey denotes frozen components. A ) illustrates the incremental learning tasks, which include t tasks. Specifically, we finetune the trainable parameters in SimE for task 1, while freezing all the parameters in SimE for the remaining tasks. B ) The learning process for Task 1 can be divided into three stages: in the Adapter stage, the image encoder is finetuned using adapters; in the Prototype 1 stage, prototypes are computed based on the finetuned image encoder, and the classifier is updated; in the Test 1 stage, the classification performance of the model is evaluated. C ) In the computation process for subsequent tasks $i$ ( $1<i<t$ ), all weights are frozen, only the prototypes are computed, and the classifier is updated. D ) depicts the architectures of various image encoders.

Fig. 2 : Comparison of previous and current finetuning approaches: The previous approach, AdaptFormer (A), is contrasted with our Multi-Adapter finetuning (B, C, and D). The modules colored in green are trainable, while those in gray are frozen. In AdaptFormer and Multi-Adapter, the AdaptMLP, AdaptAtten, and AdaptAll modules are parameterized by a bottom-up bottleneck module with trainable parameters, whereas the original MLP and Self-Attention modules remain frozen. The AdaptFormer consists of the original frozen branch coupled with AdaptMLP. In contrast, our Multi-Adapter incorporates various trainable modules alongside the frozen branch for enhanced adaptability. And $B\times$ is represented by $B$ Blocks.

	CIFAR100						TinyImageNet
	10 Steps		20 Steps		50 Steps		5 Steps		10 Steps		20 Steps
Methods	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
UCIR [ 32 ]	58.66	43.39	58.17	40.63	56.86	37.09	50.30	39.42	48.58	37.29	42.84	30.85
PASS [ 33 ]	-	-	-	-	-	-	49.54	41.64	47.19	39.27	42.01	32.93
DyTox [ 34 ]	67.33	51.68	67.30	48.45	64.39	43.47	55.58	47.23	52.26	42.79	46.18	36.21
DER [ 35 ]	74.64	64.35	73.98	62.55	72.05	59.76	-	-	-	-	-	-

TABLE I : Comparison on the accuracy of different CIL methods. The Average and Last accuracy of different CIL methods on CIFAR100 and TinyImageNet benchmark. The † indicates the result based on the CLIP ViT-L/14 pre-trained on Laion-2B. The 100 classes of TinyImageNet are used as base classes. The best results are coloured grey.

	CIFAR100		CUB200		ImaneNet-R		ImaneNet-100		VTAB		ImaneNet-A		ObjectNet		OmniBenchmark
Methods	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
L2P [ 21 ]	81.90	73.08	71.90	62.99	81.67	75.98	80.51	67.22	77.11	77.10	49.39	41.71	63.78	52.19	73.36	64.69
DualPrompt [ 25 ]	81.45	72.51	71.74	62.14	82.01	75.77	80.65	67.38	83.36	81.23	53.71	41.67	59.27	49.33	73.92	65.52
CODA-Prompt [ 50 ]	76.98	62.25	66.61	50.88	78.00	67.52	64.13	34.76	83.90	83.02	53.54	42.73	66.07	53.29	77.03	68.09
Boosting-CL [ 13 ]	85.21	77.52	69.56	55.09	83.48	74.81	88.05	80.90	76.69	66.25	61.00	49.04	68.12	55.30	82.13	73.88
APER [ 36 ]	75.76	65.50	78.80	70.61	78.62	71.35	85.84	76.40	85.95	84.35	60.47	49.37	67.18	55.24	80.75	74.37

TABLE II : Comparison between SimE and famous parameter-efficient CIL methods that utilize pre-trained models on multiple datasets. All datasets are split into 10 steps, except ObjectNet, which is split into 20 steps. All experiments are conducted based on CLIP ViT-B/16. The best results are coloured grey.

Panel of Figure 5 — Fig. 3 : Comparison on the efficiency of different CIL methods. The dotted line and right axis coloured orange present the Last accuracy or Avg accuracy. (a)(b)(c) denote the Training parameters, GPU usage, Memory bank size and Last accuracy of different CIL methods respectively, (d)(e) is the Training parameters and Avg accuracy of Ours under different bottleneck dimensions and number of adapters. (f) show the comparison between Ours and other CIL methods in training parameters and Avg accuracy. All the experiments are conducted on CIFAR100, and (a)-(e) are conducted in 10steps.

Panel of Figure 6 — Fig. 4 : Influence of adapters’ position and number between transformer blocks. The x-axis represents the number of adapters in the encoder, with the numerical ranges indicating the positions of the adapters. For example, ”1-3” signifies that adapters are inserted in the first 3 blocks. The accuracy shown in (c) and (d) represents the average results for different adapter positions with the same number of adapters. All results are based on CIFAR100.

				10 steps		20 steps		50 steps
Adapt-MLP	Adapt-Atten	Adapt-All	Para(M)	Avg	Last	Avg	Last	Avg	Last
✗	✗	✗	0	79.69	70.08	80.41	70.08	80.80	70.08
✔	✗	✗	1.19	85.60	76.70	85.30	76.02	84.09	73.77
✗	✔	✗	1.19	85.94	77.10	85.67	76.61	84.16	73.88
✗	✗	✔	1.19	85.77	76.83	85.48	76.16	84.16	73.86
✔	✔	✗	2.38	85.73	76.79	85.42	76.09	84.76	74.66

TABLE III : The results of different implementations of Multi-Adapter. The structures of Adapter-MLP, Adapter-Atten and Adapt-All are shown in Fig. 2 . ”Para” refers to trainable parameters, with ”M” standing for million. All experiments are conducted on CIFAR100 and the best results are coloured grey

	CIFAR100						TinyImageNet
	10 steps		20 steps		50 steps		10 steps		20 steps		50 steps
Blocks	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
WIT-400M [ 7 ]	85.60	76.70	85.30	76.02	84.09	73.77	79.35	75.37	79.32	75.37	79.29	75.37
Laion-400M [ 38 ]	87.14	79.54	86.86	78.82	85.95	77.63	80.62	78.01	80.46	77.48	81.06	79.22
Laion-2B [ 38 ]	88.34	81.33	88.47	80.89	87.91	80.09	81.98	79.99	81.83	79.77	82.78	81.63
DataComp-1B [ 16 ]	88.04	80.77	87.89	79.88	87.57	79.25	81.38	79.11	81.49	78.70	82.51	80.89

TABLE IV : The influence of CLIP pre-trained datasets. WIT-400M is the closed-source dataset of OpenAI while others are from Open_CLIP. All results are conducted on ViT-B/16 and the 100 classes of TinyImageNet are used as base classes. The best results are coloured grey.

Panel of Figure 9 — Fig. 5 : The t-SNE visualization of CLIP pre-trained on different datasets. All results are conducted on CIFAR100 with ViT-B/16 as backbone containing 10 classes.

	10 steps		20 steps		50 steps
Blocks	Avg	Last	Avg	Last	Avg	Last
ViT-B/16	85.94	77.10	85.67	76.61	84.16	73.88
ViT-B/32	83.60	74.43	82.02	71.74	81.18	70.06
ViT-L/14-336px	88.53	80.85	88.02	80.45	89.12	81.65
ViT-L/14	88.79	81.44	88.57	81.01	89.73	82.60

TABLE V : The influence of CLIP ViT backbones size. The experiments use the corresponding data preprocessing while ”336px” indicates the images are resized to 336. All experiments conducted on CIFAR100 and the best results are colored grey.

	CIFAR100						ImageNet-100
	10 Steps		20 Steps		50 Steps		5 Steps		10 Steps		20 Steps
Methods	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
UCIR [ 32 ]	58.70	42.90	58.20	41.10	-	-	76.00	64.00	70.50	55.30	64.70	47.80
iCaRL [ 6 ]	79.35	70.97	73.32	64.55	71.28	59.07	-	-	-	-	-	-
DyTox [ 34 ]	77.00	67.50	76.80	64.30	75.50	59.50	-	-	80.80	72.50	-	-
DER [ 35 ]	74.64	64.35	73.98	62.55	72.05	59.76	-	-	76.10	66.10	-	-

TABLE VI : Comparison on the accuracy of different CIL methods. The Average and Last accuracy of different CIL methods on CIFAR100 and ImageNet-100 benchmark. All the methods use CLIP ViT-B/16 as the backbone. In SimE, all results use 12 adapters (one per transformer block). The best results are coloured grey.

	CIFAR100						ImageNet-100
	10 Steps		20 Steps		50 Steps		5 Steps		10 Steps		20 Steps
$imb\_factor$	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
0.01	80.73	71.12	81.42	70.05	77.10	65.64	87.70	79.96	88.82	79.68	87.87	78.74
0.05	84.21	75.50	83.77	73.97	80.38	69.22	88.58	80.58	89.03	80.18	88.63	79.18
0.1	84.70	75.98	84.10	74.57	80.73	69.35	88.92	81.04	89.32	80.50	89.28	79..82
0.5	85.71	76.87	85.66	76.44	83.00	72.41	89.23	81.28	89.60	80.68	89.86	80.60

TABLE VII : The Average and Last accuracy on CIFAR100 and ImageNet-100 benchmark under different classes imbalance ratio

imb\_factor

imb\_factor=1

means every classes have same number of samples. The best results are coloured grey.

Method	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9	Task 10
LwF [ 4 ]	89.3	70.1	54.3	45.8	39.8	36.1	31.7	28.9	24.4	23.9
iCaRL [ 6 ]	88.7	78.1	72.4	67.2	63.7	60.2	56.4	54.4	51.9	49.5
iTAML [ 51 ]	89.2	89.0	87.3	86.2	84.3	82.1	80.7	79.1	78.4	77.8
ARI [ 52 ]	88.6	86.9	85.8	84.6	83.1	81.8	81.6	81.0	80.2	80.9
CoOp(W ViT-L/14) [ 11 ]	95.8	90.7	85.2	83.4	80.8	75.8	74.7	71.7	71.3	67.6
Continual-CLIP(ViT-L/14) [ 8 ]	96.7	92.2	86.0	80.4	77.5	75.8	73.0	71.4	69.8	66.7

TABLE VIII : Last accuracy of different CIL methods on CIFAR100. The accuracy of Task

t

t\in\{1,2,\ldots,10\}

reported here is the last accuracy over all the previous tasks (i.e., Tasks

1,2,\ldots,t

). If not otherwise specified, the method uses ResNet as the backbone, where

{\dagger}

indicates the result based on the CLIP ViT-L/14 pre-trained on Laion-2B. The best results are coloured grey.

Panel of Figure 14 — Fig. 6 : Last accuracy of every Task in 10 steps. The Last accuracy of Task $t$ , $t\in\{1,2,...,10\}$ is the Top-1 accuracy over all the previous Tasks (i.e., Tasks $1,2,...,t$ ). The results in left and middle are conducted on CIFAR100. The result of ”Ours” in left is based on ViT-L/14 and in middle and right are based on ViT-B/16.

Panel of Figure 15 — Fig. 7 : Avg accuracy and Last accuracy of every Task on CIFAR100 and TinyImageNet with CLIP ViT-B/16. The Last accuracy of Task $t$ , $t\in\{1,2,...,20\}$ is the Top-1 accuracy over all the previous Tasks (i.e., Task $1,2,...,t$ ). and the Avg accuracy is the average value of Last accuracy over all the previous Tasks. The 100 classes of TinyImageNet are used as base classes.

Method	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9	Task 10
CLIP	91.60	89.60	85.30	82.60	80.48	78.13	75.17	72.65	71.30	70.08
1-3	95.60	93.55	89.77	87.50	85.70	83.48	81.49	79.35	77.77	76.47
1-6	95.80	93.20	89.90	87.45	85.84	83.87	81.94	79.66	78.04	76.65
1-9	97.00	93.95	89.87	87.40	85.64	83.30	81.17	78.91	77.28	76.00
1-12	97.10	94.35	90.37	87.90	86.10	84.02	82.00	79.49	78.00	76.70
4-6	95.20	92.90	89.43	87.00	85.50	83.48	81.61	79.34	77.63	76.26

TABLE IX : Average accuracy of different continual learning methods on CIFAR100 with CLIP ViT-B/16. For example, ”1-3” signifies that adapters are inserted in the first 3 blocks. The accuracy of Task

t

t\in\{1,2,...,10\}

reported here is the Top-1 accuracy over all the previous tasks (i.e., Tasks

1,2,...,t

). ’CLIP’ means no adapter in model.

	10 steps		20 steps		50 steps
Blocks	Avg	Last	Avg	Last	Avg	Last
CLIP	79.69	70.08	80.41	70.08	80.80	70.08
1-3	85.07	76.47	84.76	75.66	83.10	72.86
1-6	85.23	76.65	85.40	75.84	83.86	73.50
1-9	85.05	76.00	85.26	75.70	84.01	73.66
1-12	85.60	76.70	85.30	76.02	84.09	73.77

TABLE X : Effect of number and position of adapters loaded in CLIP image encoder. For example, ”1-3” signifies that adapters are inserted in the first 3 blocks. All experiments are conducted on CIFAR100 with CLIP ViT-B/16, ’CLIP’ means no adapter in model. The best results are coloured gray

	10 steps		20 steps
Bottleneck	Avg	Last	Avg	Last
1	85.94	77.61	85.33	75.98
2	86.14	77.68	85.50	76.16
4	85.84	77.01	85.56	76.44
8	85.85	77.10	85.63	76.58
16	85.92	77.14	85.64	76.6

TABLE XI : The influence of bottleneck dimension of adapters. All experiments are conducted on CIFAR100 with CLIP ViT-B/16. ”Bottleneck” means the projection dimension of adapters. Other experiment settings are the same as Appendix D .

Panel of Figure 19 — Fig. 8 : The Last accuracy of different Multi-Adapter implementations on classes 70-100 in 50 steps. Every step contains 2 classes, so numbers like ”36” means classes 70-72. Experiments are conducted on CIFAR100 with CLIP ViT-B/16 and the classes are tested with the same sequence.

				10 steps		20 steps		50 steps
Adapt-MLP	Adapt-Atten	Adapt-All	Para(M)	Avg	Last	Avg	Last	Avg	Last
✗	✗	✗	0	79.69	70.08	80.41	70.08	80.80	70.08
✔	✗	✗	1.19	85.31	76.72	85.47	76.20	83.93	73.59
✗	✔	✗	1.19	85.94	77.61	85.33	75.98	83.80	73.50
✗	✗	✔	1.19	85.48	76.19	85.48	76.19	83.93	73.60
✔	✔	✗	2.38	85.98	77.41	85.67	76.42	84.60	74.54

TABLE XII : The results of different implementations of Multi-Adapter with the bottleneck dimension being 1 . The structures of Adapter-MLP, Adapter-Atten and Adapt-All are shown in Fig. 2 . ”Para” refers to trainable parameters, with ”M” standing for million. All experiments are conducted on CIFAR100 and the best results are coloured grey

Incremental Learning

Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.

See all 146 papers

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learning
Apr 20, 2026Tianqi Wang, Jingcai GuoClass-Incremental Learning VLM Adaptation
Department of COMP The Hong Kong Polytechnic University Hong Kong SAR · Department of COMP/LSGI The Hong Kong Polytechnic University Hong Kong SAR

Nov 14, 2025J/K move · Enter open · S save

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang+2

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Class-Incremental Learning VLM Adaptation Incremental Learning

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.

Abstract

Incremental Learning

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learning