Visual Branch is What You Need for CLIP-based Class-Incremental Learning

Tao Hu; Zhen-Hao Xie; Jingcai Guo; De-Chuan Zhan; Da-Wei zhou

cs.CVSep 29, 2026

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

Authors: Tao Hu, Zhen-Hao Xie, Jingcai Guo, De-Chuan Zhan, Da-Wei zhou

Organizations: School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University

Abstract

Class-Incremental Learning (CIL) requires models to recognize new classes over time without forgetting previously learned ones. With the rise of vision-language pre-training, CLIP has become a strong foundation for CIL. A common design in CLIP-based CIL is to construct textual classifier weights by encoding class-name templates with the CLIP text encoder, and then classify visual features by image-text cosine similarity. This design is appealing: since CLIP aligns images and text in a shared embedding space, textual weights appear to provide an off-the-shelf classifier for incremental classes. However, we show that this seemingly natural design is not always beneficial, as a modality gap can still separate the two modalities and make textual classifier weights deviate from visual class distributions. Empirically, under identical task-wise CIL training, initializing the cosine classifier with visual class centers yields lower loss and better incremental accuracy than using CLIP textual features. Motivated by these observations, we propose VIS, a visual-only method for CLIP-based CIL that removes the deployed textual branch and constructs the incremental classifier entirely in the visual space. To obtain stronger task-adaptive visual representations, VIS uses only base-session data to enhance CLIP's final visual representation with informative visual-layer features. Built on the enhanced visual representation, VIS employs a simple kernelized incremental least-squares SVM, whose classifier weights are solved in closed form from additive sufficient statistics. When new classes arrive, VIS accumulates their sufficient statistics and recomputes the classifier weights for all seen classes, enabling efficient incremental updates while preserving historical class knowledge. Extensive experiments show that VIS achieves state-of-the-art performance without a textual branch.

Figures & tables

Panel of Figure 1 — Figure 1: Preliminary diagnosis of textual embeddings in CLIP-based CIL. (a) t-SNE visualization shows a clear separation between visual prototypes and textual embeddings in CLIP’s shared image-text embedding space. (b) The class-level modality gap $g_{c}=1-\bm{\mu}_{c}^{\top}\bm{t}_{c}$ is positively correlated with textual-head degradation $\Delta_{c}=\mathrm{acc}^{\mathrm{NCM}}_{c}-\mathrm{acc}^{\mathrm{text}}_{c}$ . (c) Visual-prototype initialization yields lower training loss and higher seen-class accuracy than textual initialization under the same cosine classifier.

Figure 2: Illustration of Vis . The classifier is constructed entirely in the visual space: the frozen CLIP visual encoder extracts multi-layer CLS features, and the residual fusion module is trained with the adaptation loss $\mathcal{L}_{\mathrm{adapt}}$ . Residual fusion and the fixed kernel feature map then produce the kernel feature $\bm{\phi}({\bm{x}})$ for classification. In the base session, Vis initializes the LS-SVM sufficient statistics $(G_{1},Q_{1},\mathbf{s}_{1})$ ; in incremental sessions $(t>1)$ , it updates $(G_{t},Q_{t},\mathbf{s}_{t})$ with new data and recomputes the classifier weights $\tilde{W}_{t}^{\star}$ in closed form.

Method	Aircraft				CIFAR100				Cars
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10
	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$	$\bar{\mathcal{A}}$	${\mathcal{A}_{B}}$
SimpleCIL Zhou et al. (2025a)	59.24	48.09	53.05	48.09	84.15	76.63	80.20	76.63	92.04	86.85	88.96	86.85
ACIL ( Zhuang et al., 2022 )	64.99	56.68	58.48	56.71	89.41	83.73	86.56	83.72	92.91	87.48	89.79	87.48
DualPrompt Wang et al. (2022b)	44.30	25.83	46.07	33.57	81.63	72.44	80.12	72.57	76.26	62.94	76.88	67.55
CODA-Prompt Smith et al. (2023)	45.98	27.69	45.14	32.28	82.43	73.43	78.69	71.58	80.21	66.47	75.06	64.19

Table 1: Main benchmark results in terms of average accuracy

\bar{\mathcal{A}}

and final-session accuracy

\mathcal{A}_{B}

. The best results are shown in bold. All methods start from the same pre-trained CLIP backbone for fair comparison.

Panel of Figure 3 — Figure 3: Incremental performance of different methods. We report the performance gap after the last incremental stage of Vis and the runner-up method at the end of the line. More results are in the supplementary.

Panel of Figure 4 — Figure 4: Ablation study, trainable-parameter comparison, and parameter sensitivity.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

\tilde{\Phi}_{t}\leftarrow\mathrm{Lift}(\mathcal{D}_{t};\Phi_{v},\mathcal{B},\mathcal{M},R).

Appendix

Algorithm 1 Vis task-wise update.

Panel of Figure 5 — Figure 5: Additional analysis of Vis . Left: robustness to different class-order seeds. Middle: performance under different CLIP pre-trained weights. Right: training time comparison with representative baselines.

Method	Aircraft B0 Inc10		CIFAR100 B0 Inc10
Method	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$
ACIL ( Zhuang et al., 2022 )	64.99	56.68	89.41	83.73
RanPAC ( McDonnell et al., 2023 )	69.77	60.28	89.30	83.18
BOFA ( Li et al., 2026 )	70.96	60.43	86.07	79.19
ENGINE ( Zhou et al., 2025b )	69.74	58.51	86.89	79.29
Vis (Ours)	$\mathbf{75.236\pm 0.106}$	$\mathbf{66.520\pm 0.277}$	$\mathbf{90.598\pm 0.073}$	$\mathbf{85.428\pm 0.167}$

Appendix

Table 2: Random-feature robustness under different initialization seeds of

R

. Results of Vis are reported as mean

\pm

standard deviation across random-feature seeds, while the other methods are included as references under the same incremental protocols.

Backbone	Method	Aircraft		CIFAR100
Backbone	Method	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$
DINOv2 ViT-B/14	SimpleCIL ( Zhou et al., 2025a )	46.00	37.23	92.36	88.10
	RanPAC ( McDonnell et al., 2023 )	87.39	79.81	90.09	83.58
	Vis (Ours)	87.68	80.53	95.17	91.98
SigLIP ViT-B/16	SimpleCIL ( Zhou et al., 2025a )	78.18	69.76	81.16	74.34
	RanPAC ( McDonnell et al., 2023 )	76.35	66.31	86.87	79.85

Appendix

Table 3: Generalization across different pre-trained backbones under the B0 Inc10 protocol. All methods use the same backbone within each group. We report average accuracy

\bar{\mathcal{A}}

and final-session accuracy

\mathcal{A}_{B}

Method	Food		Cars		UCF		SUN
Method	B0 Inc10	B50 Inc10	B0 Inc10	B50 Inc10	B0 Inc10	B50 Inc10	B0 Inc30	B150 Inc30
SimpleCIL ( Zhou et al., 2025a )	6.22	3.76	5.11	2.51	4.86	2.31	8.12	3.51
ACIL ( Zhuang et al., 2022 )	4.97	2.94	5.48	2.27	1.17	0.74	8.41	4.52
DualPrompt ( Wang et al., 2022b )	24.28	20.77	7.35	4.71	9.26	7.95	23.54	17.63
CODA-Prompt ( Smith et al., 2023 )	25.82	22.41	6.14	4.59	9.80	9.21	22.71	17.69
RanPAC ( McDonnell et al., 2023 )	5.39	2.56	3.96	1.71	1.32	0.98	7.09	3.34

Appendix

Table 4: Forgetting measure

F_{B}

of different methods under different incremental settings. Lower

F_{B}

indicates less forgetting, and the best result in each column is highlighted in bold.

Method	Aircraft B0 Inc10		CIFAR100 B0 Inc10
Method	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{B}$
ZS-CLIP ( Radford et al., 2021 )	26.66	17.22	81.81	71.38
Vis + Text	73.18	64.69	90.56	85.43
Vis	75.21	66.46	90.64	85.26

Appendix

Table 5: Controlled evaluation of textual information. ZS-CLIP serves as a textual-classifier reference, while Vis + Text and Vis form a controlled pair differing only in the use of textual class prototypes.

Method	$\bar{\mathcal{A}}$	Total Time (min) $\downarrow$	Train Time (min) $\downarrow$	Eval Time (min) $\downarrow$	Peak GPU (GB) $\downarrow$	Infer. (ms/img) $\downarrow$
DualPrompt ( Wang et al., 2022b )	63.43	48.90	45.68	3.22	6.95	8.58
CODA-Prompt ( Smith et al., 2023 )	66.81	53.23	49.64	3.59	15.53	9.58
SimpleCIL ( Zhou et al., 2025a )	92.11	1.50	0.37	1.13	0.92	3.02
RAPF ( Huang et al., 2024 )	82.12	19.96	18.62	1.34	6.43	3.59
CLG-CBM ( Yu et al., 2025 )	93.15	32.90	31.77	1.13	1.05	3.01
PROOF ( Zhou et al., 2025c )	90.44	33.26	30.85	2.41	1.16	6.43

Appendix

Table 6: Overhead comparison on Cars B0 Inc10 . All methods start from the same pre-trained CLIP backbone and are evaluated under the same hardware environment, using a single RTX 3090 GPU per method. “Total Time” denotes the end-to-end wall-clock time from the start of continual training to the completion of the final evaluation. “Peak GPU” denotes the peak training GPU memory of the current process, and “Infer.” reports the average evaluation latency per test image aggregated over all sessions. Lower is better for all overhead metrics, while higher is better for

\bar{\mathcal{A}}

Panel of Figure 6 — Figure 6: Incremental performance of different methods on half-base setting. We report the performance gap after the last incremental stage of Vis and the runner-up method at the end of the line. All methods utilize the same CLIP pre-trained weight.

Panel of Figure 7 — Figure 7: Incremental performance of different methods on B0 setting. We report the performance gap after the last incremental stage of Vis and the runner-up method at the end of the line. All methods utilize the same CLIP pre-trained weight.

Panel of Figure 8 — Figure 8: Classifier-choice ablation under fixed kernel-induced visual features. All methods use the same enhanced representation and kernel feature map, and differ only in the final classifier. NCM uses class means, Ridge uses one-hot least-squares targets, and LS-SVM uses one-vs-all $\pm 1$ targets. Ridge and LS-SVM both outperform NCM, while LS-SVM achieves comparable or better performance than Ridge in most settings, supporting our SVM-style positive-versus-negative classifier formulation.

Figure 9: Cross-class gradient cosine distributions under textual initialization and image-prototype initialization on representative datasets. We compute pairwise cosine similarities

\cos(\nabla_{c},\nabla_{c^{\prime}})

between class-wise gradient directions. Compared with textual initialization, image-prototype initialization generally produces distributions that are more concentrated around zero, indicating more balanced class-wise optimization directions.

Figure 10: Training-loss curves under textual initialization and image-prototype initialization on the same six representative datasets as Figure 12 . Textual initialization often leads to larger loss values and stronger loss fluctuations, while image-prototype initialization produces lower or smoother loss curves. This suggests that visual prototypes provide a more stable initialization for classifier optimization.

Figure 11: Additional class-wise modality-gap analysis on nine datasets. Each point denotes one class. The x-axis is the class-level modality gap

g_{c}

between the image prototype and textual embedding, and the y-axis is the class-wise accuracy improvement

\Delta_{c}

of image-prototype initialization over textual initialization. The dashed red line shows the linear fit, and the Pearson correlation coefficient is reported in each subplot. Most datasets show a positive correlation, suggesting that classes with larger image-text gaps tend to benefit more from image-prototype initialization.

Figure 12: Seen-class accuracy curves under textual initialization and image-prototype initialization on six representative datasets. The x-axis denotes global training epochs, and the y-axis denotes the accuracy over all seen classes. Image-prototype initialization generally yields higher and more stable seen-class accuracy, indicating a more favorable optimization trajectory.

Figure 13: Additional t-SNE visualizations of image samples, image prototypes, and textual embeddings on nine datasets. Blue dots denote image samples, red triangles denote image prototypes, and orange stars denote textual embeddings. Textual embeddings often lie in regions separated from the corresponding image distributions, while image prototypes remain close to image samples. This illustrates the image-text modality gap in the shared CLIP embedding space.

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning Representation Learning
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
May 13, 2026Hao Sun, Zi-Jun Ding, Da-Wei ZhouClass-Incremental Learning Visual Embeddings
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Sep 29, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +5Class-Incremental Learning Image-Text Alignment
University of Electronic Science and Technology of China, Chengdu, China · Qiyuan Lab, Beijing, China

Nov 14, 2025, cs.CVJ/K move · Enter open · S save

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang+2

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Class-Incremental Learning Representation Learning Cross-Layer Interactions

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.

Abstract

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning