Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

Chiyuan He; Zihuan Qiu; Fanman Meng; Runtong Zhang; Linfeng Xu; Qingbo Wu; Hongliang Li

cs.CVMar 12, 2026

Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

Authors: Chiyuan He, Zihuan Qiu, Fanman Meng, Runtong Zhang, Linfeng Xu, Qingbo Wu, Hongliang Li

Organizations: School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China

Abstract

Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs. Code is available at: https://github.com/chiyuan-IVIPLab/SeGP-CL.

Figures & tables

Fig. 1: Boundary vulnerability in VLM-based continual learning and our remedy. (a) Shared visual patterns near the old-new semantic interface are re-explained by new-task texts, leading to cross-modal semantic-geometry distortion and forgetting. (b) We construct adversarial anchors toward old-class semantics and perform anchor-guided cross-modal geometry distillation (ACGD) to constrain the drift in this vulnerable region.

Fig. 2: Systematic evidence of boundary vulnerability. We report JSD-based cross-modal distributional drift with respect to the relative distance

d_{\mathrm{old}}

from the ground-truth old-class text embeddings, where

d_{\mathrm{old}}

is formally defined in Sec. IV-H . Larger

d_{\mathrm{old}}

indicates that an old sample moves from the old-class semantic core toward the semantic boundary. Across different datasets, VLM backbones, and representative cross-modal-regularization CL methods, boundary regions consistently suffer larger drift, while our method reduces the drift in high-

d_{\mathrm{old}}

regions.

Fig. 3: Overview of proposed SeGP-CL . 1) Anchor construction: Dual-targeted projected gradient descent (DPGD) iteratively perturbs seed samples to synthesize adversarial anchors that are simultaneously guided in raw visual space and CLIP feature space. 2) Continual learning: A LoRA-tuned VLM is optimized on task batches with CE loss, while anchor batches and history texts impose semantic-geometry preservation via ACGD and TSGR.

Fig. 4: Post-Training: We estimate raw-space drift and transfer visual prototypes for old classes, then perform dual-path inference by ensembling logits from visual and CLIP branches.

Method	Publication	Replay	CIFAR100 (10)		ImageNet-R (10)		ImageNet-Sub (10)		CUB-200 (10)		UCF (10)
Method	Publication	Replay	Avg	Last	Avg	Last	Avg	Last	Avg	Last	Avg	Last
SLCA [ 15 ]	ICCV’23		80.5	67.6	75.9	70.4	78.6	59.9	73.3	60.4	–	–
L2P++ [ 44 ]	CVPR’22		81.9	73.1	81.7	76.0	80.5	67.2	71.9	63.0	–	–
CODA Prompt [ 45 ]	CVPR’23		77.0	62.3	78.0	67.5	64.1	34.8	66.6	50.9	87.7	82.3
Aper-Adapter [ 49 ]	IJCV’24		75.8	65.5	78.7	71.4	85.8	76.4	–	–	–	–
Continual-CLIP [ 59 ]	–		–	66.7	–	72.0	–	75.4	–	51.2	–	65.2

TABLE I: Comparison with different methods using a CLIP-ViT-B/16 backbone.

Fig. 5: Comparison with state-of-the-art CL methods in terms of per-task accuracy and global transfer. All results are achieved on the same CLIP ViT-B/16 backbone.

Methods ( On’l’y )	CIFAR100			ImageNet-R
Methods ( On’l’y )	FWT $\uparrow$	BWT $\uparrow$	F $\downarrow$	FWT $\uparrow$	BWT $\uparrow$	F $\downarrow$
ENGINE	0	-10.9	7.9	0	-7.0	7.1
RAPF	0	-8.9	8.5	0	-6.1	6.6
DesCLIP	68.7	-2.1	6.5	74.9	-1.0	1.7
MG-CLIP	70.2	-3.9	4.9	76.2	-0.01	0.88
SeGP-CL	72.3	-0.43	0.9	77.0	+0.03	0.32

TABLE II: Comparison on forward transfer (FWT), backward transfer (BWT) and forgetting (F) of the global visual-text matching.

Constraint	Distillation Source	Sample Budget	$\Delta\overline{A}_{\mathrm{new}}\uparrow$	$\Delta$ Last $\uparrow$
Naive baseline	–	–	(89.8)	(77.0)
Distillation on full new-task data
SGCL [ 26 ]	New-task data	$N_{t}$	-3.2	+2.5
CGD	New-task data	$N_{t}$	-2.8	+1.5
Matched small-budget distillation sources
SGCL [ 26 ]	Random new-task data	$N_{t}\rightarrow M_{t}$	-3.1 ${\scriptstyle\pm 0.3}$	+2.4 ${\scriptstyle\pm 0.2}$

TABLE III: Comparison of distillation schemes on CIFAR100 under different data-source and sample-budget settings. For the Naive baseline, values in parentheses denote the absolute

\overline{A}_{\mathrm{new}}

and Last ; other rows report changes relative to this baseline.

Components				CIFAR100		UCF
w/ $~\mathcal{L}_{\mathrm{ACGD}}$	w/ $~\mathcal{L}_{\mathrm{GR}}$	w/ PT	w/ V.	Last	F $\downarrow$	Last	F $\downarrow$
–	–	–	–	77.0	10.9	84.4	10.7
✓				81.7	7.0	88.2	6.6
✓	✓			82.8	5.9	88.0	6.9
✓	✓	✓		83.2	5.3	88.9	6.3
✓	✓	✓	✓	84.6	5.7	92.8	6.8

TABLE IV: Ablation on key components. ‘PT’ denotes using Prototype Transfer and ‘V.’ denotes using an additional visual branch for prediction.

Dataset	$\lambda_{p}=0$ w/o $\mathcal{L}_{\mathrm{v\text{-}adv}}$	0.2	0.3	0.4	0.5	0.6	0.7
CIFAR100	82.7	83.8	84.2	84.6	84.6	84.4	84.2
CUB-200	78.1	79.6	79.9	79.9	80.1	80.2	79.8

TABLE V: Sensitivity analysis of the visual-target correction weight

\lambda_{p}

in DPGD. The reported metric is Last accuracy.

Selection	$K_{\mathrm{seed}}$
Selection	Baseline (0)	1	3	5	10	20
Random	80.1	81.4 $\pm 1.7$	81.6 $\pm 1.2$	82.0 $\pm 0.8$	81.7 $\pm 0.6$	82.2 $\pm 0.5$
$Q(\mathbf{x},c)$	80.1	82.1	83.6	84.8	85.2	85.4

TABLE VI: Sensitivity analysis of the number of seed samples

K_{\mathrm{seed}}

on ImageNet-R. The reported metric is Last accuracy.

Fig. 6: Effect of adversarial iterations

K_{\mathrm{adv}}

. We visualize the distance evolution of anchor construction across DPGD iterations in (a) CLIP embedding space and (b) raw visual feature space, and report the corresponding continual learning performance in (c).

Fig. 7: Ablation of temperature

\tau_{A}

and

\tau_{T}

Fig. 8: Ablation of

\lambda_{\mathrm{ACGD}}

\lambda_{\mathrm{GR}}

, anchor batch size, and TSGR subgraph size.

Fig. 9: Evaluation of cross-modal transfer robustness on CIFAR100. We annotate the gains in Last accuracy and Forgetting relative to the baselines.

Method	CIFAR- Last	Food101	Oxford-Pets	ImageNet-full(1K)
zero-shot CLIP	66.7	85.1	87.6	64.4
PROOF	76.3	9.8	22.8	11.2
RAPF	79.0	17.2	28.6	15.2
MoE-Adapter	78.4	82.9	84.1	66.0
SPU	77.0	84.3	85.9	62.4
MG-CLIP	80.6	85.7	88.2	67.3

TABLE VII: Cross-scenario robustness evaluation.

Setting		CIFAR- Last	#Trainable Params	Avg Time (ms) / Iter.
Ours (rank=8)	K,V	83.1	0.25M	330 +53
	FFN	82.3	0.61M	401 +69
	K,V+FFN	83.4	0.86M	410 +74
Ours (rank=32)	K,V	83.9	0.98M	339 +55
	FFN	83.1	2.46M	419 +68
	K,V+FFN	84.6	3.44M	424 +79

TABLE VIII: Comparison on trainable parameters and time cost. K,V represents the key/value attention projection weights. Underlined values indicate the baseline average time.

Fig. 10: CE-loss curves for new-task plasticity analysis.

Fig. 11: DPGD optimization trajectories of adversarial anchors in visual-textual (CLIP) space and raw visual space.

Fig. 12: Effect of perturbation budget on adversarial anchors. We visualize anchors generated with increasing $\ell_{\infty}$ radius $\epsilon$ , and report their similarities to the targeted old text embedding and raw visual prototype, together with the resulting ImageNet-R CL performance.

Fig. 13: Visualization of boundary semantic drift after task learning on CIFAR-100.

Method	CIFAR100				ImageNet-R
	5 Tasks		10 Tasks		5 Tasks		10 Tasks
	Avg	Last	Avg	Last	Avg	Last	Avg	Last
Continual-CLIP [ 59 ]	82.5	81.8	87.9	86.9	89.3	88.9	84.5	84.4
RAPF [ 13 ]	92.5	92.2	95.2	94.8	91.0	91.2	94.1	93.9
DesCLIP [ 17 ]	89.3	89.1	93.0	92.5	90.4	90.5	93.5	93.1
ENGINE [ 12 ]	90.2	90.0	93.8	93.1	89.8	88.3	93.1	91.4

TABLE IX: Results under the standard TIL setting. The task identity is given at inference, and prediction is performed within the corresponding task label space. We report Avg and Last accuracy under 5-task and 10-task splits.

Method	Aux. Source	Order-I			Order-II
Method	Aux. Source	Transfer	Avg	Last	Transfer	Avg	Last
Methods without auxiliary data
Zero-shot	None	69.4	65.3	65.3	65.4	65.3	65.3
Finetune	None	44.6	55.9	77.3	46.6	56.2	67.4
Finetune + $l_{2}$ [ 22 ]	None	61.0	62.7	75.9	60.6	68.8	77.2
LwF [ 42 ]	None	56.9	64.7	74.6	53.2	62.2	71.9

TABLE X: Comparison on the MTIL benchmark under Order-I and Order-II [ 22 ] .

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models
Aug 21, 2026Chang Sun, Francesco Barbato, Matteo Caligiuri +1Continual Learning for VLMs Catastrophic Forgetting
University of Padova, Padova(PD), Italy
Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models
Jul 26, 2025Tiantian Peng, Yuyang Liu, Shuo Yang +2Continual Learning for VLMs Modality Gap in VLMs
Peking University, Shenzhen Graduate School · Peng Cheng Laboratory
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Oct 16, 2026Seungyoon Woo, Keonhee Park, Gunhee KimContinual Learning for VLMs VLM Distillation
Seoul National University

Aug 21, 2026J/K move · Enter open · S save

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Chang Sun, Francesco Barbato, Matteo Caligiuri, Pietro Zanuttigh

University of Padova, Padova(PD), Italy

Continual Learning for VLMs Catastrophic Forgetting VLM Distillation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks. However, during continual adaptation, both catastrophic forgetting and zero-shot degradation occur, severely degrading performance. In this paper, we introduce TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity. We achieve this with two complementary techniques: subspace learning and geometry-aware knowledge distillation. Specifically, we first learn a sequence of task-specific low-rank projectors, which we use to project the latent representations before optimizing cross-entropy. Secondly, we employ a geodesic-distance-based loss that distills knowledge from the previous-task model while effectively preserving the latent space geometry. These design choices not only avoid unnecessary parameter updates along the full embedding dimensions but also improve learning by focusing on task-specific manifolds. Moreover, the geometry-aware distillation provides strong regularization and significantly reduces both catastrophic forgetting and zero-shot degradation throughout the continual learning sequence. Experimental results with the CLIP vision language model in the multi-domain task incremental and class incremental learning benchmarks demonstrate clear improvements over state-of-the-art methods in mitigating forgetting and preserving zero-shot capabilities.

Abstract

Continual Learning

Explore similar work

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning