Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs. Code is available at: https://github.com/chiyuan-IVIPLab/SeGP-CL.
Figures & tables
Fig. 1: Boundary vulnerability in VLM-based continual learning and our remedy. (a) Shared visual patterns near the old-new semantic interface are re-explained by new-task texts, leading to cross-modal semantic-geometry distortion and forgetting. (b) We construct adversarial anchors toward old-class semantics and perform anchor-guided cross-modal geometry distillation (ACGD) to constrain the drift in this vulnerable region.
Fig. 2: Systematic evidence of boundary vulnerability. We report JSD-based cross-modal distributional drift with respect to the relative distance dold from the ground-truth old-class text embeddings, where dold is formally defined in Sec. IV-H . Larger dold indicates that an old sample moves from the old-class semantic core toward the semantic boundary. Across different datasets, VLM backbones, and representative cross-modal-regularization CL methods, boundary regions consistently suffer larger drift, while our method reduces the drift in high- dold regions.
Fig. 3: Overview of proposed SeGP-CL . 1) Anchor construction: Dual-targeted projected gradient descent (DPGD) iteratively perturbs seed samples to synthesize adversarial anchors that are simultaneously guided in raw visual space and CLIP feature space. 2) Continual learning: A LoRA-tuned VLM is optimized on task batches with CE loss, while anchor batches and history texts impose semantic-geometry preservation via ACGD and TSGR.
Fig. 4: Post-Training: We estimate raw-space drift and transfer visual prototypes for old classes, then perform dual-path inference by ensembling logits from visual and CLIP branches.
Method
Publication
Replay
CIFAR100 (10)
ImageNet-R (10)
ImageNet-Sub (10)
CUB-200 (10)
UCF (10)
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
SLCA [ 15 ]
ICCV’23
80.5
67.6
75.9
70.4
78.6
59.9
73.3
60.4
–
–
L2P++ [ 44 ]
CVPR’22
81.9
73.1
81.7
76.0
80.5
67.2
71.9
63.0
–
–
CODA Prompt [ 45 ]
CVPR’23
77.0
62.3
78.0
67.5
64.1
34.8
66.6
50.9
87.7
82.3
Aper-Adapter [ 49 ]
IJCV’24
75.8
65.5
78.7
71.4
85.8
76.4
–
–
–
–
Continual-CLIP [ 59 ]
–
–
66.7
–
72.0
–
75.4
–
51.2
–
65.2
TABLE I: Comparison with different methods using a CLIP-ViT-B/16 backbone.
Fig. 5: Comparison with state-of-the-art CL methods in terms of per-task accuracy and global transfer. All results are achieved on the same CLIP ViT-B/16 backbone.
Methods ( On’l’y )
CIFAR100
ImageNet-R
FWT ↑
BWT ↑
F ↓
FWT ↑
BWT ↑
F ↓
ENGINE
0
-10.9
7.9
0
-7.0
7.1
RAPF
0
-8.9
8.5
0
-6.1
6.6
DesCLIP
68.7
-2.1
6.5
74.9
-1.0
1.7
MG-CLIP
70.2
-3.9
4.9
76.2
-0.01
0.88
SeGP-CL
72.3
-0.43
0.9
77.0
+0.03
0.32
TABLE II: Comparison on forward transfer (FWT), backward transfer (BWT) and forgetting (F) of the global visual-text matching.
Constraint
Distillation Source
Sample Budget
ΔAnew↑
Δ Last ↑
Naive baseline
–
–
(89.8)
(77.0)
Distillation on full new-task data
SGCL [ 26 ]
New-task data
Nt
-3.2
+2.5
CGD
New-task data
Nt
-2.8
+1.5
Matched small-budget distillation sources
SGCL [ 26 ]
Random new-task data
Nt→Mt
-3.1 ±0.3
+2.4 ±0.2
TABLE III: Comparison of distillation schemes on CIFAR100 under different data-source and sample-budget settings. For the Naive baseline, values in parentheses denote the absolute Anew and Last ; other rows report changes relative to this baseline.
Components
CIFAR100
UCF
w/ LACGD
w/ LGR
w/ PT
w/ V.
Last
F ↓
Last
F ↓
–
–
–
–
77.0
10.9
84.4
10.7
✓
81.7
7.0
88.2
6.6
✓
✓
82.8
5.9
88.0
6.9
✓
✓
✓
83.2
5.3
88.9
6.3
✓
✓
✓
✓
84.6
5.7
92.8
6.8
TABLE IV: Ablation on key components. ‘PT’ denotes using Prototype Transfer and ‘V.’ denotes using an additional visual branch for prediction.
Dataset
λp=0 w/o Lv-adv
0.2
0.3
0.4
0.5
0.6
0.7
CIFAR100
82.7
83.8
84.2
84.6
84.6
84.4
84.2
CUB-200
78.1
79.6
79.9
79.9
80.1
80.2
79.8
TABLE V: Sensitivity analysis of the visual-target correction weight λp in DPGD. The reported metric is Last accuracy.
Selection
Kseed
Baseline (0)
1
3
5
10
20
Random
80.1
81.4 ±1.7
81.6 ±1.2
82.0 ±0.8
81.7 ±0.6
82.2 ±0.5
Q(x,c)
80.1
82.1
83.6
84.8
85.2
85.4
TABLE VI: Sensitivity analysis of the number of seed samples Kseed on ImageNet-R. The reported metric is Last accuracy.
Fig. 6: Effect of adversarial iterations Kadv . We visualize the distance evolution of anchor construction across DPGD iterations in (a) CLIP embedding space and (b) raw visual feature space, and report the corresponding continual learning performance in (c).
Fig. 7: Ablation of temperature τA and τT .
Fig. 8: Ablation of λACGD , λGR , anchor batch size, and TSGR subgraph size.
Fig. 9: Evaluation of cross-modal transfer robustness on CIFAR100. We annotate the gains in Last accuracy and Forgetting relative to the baselines.
Method
CIFAR- Last
Food101
Oxford-Pets
ImageNet-full(1K)
zero-shot CLIP
66.7
85.1
87.6
64.4
PROOF
76.3
9.8
22.8
11.2
RAPF
79.0
17.2
28.6
15.2
MoE-Adapter
78.4
82.9
84.1
66.0
SPU
77.0
84.3
85.9
62.4
MG-CLIP
80.6
85.7
88.2
67.3
TABLE VII: Cross-scenario robustness evaluation.
Setting
CIFAR- Last
#Trainable Params
Avg Time (ms) / Iter.
Ours (rank=8)
K,V
83.1
0.25M
330 +53
FFN
82.3
0.61M
401 +69
K,V+FFN
83.4
0.86M
410 +74
Ours (rank=32)
K,V
83.9
0.98M
339 +55
FFN
83.1
2.46M
419 +68
K,V+FFN
84.6
3.44M
424 +79
TABLE VIII: Comparison on trainable parameters and time cost. K,V represents the key/value attention projection weights. Underlined values indicate the baseline average time.
Fig. 10: CE-loss curves for new-task plasticity analysis.
Fig. 11: DPGD optimization trajectories of adversarial anchors in visual-textual (CLIP) space and raw visual space.
Fig. 12: Effect of perturbation budget on adversarial anchors. We visualize anchors generated with increasing ℓ∞ radius ϵ , and report their similarities to the targeted old text embedding and raw visual prototype, together with the resulting ImageNet-R CL performance.
Fig. 13: Visualization of boundary semantic drift after task learning on CIFAR-100.
Method
CIFAR100
ImageNet-R
5 Tasks
10 Tasks
5 Tasks
10 Tasks
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Continual-CLIP [ 59 ]
82.5
81.8
87.9
86.9
89.3
88.9
84.5
84.4
RAPF [ 13 ]
92.5
92.2
95.2
94.8
91.0
91.2
94.1
93.9
DesCLIP [ 17 ]
89.3
89.1
93.0
92.5
90.4
90.5
93.5
93.1
ENGINE [ 12 ]
90.2
90.0
93.8
93.1
89.8
88.3
93.1
91.4
TABLE IX: Results under the standard TIL setting. The task identity is given at inference, and prediction is performed within the corresponding task label space. We report Avg and Last accuracy under 5-task and 10-task splits.
Method
Aux. Source
Order-I
Order-II
Transfer
Avg
Last
Transfer
Avg
Last
Methods without auxiliary data
Zero-shot
None
69.4
65.3
65.3
65.4
65.3
65.3
Finetune
None
44.6
55.9
77.3
46.6
56.2
67.4
Finetune + l2 [ 22 ]
None
61.0
62.7
75.9
60.6
68.8
77.2
LwF [ 42 ]
None
56.9
64.7
74.6
53.2
62.2
71.9
TABLE X: Comparison on the MTIL benchmark under Order-I and Order-II [ 22 ] .
Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks. However, during continual adaptation, both catastrophic forgetting and zero-shot degradation occur, severely degrading performance. In this paper, we introduce TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity. We achieve this with two complementary techniques: subspace learning and geometry-aware knowledge distillation. Specifically, we first learn a sequence of task-specific low-rank projectors, which we use to project the latent representations before optimizing cross-entropy. Secondly, we employ a geodesic-distance-based loss that distills knowledge from the previous-task model while effectively preserving the latent space geometry. These design choices not only avoid unnecessary parameter updates along the full embedding dimensions but also improve learning by focusing on task-specific manifolds. Moreover, the geometry-aware distillation provides strong regularization and significantly reduces both catastrophic forgetting and zero-shot degradation throughout the continual learning sequence. Experimental results with the CLIP vision language model in the multi-domain task incremental and class incremental learning benchmarks demonstrate clear improvements over state-of-the-art methods in mitigating forgetting and preserving zero-shot capabilities.
The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.
Tiantian Peng, Yuyang Liu, Shuo Yang +2
Peking University, Shenzhen Graduate School · Peng Cheng Laboratory
Vision-language models (VLMs) such as CLIP retain zero-shot recognition ability, but their open-world use often requires task-agnostic continual learning: tasks arrive sequentially, past data are unavailable, task identity is unknown, and the backbone cannot be retrained. We argue that this setting fails VLMs through three coupled pressures beyond the stability-plasticity tradeoff: (1) modal-consistency drift, (2) representation-space interference, and (3) per-task capacity shortfall. We propose COMET (Collaborative Mixture-of-Expert Transfer) to resolve these three issues respectively as follows. A frozen teacher provides only a feature reference on a shared image-text pool; Tri-Affinity Distillation preserves image-image, text-text, and image-text geometry; and Student-Aware Cross-Expert Distillation aligns new experts with compatible prior experts so routing errors degrade gracefully. We design COMET to keep capacity and inference separate: a contextual bandit keeps or merges full per-task experts using post-training signals, while a reconstruction router selects sparse experts at test time without teacher semantics. On the X-TAIL benchmark, we show that each proposed component improves continual learning performance. COMET shows that continual vision-language adaptation can preserve multimodal geometry, coordinate accumulated experts, and allocate capacity jointly when feature reference, expert shaping, routing, and capacity control are separated by construction.