COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning

Seungyoon Woo; Keonhee Park; Gunhee Kim

cs.CVOct 16, 2026

COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning

Authors: Seungyoon Woo, Keonhee Park, Gunhee Kim

Organizations: Seoul National University

Abstract

Vision-language models (VLMs) such as CLIP retain zero-shot recognition ability, but their open-world use often requires task-agnostic continual learning: tasks arrive sequentially, past data are unavailable, task identity is unknown, and the backbone cannot be retrained. We argue that this setting fails VLMs through three coupled pressures beyond the stability-plasticity tradeoff: (1) modal-consistency drift, (2) representation-space interference, and (3) per-task capacity shortfall. We propose COMET (Collaborative Mixture-of-Expert Transfer) to resolve these three issues respectively as follows. A frozen teacher provides only a feature reference on a shared image-text pool; Tri-Affinity Distillation preserves image-image, text-text, and image-text geometry; and Student-Aware Cross-Expert Distillation aligns new experts with compatible prior experts so routing errors degrade gracefully. We design COMET to keep capacity and inference separate: a contextual bandit keeps or merges full per-task experts using post-training signals, while a reconstruction router selects sparse experts at test time without teacher semantics. On the X-TAIL benchmark, we show that each proposed component improves continual learning performance. COMET shows that continual vision-language adaptation can preserve multimodal geometry, coordinate accumulated experts, and allocate capacity jointly when feature reference, expert shaping, routing, and capacity control are separated by construction.

Figures & tables

Figure 1: Overview of COMET. Targeting task-agnostic continual vision–language learning without replay or full-backbone retraining. It decomposes three failure modes: modal drift, routing-induced expert interference, and capacity shortfall into three role-disjoint remedies: TAD for CLIP-geometry preservation, SACED for cross-expert compatibility, and CMAB for adaptive expert allocation.

TinyCLIP ViT-39M/16@224px
Method	Aircraft	Caltech101	DTD	EuroSAT	Flowers	Food	MNIST	Pets	Cars	Sun397	Average
Zero-shot	17.0	75.6	39.0	40.6	65.0	77.4	22.0	78.9	47.8	67.2	53.1
CLIP ViT-B/16@224px
Zero-shot	23.8	74.3	36.4	37.4	64.1	83.4	43.9	87.8	65.5	60.8	57.7
Transfer
LwF ( Li and Hoiem, 2018 )	–	64.8	29.1	7.6	49.7	72.0	33.3	64.1	29.8	53.7	44.9

Table 1: Comparison of different continual learning methods on X-TAIL full-shot for each task in terms of Transfer , Average , and Last scores (%). The best results are highlighted with bold style.

Zero-shot
Method	Cov19	KT	NCT	SEM	Avg.
TinyCLIP-ZS	48.6	5.8	17.0	22.3	23.4
CLIP-B-ZS	48.9	16.3	15.6	24.9	26.4
Transfer
CLIP-FT	–	10.3	0.0	21.5	10.6
WISE-FT	–	12.0	0.1	23.6	11.9

Table 2: Heterogeneous 4-dataset full-shot results. Transfer, Average, and Last scores (%). The best result in each metric and column is in bold .

Full-shot
TAD	SACED	Trans.	Avg.	Last
–	–	56.6	73.9	85.6
$\checkmark$	–	61.3	75.4	87.6
–	$\checkmark$	59.9	75.1	85.3
$\checkmark$	$\checkmark$	62.6	76.1	88.1
16-shot

Table 3: Ablations and inference cost. (a) TAD/SACED ablation on X-TAIL; (b) merge policy; (c) inference cost. Best results are in bold .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Ordering	Avg	Last	$M$	Forgetting
Default	76.1	88.1	8	—
EuroSAT-last	72.0	83.3	5	$2.3$
Random seed $7$	71.9	81.0	5	$2.2$
Random seed $99$	71.3	83.8	5	$2.1$
Mean $\pm$ std	$72.8{\pm}2.2$	$84.1{\pm}3.0$	$5.8{\pm}1.5$	$2.2{\pm}0.1$

Appendix

Table 4: Task-order sensitivity on X-TAIL. “Forgetting” is per-task degradation from peak accuracy, averaged over the ten tasks. Forgetting is not available for the default ordering, so its mean is over the three re-ordered runs.

Variant	$\mathcal{L}_{I}$	$\mathcal{L}_{T}$	$\mathcal{L}_{\mathrm{Cross}}$	Avg	Last
Full COMET (TAD $+$ SACED)	✓	✓	✓	76.1	88.1
Full TAD	✓	✓	✓	75.4	87.6
w/o $\mathcal{L}_{I}$	$\times$	✓	✓	60.4	63.5
w/o $\mathcal{L}_{T}$	✓	$\times$	✓	64.5	76.6
w/o $\mathcal{L}_{\mathrm{Cross}}$	✓	✓	$\times$	68.8	80.7
$\mathcal{L}_{I}$ only	✓	$\times$	$\times$	68.2	78.2

Appendix

Table 5: TAD sub-component ablation on X-TAIL. “Full TAD” uses all three losses; “Full COMET” adds SACED. SACED is disabled for the sub-component rows.

Figure 2: Per-input stability comparison between expA (cross-modal affinity only) and expB (image $+$ text $+$ cross-modal affinity). We plot the distribution of per-image cosine shift $1-\cos(\text{before},\text{after})$ on the two tasks (SUN397 and Caltech101), shown for both embeddings and features. Dashed vertical lines indicate the mean shift and dotted lines indicate the 90th-percentile shift ( $p90$ ).

Selection rule	Avg	Last
Full criterion (COMET)	76.1	88.1
Co-activation only	74.8	88.4
All prior experts	74.4	87.9
Random selection	74.9	87.7
Top-1 agreement only	74.2	87.0
Distribution similarity only	67.8	77.2

Appendix

Table 6: SACED selection-criterion ablation on X-TAIL with reference budget

k_{\mathrm{ref}}{=}3

. The full criterion combines all three signals.

Variant	$M$	Params	Trans.	Avg	Last	$\mathcal{A}/M$
COMET (CMAB)	8	551M	62.6	76.1	88.1	9.5
Always-Expand	10	667M	61.6	75.8	87.3	7.6
Fixed $\sigma{=}0.9$	9	609M	59.4	77.2	84.9	8.6
Fixed $\sigma{=}0.7$	2	203M	58.6	59.3	61.9	29.7
Fixed $\sigma{=}0.5$	1	145M	58.3	58.4	58.9	58.4
Always-Merge	1	145M	58.4	58.4	58.4	58.4

Appendix

Table 7: Extended expert-merging ablation on X-TAIL.

\mathcal{A}/M

= Average accuracy per stored expert. Fixed

\sigma

replaces CMAB with a non-adaptive rule that merges whenever the candidate similarity exceeds

\sigma

Dataset	CLIP ZS	TinyCLIP ZS	Gap	COMET	Recovery
Aircraft	22.6	16.9	$+5.7$	83.1	$>100\%$
Caltech101	93.5	93.9	$-0.4$	92.9	–
DTD	44.5	45.3	$-0.8$	78.8	–
EuroSAT	36.1	50.7	$-14.6$	98.5	–
Flowers102	65.7	67.9	$-2.2$	95.6	–
Food	84.4	78.5	$+5.9$	83.5	$84.7\%$

Appendix

Table 8: Per-dataset capacity-gap analysis on X-TAIL. All accuracies use each dataset’s own label space, unlike the generalized X-TAIL label space of Table 1 . ZS = zero-shot. Recovery

=(\text{COMET}{-}\text{TinyCLIP ZS})/(\text{CLIP ZS}{-}\text{TinyCLIP ZS}){\times}100\%

on positive-gap datasets; “–” = capacity gap is non-positive.

Cell	Count	Share
Route-OK $\wedge$ Predict-OK	66,664	$82.4\%$
Route-OK $\wedge$ Predict-Wrong	12,206	$15.1\%$
Route-Wrong $\wedge$ Predict-OK	1,086	$1.3\%$
Route-Wrong $\wedge$ Predict-Wrong	983	$1.2\%$

Appendix

Table 9: Four-cell decomposition of test predictions on X-TAIL (

80{,}939

samples). Route-OK = autoencoder selected the oracle expert; Predict-OK = expert produced the correct label.

Dataset	Domain	$C_{10}$ (L0)	$H$ (L0)	$C_{10}$ (L11)	$H$ (L11)
COVID-19	Medical	0.338	7.11	0.663	5.71
SEM	Microscopy	0.280	7.28	0.709	5.48
NCT-CRC	Histopath	0.228	7.39	0.760	5.08

Appendix

Table 10: Frozen teacher attention statistics on three OOD domains, averaged over

N{=}50

images per dataset.

C_{10}

: fraction of attention in the top decile of

196

patches (uniform

0.10

H

: Shannon entropy in bits (uniform

\log_{2}196\approx 7.61

Method	Trainable	Peak VRAM	Backbone in Memory
LwF	$\sim 150$ M	$\sim 4.1$ GB	Full CLIP (trainable)
WiSE-FT	$\sim 150$ M	$\sim 2.9$ GB	Full CLIP (trainable)
ZSCL	$\sim 150$ M $(+150$ M ref $)$	$\sim 4.1$ GB	Full CLIP $+$ frozen ref
MoE-Adapters	$2$ – $5$ M	$\sim 1.1$ GB	Frozen CLIP
LADA	$1$ – $3$ M	$\sim 0.9$ GB	Frozen CLIP
COMET (Ours)	$\sim 58$ M	$\sim 2.7$ GB	Frozen teacher

Appendix

Table 11: Per-task training cost on a single NVIDIA A6000 (

48

GB), batch size

64

1{,}000

iterations. “Trainable” counts only parameters that receive gradients.

TinyCLIP ViT-39M/16@224px
Method	Aircraft	Caltech101	DTD	EuroSAT	Flowers	Food	MNIST	Pets	Cars	Sun397	Average
Zero-shot	17.0	75.6	39.0	40.6	65.0	77.4	22.0	78.9	47.8	67.2	53.1
CLIP ViT-B/16@224px
Zero-shot	23.8	74.3	36.4	37.4	64.1	83.4	43.9	87.8	65.5	60.8	57.7
Transfer
LwF	–	64.8	29.1	7.6	49.7	72.0	35.6	64.0	29.0	53.7	45.1

Appendix

Table 12: Comparison of different continual learning methods on X-TAIL [ Xu et al., 2024 ] 16-shot for each task in terms of Transfer , Average , and Last scores (%). The best results are highlighted with bold style.

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 81 papers

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models
Jul 26, 2025Tiantian Peng, Yuyang Liu, Shuo Yang +2Continual Learning for VLMs Modality Gap in VLMs
Peking University, Shenzhen Graduate School · Peng Cheng Laboratory
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models
Aug 21, 2026Chang Sun, Francesco Barbato, Matteo Caligiuri +1Continual Learning for VLMs Catastrophic Forgetting
University of Padova, Padova(PD), Italy

Jul 26, 2025J/K move · Enter open · S save

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

Tiantian Peng, Yuyang Liu, Shuo Yang, Qiuhe Hong+1

Peking University, Shenzhen Graduate School · Peng Cheng Laboratory

Continual Learning for VLMs Modality Gap in VLMs

Open paper AskPDF arXiv

The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.

Abstract

Continual Learning for VLMs

Explore similar work

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models