TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Chang Sun; Francesco Barbato; Matteo Caligiuri; Pietro Zanuttigh

cs.CVAug 21, 2026

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Authors: Chang Sun, Francesco Barbato, Matteo Caligiuri, Pietro Zanuttigh

Organizations: University of Padova, Padova(PD), Italy

Abstract

Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks. However, during continual adaptation, both catastrophic forgetting and zero-shot degradation occur, severely degrading performance. In this paper, we introduce TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity. We achieve this with two complementary techniques: subspace learning and geometry-aware knowledge distillation. Specifically, we first learn a sequence of task-specific low-rank projectors, which we use to project the latent representations before optimizing cross-entropy. Secondly, we employ a geodesic-distance-based loss that distills knowledge from the previous-task model while effectively preserving the latent space geometry. These design choices not only avoid unnecessary parameter updates along the full embedding dimensions but also improve learning by focusing on task-specific manifolds. Moreover, the geometry-aware distillation provides strong regularization and significantly reduces both catastrophic forgetting and zero-shot degradation throughout the continual learning sequence. Experimental results with the CLIP vision language model in the multi-domain task incremental and class incremental learning benchmarks demonstrate clear improvements over state-of-the-art methods in mitigating forgetting and preserving zero-shot capabilities.

Figures & tables

Figure 1: TASSO architecture. At each incremental step $k$ , the vision encoder $g^{k}_{i}$ is optimized with a combination of three objectives: cross-entropy; task-specific subspace cross-entropy (computed according to the learnable projector $U$ ); and geometry-aware knowledge distillation (retaining knowledge from the previous step’s encoder $g_{i}^{k-1}$ ).

Figure 2: Schematic overview of how the metrics are computed. Accuracy denotes the mean performance over all tasks once Continual Learning has finished. Catastrophic Forgetting quantifies the average, worst-case loss in performance on previously learned tasks, relative to supervised training on each task. Zero-Shot Degradation captures the average, worst-case drop in performance on unseen future tasks, relative to the accuracy of the pretrained model before the start of continual learning.

	Method	$\mathbf{S^{1}}$	$\mathbf{S^{2}}$	$\mathbf{S^{3}}$	$\mathbf{S^{4}}$	$\mathbf{S^{5}}$	$\mathbf{S^{6}}$	$\mathbf{S^{7}}$	$\mathbf{S^{8}}$	Mean
Accuracy ( $\rightarrow$ )	Continual FT	76.16	76.24	78.03	68.69	76.64	75.44	72.71	77.45	75.17
	LwF [ 23 ]	76.78	80.45	80.65	77.52	79.64	79.45	77.31	78.70	78.81
	iCaRL [ 34 ]	77.99	79.77	79.93	76.66	79.26	79.08	77.06	78.61	78.55
	ZSCL [ 43 ]	81.89	83.98	84.30	83.49	83.41	82.38	81.92	81.97	82.92
	MoE-Adapters [ 40 ]	82.71	80.74	81.15	83.97	83.68	83.68	82.73	79.68	82.29
	GIFT [ 39 ]	81.93	84.01	84.10	83.67	84.11	84.25	82.54	83.10	83.46

Table 1: Task-Specific evaluation (MTIL task). During inference, each image is compared only against the class embeddings of its task. We report percent Accuracy (

\uparrow

), Catastrophic Forgetting (

\downarrow

), and Zero-Shot Degradation (Z. S. Deg.,

\downarrow

). Best in bold.

	Method	$\mathbf{S^{1}}$	$\mathbf{S^{2}}$	$\mathbf{S^{3}}$	$\mathbf{S^{4}}$	$\mathbf{S^{5}}$	$\mathbf{S^{6}}$	$\mathbf{S^{7}}$	$\mathbf{S^{8}}$	Mean
Accuracy ( $\rightarrow$ )	Continual FT	75.17	75.13	76.01	67.17	75.54	74.47	71.66	76.40	73.94
	LwF [ 23 ]	74.14	77.44	77.94	74.89	77.30	77.43	75.50	76.51	76.39
	iCaRL [ 34 ]	76.97	78.82	78.57	75.43	78.08	78.10	75.70	77.52	77.40
	ZSCL [ 43 ]	80.49	82.54	82.99	82.08	82.17	80.99	80.30	80.09	81.46
	GIFT [ 39 ]	79.99	81.68	81.95	83.08	83.48	83.27	81.58	81.27	82.04
	SnD [ 41 ]	83.35	83.57	83.88	83.70	84.46	83.82	83.89	83.43	83.76

Table 2: Task-Agnostic evaluation (MCIL task). During inference, each image is compared against all 694 textual class embeddings. We report percent Accuracy (

\uparrow

), Catastrophic Forgetting (

\downarrow

), and Zero-Shot Degradation (Z. S. Deg.,

\downarrow

). Best in bold.

Panel of Figure 3 — Figure 3: Distribution of cosine distances between text anchors and vision class-prototypes. TASSO (in blue ) consistently achieves lower distances than SnD (the closest competitor, in orange ). Metrics computed in the last continual step of $\mathbf{S^{1}}$ ordering.

Figure 4: Stacked boxplot comparing the cosine distances between image prototypes and textual class embeddings (lower is better). TASSO (in blue ) and SnD (the closest competitor, in orange ). Metrics computed in the last continual step of $\mathbf{S^{1}}$ ordering.

$\mathcal{L}_{KD}$	Subspace	Accuracy	Forgetting	Zero-Shot
$\mathcal{L}_{KD}$	Learning	Accuracy	Forgetting	Degradation
✗	✗	76.86	9.82	11.63
$L2$	✗	84.48	1.71	1.75
$\mathcal{L}_{\text{geo}}$	✗	85.07	1.00	0.25
✗	✓	75.24	13.01	17.60
$L2$	✓	84.73	2.26	1.86

Table 3: Ablation study on the method components in the MTIL setting

$r$	Acc.	Forg.	Z.Shot
72	85.70	0.85	0.41
144	85.74	0.81	0.36
288	85.72	0.85	0.38

Table 4: Ablation study on the method hyperparameters in the MTIL sequence

\mathbf{S^{1}}

setting: a) the rank

r

of the projector matrix

U

; b) subspace learning loss scale

\alpha

; c) knowledge distillation loss weight

\beta

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$\mathcal{L}_{KD}$	Subspace	Accuracy	Forgetting	Zero-Shot
$\mathcal{L}_{KD}$	Learning	Accuracy	Forgetting	Degradation
✗	✗	75.70	9.99	13.73
$L2$	✗	83.27	1.85	1.72
$\mathcal{L}_{\text{geo}}$	✗	83.83	1.05	0.39
✗	✓	74.33	13.73	19.60
$L2$	✓	83.83	2.77	2.79

Appendix

Table 5: Ablation study on the method components in the MCIL setting.

Table 6: Ablation study on the hyperparameters. MCIL sequence

\mathbf{S^{1}}

	Accuracy	Forgetting	Z.S. Degradation
TASSO	85.74	0.81	0.36
TASSO(no $\mathcal{L}_{sub}$ sup.)	84.88	0.79	0.13
TASSO (proj-dec)	85.37	1.21	0.44
TASSO (alt)	85.51	0.70	0.20

Appendix

Table 7: Different optimization choices for the subspace learning loss

\mathcal{L}_{sub}

Panel of Table 8 — Table 8: Subspace distillation ablation

	Dataset
	FGVC-Aircraft	DTD	EuroSAT	Flowers-102	Food-101	Oxford-Pets	Stanford-Cars	UCF-101
Original VLM	23.91	44.39	42.22	67.40	83.69	87.27	65.51	64.26
FGVC-Aircraft	53.74	44.56	42.74	66.95	83.58	86.92	65.32	64.10
DTD	53.98	80.14	45.65	66.83	83.39	87.30	64.97	64.21
EuroSAT	53.86	80.08	98.80	67.15	83.31	86.97	64.83	64.00
Flowers-102	53.95	80.26	98.72	99.07	83.25	87.05	64.71	63.89

Appendix

Table 9: Accuracy for each domain at each incremental training step on sequence

S^{1}

in the MTIL benchmark.

	Dataset
	FGVC-Aircraft	DTD	EuroSAT	Flowers-102	Food-101	Oxford-Pets	Stanford-Cars	UCF-101
Original VLM	23.91	36.52	32.80	67.40	83.15	87.05	65.51	63.68
FGVC-Aircraft	52.84	36.94	33.05	66.91	83.06	86.70	65.32	63.55
DTD	52.69	78.31	34.30	66.42	82.40	87.11	64.97	63.63
EuroSAT	52.51	78.07	98.31	66.71	82.32	86.84	64.83	63.49
Flowers-102	52.72	77.96	98.17	98.94	82.26	86.86	64.71	63.34

Appendix

Table 10: Accuracy for each domain at each incremental training step on sequence

S^{1}

in the MCIL benchmark.

Seed	Accuracy	Forgetting	Z.S. Degradation
$42$	85.73	0.89	0.45
$123$	86.03	0.65	0.74
$1234$	85.62	0.73	0.43
$12345$	85.39	1.21	0.79
$123456$	85.79	0.61	0.50
$1234567$	85.87	0.63	0.85

Appendix

Table 11: Statistical significance of TASSO on

\mathbf{S^{1}}

of MTIL. For fairness, we used seed 1102 to match the one used by the main competitor SnD [ 41 ] .

Dataset	Classes	Train	Test
FGVC-Aircraft [ 29 ]	100	3,334	3,333
DTD [ 6 ]	47	2,820	1,692
EuroSAT [ 10 ]	10	13,500	8,100
Flowers-102 [ 31 ]	102	4,093	2,463
Food-101 [ 3 ]	101	50,500	30,300
Oxford-Pets [ 32 ]	37	2,944	3,669

Appendix

Table 12: Number of classes and samples in the training and test splits of the dataset used in TASSO. (*): Note that only a subset of the 1.28M training samples from ImageNet has been used.

Sequence	$\mathcal{T}^{1}$	$\mathcal{T}^{2}$	$\mathcal{T}^{3}$	$\mathcal{T}^{4}$	$\mathcal{T}^{5}$	$\mathcal{T}^{6}$	$\mathcal{T}^{7}$	$\mathcal{T}^{8}$
$\mathbf{S^{1}}$	FVGC	DTD	EuroSAT	Flowers	Food	Oxford	Stanford	UCF
$\mathbf{S^{1}}$	Aircraft			102	101	Pets	Cars	101
$\mathbf{S^{2}}$	DTD	EuroSAT	Flowers	Food	Oxford	Stanford	UCF	FVGC
$\mathbf{S^{2}}$			102	101	Pets	Cars	101	Aircraft
$\mathbf{S^{3}}$	EuroSAT	Flowers	Food	Oxford	Stanford	UCF	FVGC	DTD
$\mathbf{S^{3}}$		102	101	Pets	Cars	101	Aircraft	DTD

Appendix

Table 13: Details on the 8 sequences used in the TASSO experimental evaluation.

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 81 papers

Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models
May 8, 2026Mengxin Qin, Xiang Zhang, Kun Wei +2Class-Incremental Learning Vision-Language Models
School of Electronic Engineering, Xidian University Xi’an 710071, China
Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
Mar 12, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +4Continual Learning Continual Learning for VLMs
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Oct 16, 2026Seungyoon Woo, Keonhee Park, Gunhee KimContinual Learning for VLMs VLM Distillation
Seoul National University

May 8, 2026J/K move · Enter open · S save

Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models

Mengxin Qin, Xiang Zhang, Kun Wei, Xu Yang+1

School of Electronic Engineering, Xidian University Xi’an 710071, China

Class-Incremental Learning Vision-Language Models VLM Adaptation Continual Learning Incremental Learning Continual Learning for VLMs

Open paper AskPDF arXiv

Class-incremental learning aims to continuously acquire new knowledge while preserving previously learned information, thereby mitigating catastrophic forgetting. Existing methods primarily restrict parameter updates but often overlook their structural properties in high-dimensional spaces. From a subspace perspective, updates induced by different tasks tend to lie in multiple overlapping low-rank subspaces, leading to cross-task subspace interference and severe forgetting. To address this issue, we propose HDSD, a Hierarchical Dual-Subspace Decoupling framework for continual learning in vision-language models. Specifically, we introduce a lightweight Feature Modulation Module (FMM) that explicitly decomposes the parameter space into general and task-specific subspaces. Building on this design, we develop two complementary components. First, a General Fusion Module (GFM) evaluates relative parameter changes across tasks and uses an adaptive threshold to capture stable and transferable knowledge. Second, a Hierarchical Learning Module (HLM) performs structured parameter decomposition via Singular Value Decomposition (SVD) and uses a scaling mechanism to constrain updates within distinct subspace scales. Together, these designs reduce subspace interference and parameter drift. Extensive experiments on conventional benchmarks show that HDSD achieves state-of-the-art results.

Abstract

Continual Learning for VLMs

Explore similar work

Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models

Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning