Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

Tiantian Peng; Yuyang Liu; Shuo Yang; Qiuhe Hong; YongHong Tian

cs.LGJul 26, 2025

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

Authors: Tiantian Peng, Yuyang Liu, Shuo Yang, Qiuhe Hong, YongHong Tian

Organizations: Peking University, Shenzhen Graduate School · Peng Cheng Laboratory

Abstract

The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.

Figures & tables

Fig. 1 : Modality gap between Text and Image space. Fine-tuning can damage the gap of other domains. In continual learning, unstable gap can cause performance degradation, thus we hope to maintain a stable gap to preserve the performance.

Fig. 2 : Orthogonal Gradient Projection (OGP) projects the gradient of new task (black arrow) onto the null space of previous tasks, using projected gradient (red arrow) to update without compromising the performance on previous tasks.

Fig. 3 : Framework overview of FPCD. (a) Fast Null-space Projection . The updated gradient $G^{t}_{l}$ will be projected onto the shared null space $\hat{\mathcal{N}}^{t-1}$ of previous tasks to obtain $\Delta W^{t}_{l}$ to update CLIP $\theta^{t}$ . After training task $t$ , the null space $\mathcal{N}^{t}$ will be computed to update the shared null space; (b) CE Loss and Contrastive Distillation . Cross Entropy $\mathcal{L}_{CE}$ is used to learning on new task, and Contrastive Distillation $\mathcal{L}_{CD}$ is used for matching CLIP $\theta^{t}$ and original CLIP $\theta^{0}$ on reference data.

	Dataset	Aircraft	Caltech101	CIFAR100	DTD	EuroSAT	Flowers	Food	MNIST	OxfordPet	StanfordCars	SUN397	Avg
CLIP	Zero-Shot	24.4	87.9	68.2	45.1	54.6	71.3	88.8	59.5	89.1	64.5	64.9	69.4
CLIP	Upper Bound	62.0	95.1	89.6	79.5	98.9	97.5	92.7	99.6	94.7	89.6	81.8	89.2
Transfer	LwF [ 59 ]		74.5	56.9	39.1	51.1	52.6	72.8	60.6	75.1	30.3	55.9	56.89
	iCaRL [ 60 ]		56.6	44.6	32.7	39.3	46.6	68.0	46.0	77.4	31.9	60.5	50.36
	LwF-VR [ 61 ]		77.1	61.0	40.5	45.3	54.4	74.6	47.9	76.7	36.3	58.6	57.24
	WiSE-FT [ 62 ]		73.5	55.6	35.6	41.5	47.0	68.3	53.9	69.3	26.8	51.9	52.34

TABLE I : Comparison of SOTA methods on MTIL Order I. Results of other methods are taken from published papers.

	Dataset	StanfordCars	Food	MNIST	OxfordPet	Flowers	SUN397	Aircraft	Caltech101	DTD	EuroSAT	CIFAR100	Avg
CLIP	Zero-Shot	64.5	88.8	59.5	89.1	71.3	64.9	24.4	87.9	45.1	54.6	68.2	65.4
CLIP	Upper Bound	89.6	92.7	99.6	94.7	97.5	81.8	62.0	95.1	79.5	98.9	89.6	89.2
Transfer	LwF [ 59 ]		87.8	58.5	71.9	46.6	57.3	12.8	81.4	34.5	34.5	46.8	53.21
	iCaRL [ 60 ]		86.1	51.8	67.6	50.4	57.9	11.0	72.3	31.2	32.7	48.1	50.91
	LwF-VR [ 61 ]		88.2	57.0	71.4	50.0	58.0	13.0	82.0	34.4	29.3	47.6	53.09
	WiSE-FT [ 62 ]		87.2	57.6	67.0	45.0	54.0	12.9	78.6	35.5	28.4	44.3	51.05

TABLE II : Comparison of SOTA methods on MTIL Order II. Results of other methods are taken from published papers.

Fig. 4 : Upper: The variation of modality gap when fine-tuning using different methods, with the gray area representing the fine-tuning position. All have undergone significant changes before and after fine-tuning. Bottom: The variation of accuracy and recall@1. Our method achieves the most stable modality gap and the best performance.

Fig. 5 : Comparison of retrieval task on COCO Captions and Flickr30k after training on MTIL. FPCD is closest to original CLIP. The horizontal axis represents the number of data used

Method			Transfer	Avg	Last
Zero-shot			65.4	65.3	65.3
+FNP	+CD	+FD	Transfer	Avg	Last
		✓	58.8	68.5	76.9
	✓		64.0	72.7	80.7
✓			61.3	73.4	86.5
✓		✓	61.5	73.7	87.1

TABLE III : Ablation study of FNP, CD, and FD.

Category	Setting	Transfer	Avg	Last
Projection	$\rho=0.1$	65.3	76.2	86.8
	$\rho=0.15$	65.2	76.4	87.6
	$\rho=0.2$	65.1	76.3	87.3
	Dimension = $d/2$	65.2	76.0	86.9
	$\sigma_{m}=50\sigma_{min}$ [ 28 ]	64.7	73.9	83.6
Ref. Data	ImageNet	65.2	76.4	87.6

TABLE IV : Ablation study on hyperparameter choices. Default settings are marked by underline.

Fig. 6 : Eigenvalues of gram matrix at different FFN layers of CLIP. The eigenvalues smaller than $\sigma_{m}$ are colored in orange. As $\rho$ increases, the dimension of null space increases.

Method	Params	GPU Memory	Time (s/iter)
LWF [ 59 ]	149.6M	31.4GB	1.54s/it
ZSCL [ 16 ]	149.6M	25.7GB	3.94s/it
Adapters [ 19 ]	51.1M	21.8GB	1.58s/it
GIFT $\dagger$ [ 20 ]	149.6M	34.0GB	2.37s/it
FNP-only	57M	10.8GB	0.63s/it
FNP+CD	57M	19.6GB	1.48s/it

TABLE V : Comparison of computational cost on MTIL in terms of training parameters, GPU burdens and training speed. The

\Delta

is the improvement relative to GIFT. Results marked with

\dagger

are reproduced by us; others are taken from paper [ 19 ] .

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 82 papers

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal Alignment Modality Gap in VLMs
University of Electronic Science and Technology of China · University of Bristol
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Oct 16, 2026Seungyoon Woo, Keonhee Park, Gunhee KimContinual Learning for VLMs VLM Distillation
Seoul National University
Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
Mar 12, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +4Continual Learning Continual Learning for VLMs
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China

Mar 31, 2026J/K move · Enter open · S save

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Hongyuan Liu, Qinli Yang, Wen Li, Zhong Zhang+5

University of Electronic Science and Technology of China · University of Bristol

Cross-Modal Alignment Modality Gap in VLMs VLM Adaptation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality (

R^2 = 0.986

), whereas the commonly used Raw Gap is misleading (

R^2 = 0.691

). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With

α_{\text{target}}{=}0.05

, the modality gap is reduced by 66.6% with only 4.84% accuracy drop. Under stronger alignment (

α_{\text{target}}{=}0.5

), the gap is reduced by 82.3%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.

Abstract

Continual Learning for VLMs

Explore similar work

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning

Continual Learning with Vision-Language Models via Semantic-Geometry Preservation