Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.
Figures & tables
Figure 1: Left: Projections of CLIPs embedding of the MS-COCO validation set onto its first 3 principal components. A clear separation between images and texts is evident. Right: An image from the Imagenet [ 1 ] validation set that’s misclassified by CLIP when changing the caption template. Multi-modal models can lose more than 6% of their accuracy when replacing the caption template.
Figure 2: Is the modality gap a bug or a feature? Changing the gap by moving the text embeddings by α⋅g has an inconsistent effect on downstream performance. The figure follows [ 11 ] and shows that for some datasets, models benefit from slightly enlarging the gap ( 2a ), some from maintaining it ( 2b ) .
Figure 3: Three points in each modality in R2 and the corresponding multi-modal contrastive loss along with the magnitude of the gradient of the loss. Lines connect true pairs. As long as the points satisfy relative alignment - the true pair of any point is also its nearest neighbor - the loss and the gradient magnitude are close to zero, even when there exists a gap.
Figure 4: The evolution of embeddings using gradient descent on the contrastive loss ( Eq. 6 ) starting with two tight clusters. Both when using an unnormalized embedding space (top) and when constricting the embeddings to the sphere (bottom) they converge to a solution that has almost zero loss and for which a global gap vector exists and the gap vector is orthogonal to both modalities. We prove that minimizing the contrastive loss will lead to such a solution under certain assumptions. For full training details see Appendix H .
Figure 5: (Top:) Two initial embeddings where the bottom embedding is color-coded based on Siy . Siy decreases with distance to the other modality. ( Bottom: ) The training dynamics of a toy model initialized with isotropic Gaussians. Training starts by shrinking variance in the direction of the gap according to Theorem 3.1 .
Figure 6: Empirical evidence for the assumptions of our thoeorems on training dynamics Left : The variance of Six,Siy in log scale throughout training of Fig. 5 . The values of Six and Siy approach their means as training progresses, which is 1 by definition, meaning that the matrices Qx,Qy are approximately doubly-stochastic. Right : KDE plot of the gap’s norm and the distances between the embeddings and their means calculated over 50 random initializations of CLIP (ViT-B-32) on 5000 randomly sampled images and texts from the Flickr30k dataset [ 10 ] . The within modality distance is much smaller than the distance between the modalities.
Figure 7: Illustration of the relationship between robustness and the modality gap. Two image embeddings (blue squares) are classified identically when using the black texts. However, once slight noise is added to the texts (red), only the image with a smaller "gap" will maintain its classification.
Figure 8: The zero shot classification accuracy, multiple choice VQA accuracy and robustness under noise η∼N(0,σ2I) , on various models and datasets. As predicted by our theory, robustness monotonically increases when the gap is closed while accuracy is maintained.
Figure 9: CLIP variants are increasingly more robust to quantization as the gap is closed. The gap norm is measured in the non-quantized space.
Figure 10: We apply rephrasing noise to textual inputs on the A-OKVQA dataset, and measure average accuracy and rephrase accuracy in the noisy setting over K=500 samples of wrong and correct answers for each question. Models are increasingly more accurate once the gap is closed, without change in clean accuracy, even though this setting is far from our theoretical assumptions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: (Top: Two set of points that perfectly satisfy dimensionality collapse. The bottom points are color-coded by Siy . Note that Si>1 for points near the center. (bottom: The training dynamics. Despite no variance in the direction of the gap for both modalities, the solution converged to has no gap and is perfectly aligned. This is because in the initial iterations, points near the center are pushed away from the other modality hence destroying the original low dimensionality of each modality.
Figure 12: We calculate Six,Siy throughout the training of N=500 embedding pairs initialized from a Gaussian distribution with variance σ2=0.01 and gap of ∥g∥=1.8 . As training progresses, the values of Six,Siy are concentrated around their means, which by definition equal 1 , making the matrices Qx,Qy more doubly-stochastic.
Figure 13: The orthogonality assumption - the cosine of angle θ between g and each embedding in each modality is nearly 0 for different models and datasets, confirming assumption 3.3 .
Figure 14: Results for different noise distributions and models on CIFAR10. All are normalized to have variance σ2=0.0252 . Robustness increases when the gap is closed.
Figure 15: We compute d(C) according to Eq. 50 . For all models we tested, with over 400 different captions, d(C)≈1 suggesting the noise is extremely correlated in the embedding space.
Figure 16: We follow Shi et al. [27] and learn embeddings X,Y∈R1000×512 on the hypersphere using the contrastive loss ( Eq. 6 ) and gradient descent. While the loss decreases and training converges (left), a major gap remains between the embeddings and orthogonality holds (right, measured using 9 ).
Figure 17: Top: When training with τ=1 training converges to a solution without a gap, despite existence of an initial gap and orthogonality. Middle: This is consistent for training in higher dimensions as well - different temperatures have different effects on how much of the gap is closed. When temperatures are ≥1 , the gap closes throughout training. In higher temperatures training hardly differs from initialization. When the gap is initialized at zero it remains so for all temperatures. Bottom: Example of dynamics in R256 with τ=10 . The gap closes throughout training as it converges to perfect alignment.
Figure 18: The zero shot classification accuracy and robustness under noise η∼N(0,σ2I) , on ImageNet and MS-COCO. Top row: When using Algorithm 1 with ϵ=5% of the variance, decrease in accuracy is negligible ( <1% ) relative to the robustness gained, which can be ∼10% . Bottom row: As the threshold ϵ grows larger, more of the gap is closed, greatly increasing robustness at a negligible cost of accuracy.
Figure 19: Even when using Algorithm 1 the drop in R@1 for SigLIP [ 40 ] on image to text retrieval on MS-COCO dataset [ 13 ] is negligible relative to the improvement in robustness for different Gaussian noises (left). Fig. 19 shows the ranges of the singular value threshold ϵ for which the increment in robustness (for Gaussian noise with σ2=0.01 ) is larger than the decrease in R@1.
Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.
We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.
Fabian Mager, Hiba Nassar, Lars Kai Hansen
Department of Applied Mathematics and Computer Science Technical University of Denmark
Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -- particularly in medical domains -- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.
David Restrepo, Miguel L Martins, Chenwei Wu +5
CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3