Is the Modality Gap a Bug or a Feature? A Robustness Perspective

Rhea Chowers; Oshri Naparstek; Udi Barzelay; Yair Weiss

cs.CVMar 30, 2026

Is the Modality Gap a Bug or a Feature? A Robustness Perspective

Authors: Rhea Chowers, Oshri Naparstek, Udi Barzelay, Yair Weiss

Organizations: Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel

Abstract

Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Projections of CLIPs embedding of the MS-COCO validation set onto its first 3 principal components. A clear separation between images and texts is evident. Right: An image from the Imagenet [ 1 ] validation set that’s misclassified by CLIP when changing the caption template. Multi-modal models can lose more than $6\%$ of their accuracy when replacing the caption template.

Panel of Figure 2 — Figure 2: Is the modality gap a bug or a feature? Changing the gap by moving the text embeddings by $\alpha\cdot\vec{g}$ has an inconsistent effect on downstream performance. The figure follows [ 11 ] and shows that for some datasets, models benefit from slightly enlarging the gap ( 2a ), some from maintaining it ( 2b ) .

Figure 3: Three points in each modality in

\mathbb{R}^{2}

and the corresponding multi-modal contrastive loss along with the magnitude of the gradient of the loss. Lines connect true pairs. As long as the points satisfy relative alignment - the true pair of any point is also its nearest neighbor - the loss and the gradient magnitude are close to zero, even when there exists a gap.

Figure 4: The evolution of embeddings using gradient descent on the contrastive loss ( Eq. 6 ) starting with two tight clusters. Both when using an unnormalized embedding space (top) and when constricting the embeddings to the sphere (bottom) they converge to a solution that has almost zero loss and for which a global gap vector exists and the gap vector is orthogonal to both modalities. We prove that minimizing the contrastive loss will lead to such a solution under certain assumptions. For full training details see Appendix H .

Panel of Figure 5 — Figure 5: (Top:) Two initial embeddings where the bottom embedding is color-coded based on $S^{y}_{i}$ . $S^{y}_{i}$ decreases with distance to the other modality. ( Bottom: ) The training dynamics of a toy model initialized with isotropic Gaussians. Training starts by shrinking variance in the direction of the gap according to Theorem 3.1 .

Figure 6: Empirical evidence for the assumptions of our thoeorems on training dynamics Left : The variance of

S_{i}^{x},S_{i}^{y}

in log scale throughout training of Fig. 5 . The values of

S_{i}^{x}

and

S_{i}^{y}

approach their means as training progresses, which is

1

by definition, meaning that the matrices

Q^{x},Q^{y}

are approximately doubly-stochastic. Right : KDE plot of the gap’s norm and the distances between the embeddings and their means calculated over 50 random initializations of CLIP (ViT-B-32) on 5000 randomly sampled images and texts from the Flickr30k dataset [ 10 ] . The within modality distance is much smaller than the distance between the modalities.

Figure 7: Illustration of the relationship between robustness and the modality gap. Two image embeddings (blue squares) are classified identically when using the black texts. However, once slight noise is added to the texts (red), only the image with a smaller "gap" will maintain its classification.

Panel of Figure 8 — Figure 8: The zero shot classification accuracy, multiple choice VQA accuracy and robustness under noise $\eta\sim\mathcal{N}(0,\sigma^{2}I)$ , on various models and datasets. As predicted by our theory, robustness monotonically increases when the gap is closed while accuracy is maintained.

Panel of Figure 9 — Figure 9: CLIP variants are increasingly more robust to quantization as the gap is closed. The gap norm is measured in the non-quantized space.

Panel of Figure 10 — Figure 10: We apply rephrasing noise to textual inputs on the A-OKVQA dataset, and measure average accuracy and rephrase accuracy in the noisy setting over $K=500$ samples of wrong and correct answers for each question. Models are increasingly more accurate once the gap is closed, without change in clean accuracy, even though this setting is far from our theoretical assumptions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure 11: (Top: Two set of points that perfectly satisfy dimensionality collapse. The bottom points are color-coded by $S_{i}^{y}$ . Note that $S_{i}>1$ for points near the center. (bottom: The training dynamics. Despite no variance in the direction of the gap for both modalities, the solution converged to has no gap and is perfectly aligned. This is because in the initial iterations, points near the center are pushed away from the other modality hence destroying the original low dimensionality of each modality.

Figure 12: We calculate

S_{i}^{x},S_{i}^{y}

throughout the training of

N=500

embedding pairs initialized from a Gaussian distribution with variance

\sigma^{2}=0.01

and gap of

\left\lVert\vec{g}\right\rVert=1.8

. As training progresses, the values of

S_{i}^{x},S_{i}^{y}

are concentrated around their means, which by definition equal

1

, making the matrices

Q^{x},Q^{y}

more doubly-stochastic.

Panel of Figure 13 — Figure 13: The orthogonality assumption - the cosine of angle $\theta$ between $\vec{g}$ and each embedding in each modality is nearly $0$ for different models and datasets, confirming assumption 3.3 .

Figure 14: Results for different noise distributions and models on CIFAR10. All are normalized to have variance

\sigma^{2}=0.025^{2}

. Robustness increases when the gap is closed.

Figure 15: We compute

d(C)

according to Eq. 50 . For all models we tested, with over 400 different captions,

d(C)\approx 1

suggesting the noise is extremely correlated in the embedding space.

Panel of Figure 16 — Figure 16: We follow Shi et al. [27] and learn embeddings $\mathcal{X},\mathcal{Y}\in\mathbb{R}^{1000\times 512}$ on the hypersphere using the contrastive loss ( Eq. 6 ) and gradient descent. While the loss decreases and training converges (left), a major gap remains between the embeddings and orthogonality holds (right, measured using 9 ).

Panel of Figure 17 — Figure 17: Top: When training with $\tau=1$ training converges to a solution without a gap, despite existence of an initial gap and orthogonality. Middle: This is consistent for training in higher dimensions as well - different temperatures have different effects on how much of the gap is closed. When temperatures are $\geq 1$ , the gap closes throughout training. In higher temperatures training hardly differs from initialization. When the gap is initialized at zero it remains so for all temperatures. Bottom: Example of dynamics in $\mathbb{R}^{256}$ with $\tau=10$ . The gap closes throughout training as it converges to perfect alignment.

Figure 18: The zero shot classification accuracy and robustness under noise

\eta\sim\mathcal{N}(0,\sigma^{2}I)

, on ImageNet and MS-COCO. Top row: When using Algorithm 1 with

\epsilon=5\%

of the variance, decrease in accuracy is negligible (

<1\%

) relative to the robustness gained, which can be

\sim 10\%

. Bottom row: As the threshold

\epsilon

grows larger, more of the gap is closed, greatly increasing robustness at a negligible cost of accuracy.

Figure 19: Even when using Algorithm 1 the drop in R@1 for SigLIP [ 40 ] on image to text retrieval on MS-COCO dataset [ 13 ] is negligible relative to the improvement in robustness for different Gaussian noises (left). Fig. 19 shows the ranges of the singular value threshold

\epsilon

for which the increment in robustness (for Gaussian noise with

\sigma^{2}=0.01

) is larger than the decrease in R@1.

Modality Gap in VLMs

Modality gaps in vision-language models are mismatches between visual and textual representations or their use in downstream tasks.

See all 59 papers

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaCross-Modal Retrieval Modality Gap in VLMs
Indian Institute of Technology Jodhpur
On the modality gap and the contrastive loss in multi-modal representation learning
Jul 12, 2026Fabian Mager, Hiba Nassar, Lars Kai HansenCross-Modal Alignment Multimodal Contrastive Learning
Department of Applied Mathematics and Computer Science Technical University of Denmark
On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings
Mar 18, 2026David Restrepo, Miguel L Martins, Chenwei Wu +5Modality Gap in VLMs Multimodal Learning
CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3

Sep 28, 2026J/K move · Enter open · S save

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Aditya Sharma, Divya Saxena

Indian Institute of Technology Jodhpur

Cross-Modal Retrieval Modality Gap in VLMs Vision-Language Alignment Contrastive Learning for VLMs

Open paper AskPDF arXiv

Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.

Abstract

Modality Gap in VLMs

Explore similar work

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

On the modality gap and the contrastive loss in multi-modal representation learning

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings