On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings
Authors: David Restrepo, Miguel L Martins, Chenwei Wu, Luis Filipe Nakayama, Diego M Lopez, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante
Organizations: CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA · Federal University of São Paulo, Brazil · Universidad del Cauca, Colombia · Universidad de Buenos Aires, Argentina
Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -- particularly in medical domains -- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.
Figures & tables
Figure 1: Overview of the proposed framework. (A) Embedding alignment pipeline: raw image and text features are extracted, shifted by a hyperparameter λ to modulate the modality gap, and evaluated via linear probing. (B) Overview of medical datasets and corresponding multimodal tasks used in this study.
Figure 2: Low-dimensional visualization of image and text embeddings across datasets and backbones. Projections are shown using the first two principal components in three-dimensional ambient space.
Text Embeddings ( R )
Image Embeddings ( R )
Dataset
CLIP
SigLIP
BioMed
MedSig
CLIP
SigLIP
BioMed
MedSig
Medical Datasets
MIMIC-CXR
0.915
0.862
0.899
0.726
0.970
0.974
0.910
0.735
HAM10000
0.919
0.809
0.889
0.970
0.920
0.933
0.941
0.858
BRSET
0.985
0.956
0.990
0.980
0.986
0.980
0.964
0.731
mBRSET
0.990
0.973
0.975
0.957
0.986
0.982
0.958
0.770
Table 1: Mean Resultant Length ( R ), quantifying the degree of embedding concentration for Text and Image. Higher values indicate narrower cones (stronger concentration). R∈[0,1] .
Figure 3: Downstream AUC as a function of alignment strength λ across datasets and backbones. Moderate reduction of the modality gap consistently improves performance, while excessive closure can lead to saturation or degradation, particularly for medical-domain models. Error bars denote standard deviation across seeds.
Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.
Rhea Chowers, Oshri Naparstek, Udi Barzelay +1
Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel
Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.
In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.