In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.
Figures & tables
Fig. 1 : Illustration of the modality gap in the medical imaging domain. Triangles represent embeddings extracted by the image encoder, while diamonds represent embeddings from the text encoder. Colors indicate shared semantic meaning. In standard CLIP-based training, the resulting latent space shows a significant modality gap (i.e., embeddings from different modalities with the same meaning remain far apart). Our method introduces additional loss functions designed to reduce this gap and to align cross-modal embeddings closely based purely on the semantic meaning.
Fig. 2 : The modality gap originates at initialization, with the two modalities clearly clustered. Such a gap still persists even after training in the conventional CLIP-based learning setting, such as MedCLIP [ 1 ] . On the contrary, the proposed method closes the gap, leveraging the whole space to cluster embeddings according to the semantics rather than to the modality type.
Method
Cos True Pairs ↑
Gap ↓
R@1
R@5
R@10
CLIP (LT) [ 2 ]
0.20
0.40
39.5
67.4
74.4
CLIP (FT) [ 9 ]
0.39
0.14
38.3
65.8
75.8
Ours
0.54
0.12
38.9
68.8
81.8
Table 1: Latent space alignment (Cos True Pairs and Gap) and retrieval results (Recall at 1,5,10) on the ROCO dataset.
Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -- particularly in medical domains -- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.
David Restrepo, Miguel L Martins, Chenwei Wu +5
CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3
While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, Sd−1, to the probability simplex, Δd−1. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid ℓ2 gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.
Shubham Gupta, Siva Reddy, Perouz Taslakian +2
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistent Modality Gap in the shared space. In this work, we revisit the geometric nature of the modality gap. We find that modality representations already share compatible dominant semantic geometry. What truly hinders modality interchangeability is not a simple global shift, but an anisotropic residual structure concentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropic modality gap alignment: effective modality alignment should align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropic geometric correction framework, AnisoAlign, for unpaired modality alignment. This framework leverages the internal geometric prior of the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts the modality gap from an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.