Closing the gap in multimodal medical representation alignment

Eleonora Grassucci; Giordano Cicchetti; Danilo Comminiello

cs.CVFeb 23, 2026

Closing the gap in multimodal medical representation alignment

Authors: Eleonora Grassucci, Giordano Cicchetti, Danilo Comminiello

Organizations: Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

Abstract

In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.

Figures & tables

Fig. 1 : Illustration of the modality gap in the medical imaging domain. Triangles represent embeddings extracted by the image encoder, while diamonds represent embeddings from the text encoder. Colors indicate shared semantic meaning. In standard CLIP-based training, the resulting latent space shows a significant modality gap (i.e., embeddings from different modalities with the same meaning remain far apart). Our method introduces additional loss functions designed to reduce this gap and to align cross-modal embeddings closely based purely on the semantic meaning.

Fig. 2 : The modality gap originates at initialization, with the two modalities clearly clustered. Such a gap still persists even after training in the conventional CLIP-based learning setting, such as MedCLIP [ 1 ] . On the contrary, the proposed method closes the gap, leveraging the whole space to cluster embeddings according to the semantics rather than to the modality type.

Method	Cos True Pairs $\uparrow$	Gap $\downarrow$	R@1	R@5	R@10
CLIP (LT) [ 2 ]	0.20	0.40	39.5	67.4	74.4
CLIP (FT) [ 9 ]	0.39	0.14	38.3	65.8	75.8
Ours	0.54	0.12	38.9	68.8	81.8

Table 1: Latent space alignment (Cos True Pairs and Gap) and retrieval results (Recall at 1,5,10) on the ROCO dataset.

Method	BLEU@1 $\uparrow$	BLEU@2 $\uparrow$	BLEU@3 $\uparrow$	BLEU@4 $\uparrow$	ROUGE L $\uparrow$	CIDER $\uparrow$
CLIP (LT) [ 2 ]	16.51	9.82	5.91	3.56	19.61	25.24
CLIP (FT) [ 9 ]	16.71	10.07	6.05	3.59	19.82	26.05
Ours	16.96	10.07	6.09	3.64	19.90	25.22

Table 2: Captioning results on the ROCO dataset.

Modality Gap in VLMs

Modality gaps in vision-language models are mismatches between visual and textual representations or their use in downstream tasks.

See all 59 papers

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings
Mar 18, 2026David Restrepo, Miguel L Martins, Chenwei Wu +5Modality Gap in VLMs Multimodal Learning
CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3
Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
Anisotropic Modality Align
May 8, 2026Xiaomin Yu, Yijiang Li, Yuhui Zhang +8Cross-Modal Learning Cross-Modal Alignment
HKUST(GZ) · NUS · Stanford +2

Mar 18, 2026J/K move · Enter open · S save

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

David Restrepo, Miguel L Martins, Chenwei Wu, Luis Filipe Nakayama+4

CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA+3

Modality Gap in VLMs Multimodal Learning Medical VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -- particularly in medical domains -- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.

Abstract

Modality Gap in VLMs

Explore similar work

On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

Closing the Modality Gap via Simplex-Constrained Representations

Anisotropic Modality Align