Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning

Tillmann Rheude; Stefan Hegselmann; Roland Eils; Benjamin Wild

cs.LGApr 7, 2026

Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning

Authors: Tillmann Rheude, Stefan Hegselmann, Roland Eils, Benjamin Wild

Organizations: Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

Abstract

Contrastive learning has become a standard approach for unsupervised learning from paired data, as demonstrated by CLIP for image-text matching. However, many domains involve more than two modalities and require objectives that capture higher-order dependencies beyond pairwise alignment. Symile extends CLIP to this setting by replacing the dot product with the multilinear inner product (MIP) over modality embeddings. In this work, we show that there is a fragility which ishidden in the multiplicative interaction: a single weakly informative, misaligned, or missing modality can propagate through the objective and distort cross-modal retrieval scores. We propose Gated Symile, a contrastive gating mechanism that adapts modality contributions on an attention-based, per-candidate basis. The gate suppresses unreliable inputs by interpolating embeddings toward learnable neutral directions with an explicit NULL option when reliable cross-modal alignment is unlikely. Across a controlled synthetic benchmark that uncovers this fragility and three real-world trimodal datasets, Gated Symile achieves higher top-1 retrieval accuracy than well-tuned state-of-the-art (sota) baselines. More broadly, our results highlight gating as a step toward robust multimodal contrastive learning beyond two modalities in the presence of noise, misalignment, or missing inputs.

Figures & tables

Figure 1: Illustrative overview of Gated Symile exemplified with the trimodal Symile-MIMIC [ 55 ] dataset. (a) During training, modality-specific encoders $E_{m}$ create embeddings $e_{m}$ and our proposed gate $G$ produces target - conditioned weights over the available modalities and a NULL option $N$ (heatmap). The gate forms gated embeddings by weighting and interpolating each modality embedding with a modality - specific neutral direction $n_{m}$ (coordinate system), enabling the model to suppress unreliable modalities while preserving useful signal. The gated embeddings are then combined via Symile’s MIP (cube with positives on the diagonal). (b) At inference, the gating is applied to compute candidate scores for zero - shot prediction.

Figure 2: Sigmoid gate with NULL option and neutral directions.

Dataset	# Samples	Modalities	Retrieval
Synthetic- XNOR	$30,000$	$A=[u,v,uv]$ , $B=[u,1,u]$ , $C=[1,v,v]$	$A$
Symile-MIMIC [ 55 ]	$10,345$	Chest X-ray, Laboratory, ECG	Chest X-ray
UKB [ 60 ]	$37,888$	Proteomics, Metabolomics, EHR	Proteomics
UKB -Union [ 60 ]	$486,400$	Proteomics, Metabolomics, EHR	Proteomics

Table 1: Overview of benchmark datasets for our evaluation. For the Synthetic- XNOR dataset,

u,v\in\{0,1\}^{K}

with

K=16

, and

uv=\texttt{XNOR}(u,v)

applied element-wise.

Method	Synthetic- XNOR $\uparrow$	Symile-Mimic $\uparrow$	UKB $\uparrow$	UKB -Union $\uparrow$
CLIP [ 50 ]	$0.2434$	$0.4103\pm 0.016$	$0.4089\pm 0.015$	$0.0516\pm 0.007$
TRIANGLE [ 11 ]	$0.6093$	$0.0948\pm 0.003$	$0.5651\pm 0.012$	$0.3597\pm 0.020$
GRAM [ 12 ]	$0.4864$	$0.2516\pm 0.015$	$0.1848\pm 0.007$	$0.2008\pm 0.014$
Symile [ 55 ]	$0.3310$	$0.4556\pm 0.006$	$0.6570\pm 0.012$	$0.5278\pm 0.009$
Gated Symile	$\mathbf{0.8733}$	$\mathbf{0.4670\pm 0.005}$	$\mathbf{0.6819\pm 0.010}$	$\mathbf{0.6000\pm 0.007}$

Table 2: Comparison of Gated Symile with well-tuned sota baselines on synthetic (

p=1.0

) and real-world datasets. Values represent top-1 accuracy of the retrieval task (mean

\pm

SE ).

Panel of Figure 3 — Figure 3: Analyses of well-tuned models on the Synthetic- XNOR dataset with probability $p$ of one non-target modality being misaligned. (Left) Decreasing retrieval accuracy under increasing misalignment. Symile only slightly outperforms CLIP along different values for $p$ . The proposed Gated Symile, preserves the accuracy of Symile and prevents a collapse of the MIP , demonstrating that gating improves robustness to misaligned modalities. (Right) Gate selects the reliable modality under misalignment. With two separate bar charts (toward left and right), we report the mean gate weight difference $w_{A\rightarrow B}-w_{A\rightarrow C}$ . When $B$ is misaligned (left), the difference becomes negative , i.e., the gate assigns a smaller weight to $B$ than to $C$ . When $C$ is misaligned (right), the difference becomes positive , i.e., the opposite behavior. Results are averaged over six random seeds.

Dataset	Modalities	$w_{t\to m}$	$\cos(e^{G}_{m},e_{m})$	$\cos(e^{G}_{m},n_{m})$
Synthetic- XNOR	B / C	$0.3428$ / $0.1599$	$0.4385$ / $0.1656$	$0.7171$ / $0.9581$
Symile-MIMIC [ 55 ]	Lab. / ECG	$0.3656$ / $0.4568$	$0.9366$ / $0.9465$	$0.3596$ / $0.4837$
UKB [ 60 ]	Metabol. / EHR	$0.5679$ / $0.3867$	$0.8921$ / $0.8203$	$0.6541$ / $0.7451$
UKB -Union [ 60 ]	Metabol. / EHR	$0.5808$ / $0.4884$	$0.7819$ / $0.6489$	$0.4854$ / $0.6408$

Table 3: Analysis of mean gate statistics for the non-target modalities. Rows correspond to datasets and columns to gate diagnostics computed for the non-target modalities

m

(2 nd column) relative to the target modality

t

. We highlight

\cos(e_{m}^{G},e_{m})

: The relative changes across datasets mirror downstream performance , i.e., larger gains are associated with stronger deviations from the original embedding, reflected by smaller cosine similarity. This underscores the gate’s effectiveness.

Panel of Figure 4 — Figure 4: Scaling analyses of well-tuned models on the Synthetic- XNOR dataset under both increasing misalignment probability $p$ and batch sizes ( $128$ , $256$ , $512$ illustrated with decreasing brightness). (Left) Joint scaling of $B$ and negatives per anchor $K$ . Symile degrades substantially under misalignment, whereas Gated Symile remains markedly more robust, indicating increased fragility at larger contrastive scales. (Right) Scaling of $B$ while keeping negatives per anchor $K$ constant. This mirrors the joint-scaling regime, suggesting that the dominant scaling pathology is driven by the enlarged candidate pool: as $B$ grows, fragility becomes more prevalent. Gating mitigates this effect by suppressing unreliable factors, thereby preserving retrieval performance as scale increases.

Ablation	Top-1 Accuracy $\uparrow$
Gated Symile	$0.6819\pm 0.010$
w/ neutral ones	$0.6708\pm 0.014$
w/o NULL option	$0.6644\pm 0.013$
w/ neutral frozen	$0.6629\pm 0.013$
w/ softmax ( w/o sigmoid)	$0.6622\pm 0.012$
w/o renorm	$0.6578\pm 0.013$

Table 4: Well-tuned ablation of the gate on the UKB (mean

\pm

SE , details in Appendix G ).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Ablation	Top-1 Accuracy $\uparrow$
Gated Symile	$0.8730$
w/ neutral ones	$0.8523$
w/o NULL option	$0.8920$
w/ neutral frozen	$0.8690$
w/ softmax ( w/o sigmoid)	$0.8813$
w/o renorm	$0.8736$

Appendix

Table 5: Well-tuned ablation of the gate on the Synthetic- XNOR dataset ordered ablation-wise according to the ablation on the UKB .

Ablation	Parameters $\downarrow$
Gated Symile	$132$ M
w/ neutral ones	$44.1$ M
w/o NULL option	$264$ M
w/ neutral frozen	$44.1$ M
w/ softmax ( w/o sigmoid)	$44.1$ M
w/o renorm	$264$ M

Appendix

Table 6: Parameter counts of (Gated) Symile and our ablated variants. Gate-related parameters (without encoder parameters) per ablation configuration are listed, so they may change across variants.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.0273882549
optimizer.lr	0.0009146280
optimizer.warmup_steps	1200
optimizer.weight_decay	0.01

Appendix

Table 7: Ablation hyperparameters: Gated Symile.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.4172494091
optimizer.lr	0.0008030235
optimizer.warmup_steps	1200
optimizer.weight_decay	0.0

Appendix

Table 8: Ablation hyperparameters: w/ neutral ones.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.0385663517
optimizer.lr	0.0024638659
optimizer.warmup_steps	500
optimizer.weight_decay	0.001

Appendix

Table 9: Ablation hyperparameters: w/o NULL option.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.0885577025
optimizer.lr	0.0007128068
optimizer.warmup_steps	1200
optimizer.weight_decay	0.001

Appendix

Table 10: Ablation hyperparameters: w/ neutral frozen.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.0981230686
optimizer.lr	0.0027758740
optimizer.warmup_steps	1000
optimizer.weight_decay	0.001

Appendix

Table 11: Ablation hyperparameters: w/ softmax.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.0676587788
optimizer.lr	0.0033153970
optimizer.warmup_steps	1000
optimizer.weight_decay	0.01

Appendix

Table 12: Ablation hyperparameters: w/o renorm.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.1154340629
optimizer.lr	0.0026584110
optimizer.warmup_steps	1200
optimizer.weight_decay	0.01

Appendix

Table 13: Ablation hyperparameters: w/o attention.

Parameter	Value
modelname.negative_sampling	pair
modelname.emb_dim	6144
modelname.logit_scale_init	-0.1298500657
optimizer.lr	0.0012920771
optimizer.warmup_steps	1200
optimizer.weight_decay	0.01

Appendix

Table 14: Ablation hyperparameters: w/o neutral & random.

Multimodal Contrastive Learning

Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.

See all 246 papers

Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation Learning Multimodal Contrastive Learning
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMs Multimodal Embedding
Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel

Oct 23, 2026J/K move · Enter open · S save

Closing the Modality Gap via Simplex-Constrained Representations

Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi+1

Mila – Québec AI Institute · ServiceNow Research · Université Laval+1

Modality Gap in VLMs Cross-Modal Alignment Representation Learning Multimodal Contrastive Learning Cross-Modal Retrieval

Open paper AskPDF arXiv

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere,

S^{d-1}

, to the probability simplex,

Δ^{d-1}

. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid

\ell_2

gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

Abstract

Multimodal Contrastive Learning

Explore similar work

Closing the Modality Gap via Simplex-Constrained Representations

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Is the Modality Gap a Bug or a Feature? A Robustness Perspective