Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

Tillmann Rheude; Roland Eils; Benjamin Wild

cs.LGOct 10, 2026

Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

Authors: Tillmann Rheude, Roland Eils, Benjamin Wild

Organizations: Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

Abstract

Multimodal contrastive learning typically relies on manually specified scoring functions, such as cosine similarity, to quantify the similarity between representations from different modalities. Several such functions have been recently proposed to extend contrastive learning beyond two modalities. In principle, however, any function that assigns high compatibility to matched samples and low compatibility to mismatched ones could serve as a scorer. Motivated by this broader view, we instantiate the scorer with a Transformer and find that even a randomly initialized model, when kept frozen, can outperform strong handcrafted multimodal contrastive baselines, whereas optimizing the scorer jointly with the encoders is less effective. Building on this observation, we introduce the Prior-Fitted Scorer (PFS): a Transformer scorer pre-trained on synthetic multimodal data generated from structural causal model (SCM) priors before transferring it to real-world data. The SCM exposes the scorer to diverse dependency structures, including shared, pairwise, and private information, providing a structured prior over higher-order cross-modal similarity. Across four real-world datasets, frozen Transformer scorers are competitive to contrastive baselines, and prior-fitting further improves the random initialization. More broadly, our results suggest that multimodal contrastive learning need not rely on manual similarity functions: Transformers can provide strong inductive biases, while causal pretraining can further shape these biases.

Figures & tables

Figure 1: Overview of the Prior-Fitted Scorer (PFS) , illustrated on the trimodal MIMIC-Symile setting ( Saporta et al., 2024 ) . (a) We sample latent causal structures and generate synthetic multimodal tuples from an SCM prior $\mathcal{M}$ . (b) A Transformer scoring function $s_{\theta}$ is pretrained on these synthetic tuples to distinguish compatible from mismatched multimodal combinations with loss $\mathcal{L}$ , thereby shaping its inductive bias. (c) The PFS is transferred to the real dataset and kept frozen while the modality encoders $E_{i}$ are optimized through the contrastive objective. (d) After extensive tuning, the PFS is evaluated on zero-shot predicting the future chest X-ray. Overall, a PFS replaces a manually specified similarity function with a Transformer whose structure is learned from a synthetic causal prior.

Figure 2: Overview of synthetic multimodal data generation and contrastive prior-fitting. A DAG

\mathcal{G}

defines causal dependencies among latent variables, while the binary assignment matrix

\mathbf{A}

selects the latent subsets contributing to each modality, inducing shared, subset-specific, and modality-private information. The selected latent variables are combined through linear and nonlinear readouts to produce modality vectors. During prior-fitting, matched tuples contain modalities from the same synthetic sample, whereas negatives are formed by replacing the target modality with a sampled mismatched candidate before scoring with

s_{\theta}

Dataset	# Samples	Modalities
MIMIC-IV ( Johnson et al., 2023b ; Johnson et al., 2023a )	$52,802$	Reports, Clinicals, Demographics
MIMIC-Symile ( Saporta et al., 2024 )	$10,345$	Chest X-rays, ECGs , laboratory
UKB ( Sudlow et al., 2015 )	$486,400$	Proteomics, Metabolomics, EHR
INSPECT ( Huang et al., 2023 )	$22,449$	CTs, Reports, EHR

Table 1: Trimodal benchmark datasets with retrieval modality as first listed modality.

Scorer	INSPECT $\uparrow$	UKB $\uparrow$	MIMIC-Symile $\uparrow$	MIMIC-IV $\uparrow$
Baselines
Clip	$0.5548\pm 0.005$	$0.4738\pm 0.008$	$0.3109\pm 0.006$	$0.6503\pm 0.001$
Triangle	$0.5183\pm 0.014$	$0.5259\pm 0.009$	$0.4477\pm 0.005$	$0.6476\pm 0.003$
Gram	$0.4586\pm 0.012$	$0.4623\pm 0.009$	$0.3240\pm 0.005$	$0.6454\pm 0.002$
Symile	$0.6979\pm 0.003$	$0.5554\pm 0.005$	$0.4707\pm 0.008$	$0.6562\pm 0.003$
Proposed

Table 2: Comparison of well-tuned models optimized with different multimodal contrastive objectives. Values represent top-1 accuracy of the retrieval task (mean

\pm

SE ). Best non-overlapping value bold.

Figure 3: Validation performance of well-tuned baselines, the frozen Transformer and the PFS on the UKB . a) Scaling the prior-fitting with more sampled SCMs improves transfer to real-world data. b) Using the frozen PFS leads to faster convergence and higher retrieval accuracy than the baselines.

Table 6

Ablation	Top-1 Acc $\uparrow$
Only private information SCMs	$0.3170\pm 0.007$
Transformer scorer (frozen, Xavier)	$0.4462\pm 0.005$
Permute BC	$0.4643\pm 0.003$
2k SCMs	$0.4723\pm 0.015$
Remove DAG edges	$0.5431\pm 0.008$
5k SCMs	$0.5527\pm 0.003$

Table 5: Ablation study for the SCM design and prior-fitting the PFS on the UKB dataset.

Metric	MLP	Transformer	PFS	Clip	Triangle	Gram	Symile
Training step wall time (s) $\downarrow$	$0.049$	$0.244$	$0.244$	$0.032$	$0.031$	$0.035$	$0.038$

Table 6: Wall time for the frozen Transformer, PFS , and the baselines averaged over

10

epochs.

Multimodal Contrastive Learning

Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.

See all 240 papers

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation Learning Multimodal Contrastive Learning
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildGeometric Representation Learning Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies
Oct 10, 2026Duo Li, Haojun Pan, Zuhao Yang +4Multimodal Embedding Multimodal Contrastive Learning
CCDS, Nanyang Technological University · Alibaba Group · Universität Hamburg +1

Sep 26, 2026J/K move · Enter open · S save

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Yavuz Yarici, Ghassan AlRegib

OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

Cross-Modal Representation Learning Multimodal Contrastive Learning Multimodal SSL

Open paper AskPDF arXiv

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a

+5.98\%

gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.

Abstract

Multimodal Contrastive Learning

Explore similar work

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies