On the Value of Cross-Modal Misalignment in Multimodal Representation Learning

Yichao Cai; Yuhang Liu; Erdun Gao; Tianjiao Jiang; Zhen Zhang; Anton van den Hengel; Javen Qinfeng Shi

cs.LGApr 14, 2025

On the Value of Cross-Modal Misalignment in Multimodal Representation Learning

Authors: Yichao Cai, Yuhang Liu, Erdun Gao, Tianjiao Jiang, Zhen Zhang, Anton van den Hengel, Javen Qinfeng Shi

Organizations: Australian Institute for Machine Learning The University of Adelaide, SA 5000, Australia

Abstract

Multimodal representation learning, exemplified by multimodal contrastive learning (MMCL) using image-text pairs, aims to learn powerful representations by aligning cues across modalities. This approach relies on the core assumption that the exemplar image-text pairs constitute two representations of an identical concept. However, recent research has revealed that real-world datasets often exhibit cross-modal misalignment. There are two distinct viewpoints on how to address this issue: one suggests mitigating the misalignment, and the other leveraging it. We seek here to reconcile these seemingly opposing perspectives and to provide a practical guide for practitioners. Using latent variable models, we thus formalize cross-modal misalignment by introducing two specific mechanisms: selection bias, where some semantic variables are absent in the text, and perturbation bias, where semantic variables are altered---both leading to misalignment in data pairs. Our theoretical analysis shows that, under suitable assumptions, population global minimizers of an alignment--entropy surrogate yield representations that depend only on the semantic variables unaffected by selection and perturbation biases and locally identify this subset almost surely. Extensive empirical studies on synthetic data and real image-text datasets further examine how cross-modal misalignment shapes semantic predictability and downstream performance, shedding light on its nuanced impact on multimodal representation learning.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the proposed latent variable model (left), with misalignment across modalities modeled via selection and perturbation bias. Image $\mathbf{x}$ is generated from semantic variables $\mathbf{s}$ and image-specific variables $\mathbf{m}_{x}$ via the generative process $g_{x}$ . The corresponding text $\mathbf{t}^{\scriptscriptstyle(\theta)}$ is generated by $g_{t^{\scriptscriptstyle(\theta)}}$ which acts on a biased subset of semantic variables $\tilde{\mathbf{s}}_{\mathbb{I}_{\theta}}=(\mathbf{s}_{\mathbb{I}^{c}_{\rho}},\tilde{\mathbf{s}}_{\mathbb{I}_{\rho}})$ , influenced by selection bias $\theta$ and perturbation bias $\rho$ , along with text-specific variables $\mathbf{m}_{t}$ . Selection bias omits $\mathbf{s}_{\mathbb{I}^{c}_{\theta}}$ , while perturbation bias marks a subset $\mathbb{I}_{\rho}\subseteq\mathbb{I}_{\theta}$ whose components may be randomly replaced to form $\tilde{\mathbf{s}}_{\mathbb{I}_{\rho}}$ . Example image-text pairs (right) illustrate the misalignment induced by these two biases.

Figure 2: Textual modifications versus latent causal interventions.

Figure 3: Mean

R^{2}

scores under selection bias settings. From left to right: predictions based on

\hat{\mathbf{z}}_{x}

and

\hat{\mathbf{z}}_{t}

with independent latent semantics, followed by those with dependent latent semantics. Negative mean

R^{2}

scores are clipped to zero for visualization.

Figure 4: Downstream performance of pretrained

\hat{\mathbf{z}}_{x}

under selection bias. Left: in-distribution (ID) regression performance. Right: ID classification and out-of-distribution (OOD) generalization. Curves show means and shaded regions show ranges across seeds; negative regression

R^{2}

scores are clipped to zero before aggregation.

Figure 6: Mean MCC scores under misalignment settings. Left to right: Image features

\hat{\mathbf{z}}_{x}

with selection and perturbation bias settings, text features

\hat{\mathbf{z}}_{t}

under the same bias settings. Scores are clipped to

[0,1]

before averaging across seeds.

Panel of Figure 8 — Figure 8: Predicting semantic variables under misalignment using image features. $R^{2}$ is reported for continuous factors and MCC for discrete factors. The prediction patterns align with our theoretical results, though additional factors may remain predictable through statistical correlations. Scores are clipped to $[0,1]$ before averaging across seeds.

Figure 9: Concept coverage by group in LAION-400M captions. See Section H.2 for details.

Figure 10: Zero-shot F1 score of OpenCLIP model trained on LAION-400M in each concept group.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Observation and Latent Spaces
$\mathcal{X}$	Image observation space ( $\subseteq\mathbb{R}^{d_{x}}$ )
$\mathcal{T}^{\scriptscriptstyle(\theta)}$	Text observation subspace under selection bias $\theta$
$\mathcal{S}$	Latent semantic space ( $\subseteq\mathbb{R}^{n_{s}}$ )
$\mathcal{M}_{x}$	Image-specific non-semantic latent space
$\mathcal{M}_{t}$	Text-specific non-semantic latent space
$\mathbb{I}_{\mathbf{s}}$	Index set of semantic variables: $[n_{s}]$

Appendix

Table 1: Notations and terminologies used throughout the paper.

$\theta$ , $\rho=1$	$1$	$11$	$56$	$176$	$386$	$638$	$848$	$968$	$1013$	$1023$
$\mathbb{I}_{\theta}$	$\{1\}$	$[2]$	$[3]$	$[4]$	$[5]$	$[6]$	$[7]$	$[8]$	$[9]$	$[10]$

Appendix

Table 2: Selection bias settings and selected semantic indices.

$\rho\mid\theta=1023$	$1$	$2$	$12$	$57$	$177$	$387$	$639$	$849$	$969$	$1014$
$\mathbb{I}_{\rho}$	$\emptyset$	$\{1\}$	$[2]$	$[3]$	$[4]$	$[5]$	$[6]$	$[7]$	$[8]$	$[9]$

Appendix

Table 3: Perturbation bias settings and perturbable semantic indices.

Figure 11: Mean

R^{2}

scores under perturbation bias settings. Left to right: predictions using representations

\hat{\mathbf{z}}_{x}

and

\hat{\mathbf{z}}_{t}

under independent latent semantic variables, followed by those under dependent latent semantic variables.

Figure 12: Evaluating linearity of learned representations under selection bias. Left to right: predictions using representations

\hat{\mathbf{z}}_{x}

and

\hat{\mathbf{z}}_{t}

under independent latent semantic variables, followed by those under dependent latent semantic variables.

Figure 13: Linearity of learned representations under perturbation bias. Left to right: predictions using representations

\hat{\mathbf{z}}_{x}

and

\hat{\mathbf{z}}_{t}

under independent latent semantic variables, followed by those under dependent latent semantic variables.

Setting		Reps.	$R^{2}$ of Predicting Latent Semantic Variables under $\mathbb{I}_{\theta}=[3]$ and $\mathbb{I}_{\rho}=\emptyset$
Setting		Reps.	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$	$s_{5}$	$s_{6}$	$s_{7}$	$s_{8}$	$s_{9}$	$s_{10}$	$\mathbf{m}_{x}$	$\mathbf{m}_{t}$
independ.	linear	$\hat{\mathbf{z}}_{x}$	0.98	0.96	0.98	0.01	0.03	0.03	0.02	0.04	0.01	0.02	0.02	0.00
		$\hat{\mathbf{z}}_{t}$	1.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.04
	non-lin.	$\hat{\mathbf{z}}_{x}$	0.98	0.99	0.99	0.00	0.02	0.02	0.03	0.04	0.00	0.00	0.00	0.00
		$\hat{\mathbf{z}}_{t}$	1.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.06
dependent	linear	$\hat{\mathbf{z}}_{x}$	0.98	0.99	0.99	0.14	0.23	0.16	0.10	0.22	0.42	0.36	0.01	0.00

Appendix

Table 4: The effect of using an incorrect encoding size. The representation size is set to 5, whereas the true dimension should be 3. The biases are defined as

\mathbb{I}_{\theta}=[3]

and

\mathbb{I}_{\rho}=\emptyset

Setting		Reps.	$R^{2}$ of Predicting Latent Semantic Variables ( $\mathbb{I}_{\theta}=[8]$ , $\mathbb{I}_{\rho}=[2]$ )
Setting		Reps.	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$	$s_{5}$	$s_{6}$	$s_{7}$	$s_{8}$	$s_{9}$	$s_{10}$	$\mathbf{m}_{x}$	$\mathbf{m}_{t}$
independ.	linear	$\hat{\mathbf{z}}_{x}$	0.00	0.01	0.97	0.98	0.97	0.95	0.99	0.98	0.00	0.00	0.00	0.00
		$\hat{\mathbf{z}}_{t}$	0.00	0.00	0.98	0.98	0.98	0.98	0.99	0.98	0.00	0.00	0.00	0.00
	non-lin.	$\hat{\mathbf{z}}_{x}$	0.00	0.01	0.98	0.98	0.98	0.98	0.96	0.99	0.98	0.00	0.00	0.00
		$\hat{\mathbf{z}}_{t}$	0.00	0.00	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.00	0.00	0.00
dependent	linear	$\hat{\mathbf{z}}_{x}$	0.67	0.57	0.97	0.99	0.99	0.98	0.99	0.97	0.63	0.63	0.00	0.00

Appendix

Table 5: Coexistence of both selection and perturbation biases. The biases are defined as

\mathbb{I}_{\theta}=[8]

and

\mathbb{I}_{\rho}=[2]

Figure 14: Downstream performance of pretrained representations

\hat{\mathbf{z}}_{x}

under perturbation bias. Top: in-distribution (ID) regression performance. Bottom: ID classification and out-of-distribution (OOD) generalization.

Factor Name	Distribution	Content Words
Object color ( color )	$\mathrm{Uniform}(\{0,\dots,3\})$	yellow , green , olive , red
Object shape ( shape )	$\mathrm{Uniform}(\{0,\dots,3\})$	coffee-cup , tennis-ball , croissant , beer-cup
Object size ( size )	$\mathrm{Uniform}(\{0,1\})$	small , large
Camera height ( cam. )	$\mathrm{Uniform}(\{0,\dots,2\})$	top , center , bottom
Background color ( back. )	$\mathrm{Uniform}(\{0,\dots,2\})$	purple , sea-green , salmon
Horizontal axis ( hori. )	$\mathrm{Uniform}(\{0,\dots,39\})$	— (image-specific factor)

Appendix

Table 6: Latent variables of MPI3D-Complex and corresponding content words in text.

Panel of Table 7 — Table 7: Misalignment settings for text generation of MPI3D-Complex dataset.

Setting	Text Generation Templates for Each Selection View
1	"An object colored { color }." "It has a { color } appearance." "Something with { color }."
2	"A { shape } that is { color }." "The { color } { shape }." "An object shaped like a { shape }, colored { color }."
3	"A { size } { shape } in { color }." "{ color }, { size }, { shape }." "The object is { size }, shaped as a { shape }, and colored { color }."
4	"A { size }{ shape } in { color }, seen from { cam. }." "Viewed from { cam. }, a { color }, { size } { shape }." "A { size } { shape } with { color }, perspective: { cam. }."
5	"A { size } { shape } in { color }, viewed from { cam. }, { back. }." "From { cam. }, you see a { color }, { size } { shape }, { back. }." "A { size } { shape }, { color }, placed { back. }, observed from { cam. }."

Appendix

Table 8: Text generation templates for MPI3D-Complex dataset for different selection settings.

Figure 15: Evaluating linearity of learned representations under misalignment. Left to right: image features

\hat{\mathbf{z}}_{x}

under selection bias, image features

\hat{\mathbf{z}}_{x}

under perturbation bias, text features

\hat{\mathbf{z}}_{t}

under selection bias, and text features

\hat{\mathbf{z}}_{t}

under perturbation bias.

Figure 16: Ablation study on encoding size under a selection bias setting where only the { color } attribute is included in text generation. Left: average MCC over three runs using a linear classifier. Right: average MCC using a nonlinear classifier.

Figure 17: Latent causal model governing image generation in the Causal3DIdent dataset. Rectangular nodes represent discrete latent random variables, while elliptical nodes denote continuous ones. Object shape ( shape ), horizontal and vertical position ( x_pos , y_pos ), object color ( color ), spotlight position and color ( s_pos , s_color ), and background color ( b_color ) are treated as latent semantic variables shared across modalities and potentially subject to misalignment. In contrast, the rotation angles— alpha , beta , and gamma —are considered image-specific latent factors.

Factor Name	Image Modality	Text Modality	Content Words
shape	$\mathrm{Uniform}(\{0,\dots,6\})$	$\mathrm{Uniform}(\{0,\dots,6\})$	teapot , hare , dragon , cow , armadillo , horse , head
x_pos	$\mathrm{Uniform}(\{0,1,2\})$	$\mathrm{Uniform}(\{0,1,2\})$	left , center , right
y_pos	$\mathrm{Uniform}(\{0,1,2\})$	$\mathrm{Uniform}(\{0,1,2\})$	top , mid , bottom
s_pos	$\mathrm{Uniform}([0,1])$	$\mathrm{Uniform}([0,1])$	northwest , northeast , center , southwest , southeast
color	$\frac{1}{6}(\texttt{x\_pos}+\texttt{y\_pos})+\frac{1}{3}\mathrm{Uniform}([0,1])$	Up to 10 colors	Color names in TABLEAU_COLORS
s_color	$\frac{1}{2}(\texttt{s\_pos}+\mathrm{Uniform}([0,1]))$	Up to 147 colors	Color names in CSS4_COLORS

Appendix

Table 9: Latent factors of Causal3DIdent and corresponding content words in text.

Panel of Table 10 — Table 10: Misalignment settings for text generation of Causal3DIdent dataset.

Figure 18: Example image-text pairs from Causal3DIdent under different selection bias settings. The left panel shows randomly selected images; the right panel presents the corresponding text from top to bottom, each generated under selection settings 1 to 7 with no perturbations, and perturbation bias 7 to 1 with full selections.

Setting	Text Generation Templates
1	"A { shape } is visible." "A { shape } is in the image." "The image shows a { shape }." "The picture is a { shape }." "There is an object in the form of a { shape }."
2	"A { shape } is visible, positioned at the { x_pos } of the image." "A { shape } is at the { x_pos } of the image." "The { x_pos } of the image shows a { shape }." "At the { x_pos } of the picture is a { shape }." "At the { x_pos } of the image, there is an object in the form of a { shape }."
3	"A { shape } is visible, positioned at the { y_pos }-{ x_pos } of the image." "A { shape } is at the { y_pos }-{ x_pos } of the image." "The { y_pos }-{ x_pos } of the image shows a { shape }." "At the { y_pos }-{ x_pos } of the picture is a { shape }." "At the { y_pos }-{ x_pos } of the image, there is an object in the form of a { shape }."
4	"A { shape } is visible, positioned at the { y_pos }-{ x_pos }, with a spotlight shining from { s_pos }." "A { shape } is at the { y_pos }-{ x_pos }, illuminated by a light from { s_pos }." "The { y_pos }-{ x_pos } shows a { shape }, highlighted by a light from { s_pos }." "At the { y_pos }-{ x_pos } is a { shape }, under a light from { s_pos }." "There is a { shape } at { y_pos }-{ x_pos }, lit from { s_pos }."
5	"A { shape } of { color } color is visible at { y_pos }-{ x_pos }, with a spotlight from { s_pos }." "A { color } { shape } is at { y_pos }-{ x_pos }, lit from { s_pos }." "The area { y_pos }-{ x_pos } shows a { color } { shape }, under a light from { s_pos }." "A { color } { shape } is illuminated at { y_pos }-{ x_pos } from { s_pos }." "A { color } object shaped like a { shape } is lit from { s_pos }."
6	"A { color } { shape } is lit by a { s_color } spotlight from { s_pos }, at { y_pos }-{ x_pos }." "At { y_pos }-{ x_pos }, a { color } { shape } is under a { s_color } light from { s_pos }." "The { shape } is { color }, under a { s_color } light at { s_pos }." "A { color } { shape } under a { s_color } spotlight at { s_pos }, located at { y_pos }-{ x_pos }." "A { color } { shape } stands under a { s_color } light from { s_pos }."

Appendix

Table 11: Text generation templates for Causal3DIdent dataset under different selection settings.

Panel of Figure 19 — Figure 19: Prediction of latent semantic variables under misalignment with text features. $R^{2}$ for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.

Panel of Figure 20 — Figure 20: Evaluating linearity of image features under misalignment settings. $R^{2}$ for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.

Panel of Figure 21 — Figure 21: Evaluating linearity of text features under misalignment settings. $R^{2}$ for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.

Group	Explanation	Concept Name
Animal	animals	dog, cat, horse, bird, elephant, giraffe, cow, zebra, rabbit, duck
Clothing	clothing	shirt, pants, dress, shoes, hat, jacket, skirt, tie, hoodie, socks
Color	colors	red, blue, green, yellow, black, white, orange, pink, gray, brown
Food	food	pizza, burger, sandwich, salad, cake, coffee, tea, beer, ice cream, noodles
Object	object	chair, table, phone, laptop, car, bottle, bag, cup, backpack, television
Role	roles	chef, teacher, athlete, doctor, engineer, artist, pilot, firefighter, police officer, lawyer

Appendix

Table 12: Taxonomy of concept categories with abbreviations, grouped by conceptual relevance.

Group	Concept	Count	Group	Concept	Count	Group	Concept	Count
Animal	dog	124	Animal	cat	112	Animal	horse	109
Animal	bird	106	Animal	elephant	98	Animal	giraffe	90
Animal	cow	88	Animal	zebra	81	Animal	rabbit	77
Animal	duck	65	Clothing	shirt	113	Clothing	pants	104
Clothing	dress	101	Clothing	shoes	96	Clothing	hat	92
Clothing	jacket	87	Clothing	skirt	84	Clothing	tie	78

Appendix

Table 13: Concept-wise image counts grouped by concept category for zero-shot evaluation.

Panel of Figure 22 — Figure 22: Statistics of concept-level coverage within each group in LAION-400M captions.

Panel of Figure 23 — Figure 23: Confusion matrix for zero-shot prediction using OpenCLIP ViT-B-32 of each group.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 210 papers

Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment
Mar 5, 2026Xiang Ma, Lexin Fang, Litian Xu +1Cross-Modal Alignment Vision-Language Alignment
Shandong University, Jinan, China · The University of Exeter, Exeter, United Kingdom

Feb 23, 2026J/K move · Enter open · S save

Closing the gap in multimodal medical representation alignment

Eleonora Grassucci, Giordano Cicchetti, Danilo Comminiello

Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

Modality Gap in VLMs Vision-Language Alignment Medical VLMs Multimodal Contrastive Learning Medical Image Representation Learning Cross-Modal Retrieval

Open paper AskPDF arXiv

In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.

Abstract

Cross-Modal Representation Learning

Explore similar work

Closing the gap in multimodal medical representation alignment

Multimodal Representation Alignment for Cross-modal Information Retrieval

Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment