Authors: Zohar Mashiah, Nir Ben-Ari, Ronen Talmon, Uri Shaham
Organizations: Department of Computer Science and AI, Bar-Ilan University, Ramat-Gan, Israel · Electrical and Computer Engineering, Technion, Haifa, Israel
Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.
Figures & tables
Figure 1: Augmentation-based objectives induce cross-modal alignment. Cross-modal text-vision alignment (mutual k -nearest neighbors) as a function of model size (number of parameters) across different text models to DINOv2. Notably, augmentation-based models achieve higher cross-modal alignment for a given model size, and yield superior alignment even with significantly fewer parameters.
Figure 2: (a) Theoretical setup visualization. Three manifolds M , U , and V generate two observation spaces X and Y with shared semantics M . Two trainable encoders then map the observations to the corresponding embedding spaces EX and EY . (b) Correlation between CKA-RBF and m k NN. m k NN is highly correlated with CKA-RBF across different choices of k and the corresponding σ . Both Spearman ( ρs ) and Pearson ( r ) correlations are close to 1.
Figure 3: (a) Empirical validation of Thm. 1 and 2 . Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ). Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ). (b) Augmentations vs. Supervision. CMA throughout training of a vision model (ViT) against frozen text models: an augmentation-based model (SBERT) and a non-augmentation-based model (BERT). Training ViT with augmentations achieves substantially higher CMA, including compared to the BERT model.
Figure 4: Alignment by model/corpus size. CMA between text models and the DINOv2 vision encoder, using augmentation-based (Aug) and next-token prediction-based (NTP) models. Shown as a function of (a) model size (number of parameters), and (b) corpus size (amount of tokens the model was trained on). Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 5: Alignment by unimodal loss. Image-text CMA between text models and DINOv2, and vision models and BLOOM as a function of unimodal (a) text InfoNCE loss , and (b) image InfoNCE loss . In contrast to model or corpus size, the unimodal Aug-SSL loss serves as a robust predictor of CMA. It cleanly separates low- and high-alignment models, exhibiting strong Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 6: Audio-text results. CMA between text models and the BYOL-A audio encoder as a function of (a) model size (number of parameters), (b) corpus size (amount of tokens the model was trained on), and (c) unimodal text InfoNCE loss . Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, while the unimodal loss is highly correlated, indicated by both Spearman ( ρs ) and Pearson ( r ) correlations.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Audio-text results on AudioCaps dataset. CMA between text models and the BYOL-A audio encoder as a function of (a) model size (number of parameters), (b) corpus size (amount of tokens the model was trained on), and (c) unimodal text InfoNCE loss . Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, while the unimodal loss is highly correlated, indicated by both Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 8: Extended Empirical validation of Thm. 1 and 2 with different LLMs augmentations. Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ) using Phi-3.5-Mini and Gemma 4 12B text augmentations. Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ).
Figure 9: Extended Empirical validation of Thm. 1 and 2 with back-translation augmentations. Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ) using back-translation text augmentations. Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ).
Figure 10: CMA by InfoNCE across LLMs text augmentations on the COCO and WIT datasets. CMA by unimodal text InfoNCE loss across different LLMs (Phi-3.5-mini, Gemma 4 12B) augmentations. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 11: CMA by InfoNCE across back-translation text augmentations on the COCO and WIT datasets. CMA by unimodal text InfoNCE loss using back-translation augmentations. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 12: CMA across vision architectures on the COCO dataset. CMA by model size (number of parameters) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 13: CMA across vision architectures on the WIT dataset. CMA by model size (number of parameters) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 14: CMA by corpus size across vision architectures on the COCO dataset. CMA by corpus size (tokens trained on) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 15: CMA by corpus size across vision architectures on the WIT dataset. CMA by corpus size (tokens trained on) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 16: CMA by InfoNCE across vision architectures on the COCO dataset. CMA by unimodal text InfoNCE loss across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 17: CMA by InfoNCE across vision architectures on the WIT dataset. CMA by unimodal text InfoNCE loss across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 18: CMA by Vision InfoNCE across text architectures on the COCO dataset. CMA by unimodal vision InfoNCE loss across different text encoder families (MiniLM, RoBERTa, Sentence-T5, BLOOM, OpenLLaMA, LLaMA, Pythia, and GPT-2) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 19: CMA by Vision InfoNCE across text architectures on the WIT dataset. CMA by unimodal vision InfoNCE loss across different text encoder families (MiniLM, RoBERTa, Sentence-T5, BLOOM, OpenLLaMA, LLaMA, Pythia, and GPT-2) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 20: CMA across audio architectures on the CLOTHO dataset. CMA by model size (number of parameters) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 21: CMA across audio architectures on the AudioCaps dataset. CMA by model size (number of parameters) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 22: CMA by corpus size across audio architectures on the CLOTHO dataset. CMA by corpus size (tokens trained on) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 23: CMA by corpus size across audio architectures on the AudioCaps dataset. CMA by corpus size (tokens trained on) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, as indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 24: CMA by InfoNCE across audio architectures on the CLOTHO dataset. CMA by unimodal text InfoNCE loss across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 25: CMA by InfoNCE across audio architectures on the AudioCaps dataset. CMA by unimodal text InfoNCE loss across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 26: Mutual k-NN across k values on the COCO and WIT datasets. Mutual k-NN by model size (number of parameters) across different k values for the metric. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all k values jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 27: CKNNA across k values on the COCO and WIT datasets. CKNNA by model size (number of parameters) across different k values for the metric. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all k values jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 28: Mutual k-NN by InfoNCE across k values on the COCO and WIT datasets. Mutual k-NN by unimodal text InfoNCE loss across different k values for the metric. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 29: CKNNA by InfoNCE across k values on the COCO and WIT datasets. CKNNA by unimodal text InfoNCE loss across different k values for the metric. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 30: CMA by test set size across capacity scales on the COCO dataset. CMA by test size across different DINOv2 capacity scales. Notably, all text models keep about the same relation across different test set size.
Figure 31: CMA by InfoNCE across different temperature on the COCO dataset. CMA by unimodal text InfoNCE loss across different temperature for the InfoNCE loss. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, very consistent across different temperature, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 32: Mutual k-NN by CKA-RBF across k values on the COCO dataset. Mutual k-NN by CKA-RBF across different k values for Mutual k-NN. Notably, Mutual k-NN is highly correlated with CKA-RBF, very consistent across different k, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 33: Visualization of the (a) lower and (b) upper bounds from Thm. 1 and 2 .
Figure 34: Extended visualization of the lower bound from Thm. 1 , across different choices of τ and σ2 .
Figure 35: Extended visualization of the upper bound from Thm. 2 , across different choices of τ and σ2 .
Cross-modal alignment is a crucial task in multimodal learning aimed at achieving semantic consistency between vision and language. This requires that image-text pairs exhibit similar semantics. Traditional algorithms pursue embedding consistency to achieve semantic consistency, ignoring the non-semantic information present in the embedding. An intuitive approach is to decouple the embeddings into semantic and modality components, aligning only the semantic component. However, this introduces two main challenges: (1) There is no established standard for distinguishing semantic and modal information. (2) The modality gap can cause semantic alignment deviation or information loss. To align the true semantics, we propose a novel cross-modal alignment algorithm via Constrained Decoupling and Distribution Sampling (CDDS). Specifically, (1) A dual-path UNet is introduced to adaptively decouple the embeddings, applying multiple constraints to ensure effective separation. (2) A distribution sampling method is proposed to bridge the modality gap, ensuring the rationality of the alignment process. Extensive experiments on various benchmarks and model backbones demonstrate the superiority of CDDS, outperforming state-of-the-art methods by 6.6% to 14.2%.
Xiang Ma, Lexin Fang, Litian Xu +1
Shandong University, Jinan, China · The University of Exeter, Exeter, United Kingdom
The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.
Simon Roschmann, Paul Krzakala, Sonia Mazelet +2
Helmholtz Munich · Technical University of Munich · Munich Center for Machine Learning +3
Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds and when each fails, a gap that leaves practitioners, especially in scientific domains with heterogeneous instruments and multiple levels of measurement, unable to diagnose why standard methods underperform the best single modality. We study both objectives under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, the ingredient that breaks the classical recovery guarantees, and derive separation ratios that expose complementary failure modes: alignment whitens each modality and fails when nuisance is strongly correlated across views; prediction encodes whatever is cross-predictable through a one-sided whitening, with recovery governed by source-modality quality. The resulting phase diagram partitions multimodal problems into four regimes --- Both, CA only, CP only, and Neither --- refined by a recovery count that separates partial recovery from complete failure. We present a data-driven procedure to locate real-world datasets in this diagram using a small labeled subsample, identifying the preferred objective and prediction direction before any cross-modal training, and identifying when no objective in the CA/CP family can improve on the stronger modality alone. Experiments on synthetic data, stereo-vision benchmarks, image--caption pairs, and two real scientific domains, astronomy and single-cell multi-omics, validate the predictions in the nonlinear regime, including both faces of the Neither regime. Code to reproduce the results is available at https://github.com/IlayMalinyak/mm_align_vs_pred.