Authors: Zohar Mashiah, Nir Ben-Ari, Ronen Talmon, Uri Shaham
Organizations: Department of Computer Science and AI, Bar-Ilan University, Ramat-Gan, Israel · Electrical and Computer Engineering, Technion, Haifa, Israel
Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.
Figures & tables
Figure 1: Augmentation-based objectives induce cross-modal alignment. Cross-modal text-vision alignment (mutual k -nearest neighbors) as a function of model size (number of parameters) across different text models to DINOv2. Notably, augmentation-based models achieve higher cross-modal alignment for a given model size, and yield superior alignment even with significantly fewer parameters.
Figure 2: (a) Theoretical setup visualization. Three manifolds M , U , and V generate two observation spaces X and Y with shared semantics M . Two trainable encoders then map the observations to the corresponding embedding spaces EX and EY . (b) Correlation between CKA-RBF and m k NN. m k NN is highly correlated with CKA-RBF across different choices of k and the corresponding σ . Both Spearman ( ρs ) and Pearson ( r ) correlations are close to 1.
Figure 3: (a) Empirical validation of Thm. 1 and 2 . Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ). Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ). (b) Augmentations vs. Supervision. CMA throughout training of a vision model (ViT) against frozen text models: an augmentation-based model (SBERT) and a non-augmentation-based model (BERT). Training ViT with augmentations achieves substantially higher CMA, including compared to the BERT model.
Figure 4: Alignment by model/corpus size. CMA between text models and the DINOv2 vision encoder, using augmentation-based (Aug) and next-token prediction-based (NTP) models. Shown as a function of (a) model size (number of parameters), and (b) corpus size (amount of tokens the model was trained on). Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 5: Alignment by unimodal loss. Image-text CMA between text models and DINOv2, and vision models and BLOOM as a function of unimodal (a) text InfoNCE loss , and (b) image InfoNCE loss . In contrast to model or corpus size, the unimodal Aug-SSL loss serves as a robust predictor of CMA. It cleanly separates low- and high-alignment models, exhibiting strong Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 6: Audio-text results. CMA between text models and the BYOL-A audio encoder as a function of (a) model size (number of parameters), (b) corpus size (amount of tokens the model was trained on), and (c) unimodal text InfoNCE loss . Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, while the unimodal loss is highly correlated, indicated by both Spearman ( ρs ) and Pearson ( r ) correlations.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Audio-text results on AudioCaps dataset. CMA between text models and the BYOL-A audio encoder as a function of (a) model size (number of parameters), (b) corpus size (amount of tokens the model was trained on), and (c) unimodal text InfoNCE loss . Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, while the unimodal loss is highly correlated, indicated by both Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 8: Extended Empirical validation of Thm. 1 and 2 with different LLMs augmentations. Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ) using Phi-3.5-Mini and Gemma 4 12B text augmentations. Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ).
Figure 9: Extended Empirical validation of Thm. 1 and 2 with back-translation augmentations. Vision (ViT) and text (SBERT) models trained to minimize/maximize the Aug-SSL loss (Eq. 3 ) using back-translation text augmentations. Minimizing both unimodal losses improves CMA (Thm. 1 ), whereas maximizing either loss prevents alignment, with CMA converging to the chance level (Thm. 2 ).
Figure 10: CMA by InfoNCE across LLMs text augmentations on the COCO and WIT datasets. CMA by unimodal text InfoNCE loss across different LLMs (Phi-3.5-mini, Gemma 4 12B) augmentations. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 11: CMA by InfoNCE across back-translation text augmentations on the COCO and WIT datasets. CMA by unimodal text InfoNCE loss using back-translation augmentations. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 12: CMA across vision architectures on the COCO dataset. CMA by model size (number of parameters) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 13: CMA across vision architectures on the WIT dataset. CMA by model size (number of parameters) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 14: CMA by corpus size across vision architectures on the COCO dataset. CMA by corpus size (tokens trained on) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 15: CMA by corpus size across vision architectures on the WIT dataset. CMA by corpus size (tokens trained on) across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 16: CMA by InfoNCE across vision architectures on the COCO dataset. CMA by unimodal text InfoNCE loss across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 17: CMA by InfoNCE across vision architectures on the WIT dataset. CMA by unimodal text InfoNCE loss across different vision encoder families (ImageNet21K, MAE, DINOv2, and CLIP) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 18: CMA by Vision InfoNCE across text architectures on the COCO dataset. CMA by unimodal vision InfoNCE loss across different text encoder families (MiniLM, RoBERTa, Sentence-T5, BLOOM, OpenLLaMA, LLaMA, Pythia, and GPT-2) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 19: CMA by Vision InfoNCE across text architectures on the WIT dataset. CMA by unimodal vision InfoNCE loss across different text encoder families (MiniLM, RoBERTa, Sentence-T5, BLOOM, OpenLLaMA, LLaMA, Pythia, and GPT-2) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 20: CMA across audio architectures on the CLOTHO dataset. CMA by model size (number of parameters) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 21: CMA across audio architectures on the AudioCaps dataset. CMA by model size (number of parameters) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 22: CMA by corpus size across audio architectures on the CLOTHO dataset. CMA by corpus size (tokens trained on) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 23: CMA by corpus size across audio architectures on the AudioCaps dataset. CMA by corpus size (tokens trained on) across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all models jointly, model/corpus sizes are uncorrelated with CMA, as indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 24: CMA by InfoNCE across audio architectures on the CLOTHO dataset. CMA by unimodal text InfoNCE loss across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 25: CMA by InfoNCE across audio architectures on the AudioCaps dataset. CMA by unimodal text InfoNCE loss across different audio encoder families (wav2vec2, HuBERT, WavLM, data2vec, AudioMAE, and BYOL-A) at various capacity scales. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 26: Mutual k-NN across k values on the COCO and WIT datasets. Mutual k-NN by model size (number of parameters) across different k values for the metric. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all k values jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 27: CKNNA across k values on the COCO and WIT datasets. CKNNA by model size (number of parameters) across different k values for the metric. Notably, Aug models consistently achieve higher CMA, even at substantially smaller model and corpus sizes. Also, considering all k values jointly, model/corpus sizes are uncorrelated with CMA, indicated by both weak Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 28: Mutual k-NN by InfoNCE across k values on the COCO and WIT datasets. Mutual k-NN by unimodal text InfoNCE loss across different k values for the metric. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 29: CKNNA by InfoNCE across k values on the COCO and WIT datasets. CKNNA by unimodal text InfoNCE loss across different k values for the metric. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 30: CMA by test set size across capacity scales on the COCO dataset. CMA by test size across different DINOv2 capacity scales. Notably, all text models keep about the same relation across different test set size.
Figure 31: CMA by InfoNCE across different temperature on the COCO dataset. CMA by unimodal text InfoNCE loss across different temperature for the InfoNCE loss. Notably, the unimodal Aug-SSL loss is highly correlated with CMA, very consistent across different temperature, clearly separating low- and high-alignment models, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 32: Mutual k-NN by CKA-RBF across k values on the COCO dataset. Mutual k-NN by CKA-RBF across different k values for Mutual k-NN. Notably, Mutual k-NN is highly correlated with CKA-RBF, very consistent across different k, and having large Spearman ( ρs ) and Pearson ( r ) correlations.
Figure 33: Visualization of the (a) lower and (b) upper bounds from Thm. 1 and 2 .
Figure 34: Extended visualization of the lower bound from Thm. 1 , across different choices of τ and σ2 .
Figure 35: Extended visualization of the upper bound from Thm. 2 , across different choices of τ and σ2 .