cs.LGNov 2, 2026

Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment

Authors: Zohar Mashiah, Nir Ben-Ari, Ronen Talmon, Uri Shaham

Organizations: Department of Computer Science and AI, Bar-Ilan University, Ramat-Gan, Israel · Electrical and Computer Engineering, Technion, Haifa, Israel

Abstract

Recent works have observed implicit cross-modal alignment between independently trained unimodal models. However, the mechanisms underlying the emergence of this phenomenon remain unresolved. In this paper, we demonstrate that unimodal augmentation-based self-supervised objectives implicitly induce cross-modal alignment. We show that low unimodal losses force high cross-modal alignment, whereas a high unimodal loss prevents such alignment. We theoretically connect the widely used InfoNCE loss to local CKA, the popular measure of cross-modal alignment. We further provide extensive empirical evidence supporting this claim, both in controlled settings and across 53 unconstrained frozen text, vision, and audio unimodal encoders. In both cases, unimodal augmentation-based objectives control cross-modal alignment. Overall, our results provide a theoretical and empirical foundation for understanding the emergence of implicit cross-modal alignment by establishing a direct connection between unimodal training objectives and cross-modal structure.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment

    Mar 5, 2026Xiang Ma, Lexin Fang, Litian Xu +1Cross-Modal AlignmentVision-Language Alignment

  2. SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

    Feb 26, 2026Simon Roschmann, Paul Krzakala, Sonia Mazelet +2Cross-Modal LearningMultimodal Embedding

  3. When to Align, When to Predict: A Phase Diagram for Multimodal Learning

    Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Cross-Modal Representation LearningCross-Modal Alignment