cs.LGNov 12, 2026

Asymmetric Multimodal Learning with Decomposed JEPAs

Authors: Yusuf Shihata

Abstract

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a sole objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom \Red(Z;X1,X2)\Red(Z; X_1, X_2), annihilating private modality-specific structure \Unq(Z;Xm)\Unq(Z; X_m) and cross-modal synergy \Syn(Z;X1,X2)\Syn(Z; X_1, X_2). We introduce Decomposed JEPA (D-JEPA), which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. When to Align, When to Predict: A Phase Diagram for Multimodal Learning

    Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Cross-Modal Representation LearningCross-Modal Alignment

  2. MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

    Oct 17, 2026Haowen Bai, Rui Gong$, Meichen Liu +3Multimodal SSLJoint-Embedding Predictive Architecture

  3. Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

    Oct 16, 2026Yi Fang, Gaurav Srivastava, Sirui Ding +1Multimodal FusionMultimodal Model Evaluation