Asymmetric Multimodal Learning with Decomposed JEPAs
Abstract
Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a sole objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom , annihilating private modality-specific structure and cross-modal synergy . We introduce Decomposed JEPA (D-JEPA), which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.
Figures & tables
| Model / Arm | Shared (%) | Private Vis (%) | Private Txt (%) | Synergy (%) | Synergy Gap (pp) |
|---|---|---|---|---|---|
| Raw (Frozen Backbone Ceiling) | |||||
| Alignment-only | |||||
| Alignment-only + Rec | |||||
| D-JEPA (No-Independence) | |||||
| D-JEPA (No-Reconstruction) | |||||
| D-JEPA (Full Proposed) |
| Modality Probing Acc (%) | Synergy Gap | Category Breakdown (%) | Paired Gain vs. | ||||||
| Model / Arm | Vision ( ) | Text ( ) | Joint ( ) | Attribute | Counting | Exist/Cmp | Other | Alignment-only (CI) | |
| Raw (Frozen Ceiling) | |||||||||
| Alignment-only | — | ||||||||
| Alignment-only + Rec | |||||||||
| D-JEPA (No-Indep) | |||||||||
| D-JEPA (No-Rec) | |||||||||
| (a) Isolated D-JEPA Subspace Probing Accuracy on CLEVR Answers (%) | |||||
|---|---|---|---|---|---|
| Metric | Shared Pair ( ) | Private Pair ( ) | Vision Private ( ) | Text Shared ( ) | Text Private ( ) |
| Probe Accuracy | |||||
| Model / Stream | Redundancy ( ) | Vision-Unq ( ) | Text-Unq ( ) | Synergy ( ) | Total | % Raw Ceiling |
|---|---|---|---|---|---|---|
| Probe-PID: Downstream CLEVR Answer Prediction (All Categories) | ||||||
| Raw ( ) | bits | |||||
| Alignment-only ( ) | bits | |||||
| Alignment-only + Rec | bits | |||||
| D-JEPA (Full Stream) | bits | |||||
| Category: Counting Only | bits | (Raw ) | ||||
| Intervention | Swap Shared Vis ( ) | Swap Private Vis ( ) | Swap Shared Txt ( ) | Swap Private Txt ( ) |
|---|---|---|---|---|
| Alignment-only ( / ) | — | — | ||
| Alignment-only + Rec | — | — | ||
| D-JEPA (No-Indep) | ||||
| D-JEPA (No-Rec) | ||||
| D-JEPA (Full Proposed) |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm / Target | v | t | joint | c | u | ||||
|---|---|---|---|---|---|---|---|---|---|
| Target: Shared Factor ( ) | |||||||||
| Raw | — | — | — | — | — | — | |||
| Alignment-only | — | — | — | — | — | — | |||
| Alignment-only + Rec | — | — | — | — | — | — | |||
| D-JEPA (Full) | |||||||||
| D-JEPA (No-Indep) | |||||||||
| Category | Parameter | Value |
|---|---|---|
| Synthetic Benchmark | Samples ( ) / views / latent factors | / views / factors ( ) |
| Steps / batch size | / | |
| Latent component dim / Total dim | / | |
| CLEVR Benchmark | Images / QA pairs per image / Total pairs | / / |
| Pretraining steps / batch size / views | / / | |
| Encoders: Vision / Language | DINOv2 ViT-S/14 ( ) / BERT-base ( ) |
| Raw Backbones | Alignment-only | D-JEPA (Proposed) | |||||||
| Subspace | shared | priv_v | priv_t | shared | priv_v | priv_t | shared | priv_v | priv_t |
| Modality 1: | |||||||||
| Modality 2: | |||||||||
| Private Vis: | — | — | — | — | — | — | |||
| Private Txt: | — | — | — | — | — | — | |||
| Model / Stream | Redundancy ( ) | Vision-Unique ( ) | Text-Unique ( ) | Synergy ( ) | Total |
|---|---|---|---|---|---|
| Target: Synergistic Interaction Factor ( ) | |||||
| Raw ( ) | bits | ||||
| Alignment-only ( ) | bits | ||||
| Alignment-only + Rec | bits | ||||
| D-JEPA (Full Stream) | bits | ||||
| D-JEPA (Shared Pair ) | bits | ||||
| Model / Representation | Redundancy | ||||||
|---|---|---|---|---|---|---|---|
| Raw ( ) | |||||||
| Alignment-only ( ) | |||||||
| Alignment-only + Rec | |||||||
| D-JEPA (Full Stream) | |||||||
| D-JEPA (Shared Pair ) | |||||||
| D-JEPA (Private Pair ) |
| Target Task | Swap (Shared Vis) | Swap (Private Vis) | Swap (Shared Txt) | Swap (Private Txt) |
|---|---|---|---|---|
| Shared Target | ||||
| Private Vision | ||||
| Private Text | ||||
| Synergy Target |
| CKA | mKNN | Procrustes | ||||
| Subspace Pair | Value | Null | Value | Null | Value | Null |
| Synthetic Ground-Truth Benchmark | ||||||
| Raw | — | — | ||||
| Alignment-only | ||||||
| Alignment-only + Rec | ||||||
| D-JEPA (Shared) | ||||||
Multimodal Learning
Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.
See all 359 papers