Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Carolin Cissee; Raneen Younis; Zahra Ahmadi

cs.LGFeb 16, 2026

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Authors: Carolin Cissee, Raneen Younis, Zahra Ahmadi

Organizations: Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

Abstract

Multimodal learning seeks to integrate information from heterogeneous sources, where signals may be shared across modalities, specific to individual modalities, or emerge only through their interaction. While self-supervised multimodal contrastive learning has achieved remarkable progress, most existing methods predominantly capture redundant cross-modal signals, often neglecting modality-specific (unique) and interaction-driven (synergistic) information. Recent extensions broaden this perspective, yet they either fail to explicitly model synergistic interactions or learn different information components in an entangled manner, leading to incomplete representations and potential information leakage. We introduce COrAL, a principled framework that explicitly and simultaneously preserves redundant, unique, and synergistic information within multimodal representations. COrAL employs a dual-path architecture with orthogonality constraints to disentangle shared and modality-specific features, ensuring a clean separation of information components. To promote synergy modeling, we introduce asymmetric masking with complementary view-specific patterns, compelling the model to infer cross-modal dependencies rather than rely solely on redundant cues. Extensive experiments on synthetic benchmarks and diverse MultiBench datasets demonstrate that COrAL consistently matches or outperforms state-of-the-art methods while exhibiting low performance variance across runs. These results indicate that explicitly modeling the full spectrum of multimodal information yields more stable, reliable, and comprehensive embeddings.

Figures & tables

Figure 1: COrAL Architecture for multimodal input $X=(X_{1},\ldots,X_{n})$ . Each modality is processed by a shared pathway $F^{SR}(\cdot)$ for an embedding $Z^{SR}$ , which captures synergistic and redundant information, and by a modality-specific unique pathway $F^{U}(\cdot)$ to produce $Z^{U}_{1},\ldots,Z^{U}_{n}$ . At inference, neither augmentation nor masking is applied.

Figure 2: COrAL asymmetric masking strategy for two views

X^{\prime}

and

X^{\prime\prime}

of a bimodal input

X=(X_{1},X_{2})

after encoding and latent conversion

E^{SR}_{i}(\cdot)

Figure 3: Schematic of the loss calculations between the embeddings of two views

X^{\prime}

and

X^{\prime\prime}

for bimodal input, as defined by the loss components of COrAL.

Z^{SR\prime}

and

Z^{SR\prime\prime}

are the embeddings containing information shared between modalities and

Z^{U\prime}_{1}

Z^{U\prime}_{2}

Z^{U\prime\prime}_{1}

, and

Z^{U\prime\prime}_{2}

are the embeddings of modality-unique information. Arrows with

\text{CEL}(\cdot)

indicate that cosine embedding loss is calculated between embeddings, and arrows with

\hat{I}_{\text{NCE}}(\cdot)

refer to the InfoNCE estimator of mutual information.

Model	Redundancy	Uniqueness	Synergy
CLIP [ 25 ] *	$\mathbf{100.0}$	$11.6$	$50.0$
FactorCL [ 20 ] *	$99.8$	$62.5$	$46.5$
CoMM [ 9 ] *	$99.9$	$87.8$	$71.9$
InfMasking [ 33 ] **	${99.9}_{\pm 0.09}$	$90.6_{\pm 2.31}$	$\mathbf{77.0}_{\pm 4.22}$
COrAL (ours)	$99.7_{\pm 0.08}$	$\mathbf{96.4}_{\pm 1.08}$	${74.4_{\pm 2.92}}$

Table 1: Linear probing accuracy (in %) on the bimodal Trifeature dataset. The best results are marked in bold. * denotes results are taken from [ 9 ] . ** denotes results are taken from [ 33 ] .

Model	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD	Average
CLIP [ 25 ]	$66.0_{\pm 0.39}$	$64.2_{\pm 0.45}$	$50.6_{\pm 0.75}$	$57.4_{\pm 1.25}$	$65.5_{\pm 2.71}$	$60.7_{\pm 5.94}$
FactorCL [ 20 ]	$65.1_{\pm 0.21}$	$\mathbf{71.7}_{\pm 1.38}$	$52.5_{\pm 1.74}$	$59.8_{\pm 0.77}$	$54.2_{\pm 2.40}$	$60.7_{\pm 7.08}$
CoMM [ 9 ]	$65.7_{\pm 0.42}$	$69.2_{\pm 0.48}$	$66.0_{\pm 1.67}$	$\underline{63.0}_{\pm 1.08}$	$\underline{65.3}_{\pm 2.21}$	$65.8_{\pm 1.98}$
InfMasking [ 33 ]	$\mathbf{67.7}_{\pm 0.37}$	$69.7_{\pm 0.23}$	$\underline{67.2}_{\pm 1.00}$	$62.8_{\pm 1.22}$	$\mathbf{65.5}_{\pm 0.62}$	$\underline{66.6}_{\pm 2.32}$
COrAL (ours)	$\underline{67.4}_{\pm 0.22}$	$\underline{70.4}_{\pm 0.21}$	$\mathbf{67.9}_{\pm 0.70}$	$\mathbf{63.5}_{\pm 1.04}$	$64.4_{\pm 1.76}$	${\mathbf{66.7}_{\pm 2.79}}$

Table 2: Linear evaluation accuracy (in %) after 100 epochs of self-supervised learning on MultiBench datasets. The best results are marked in bold, the second-best results are underlined.

Masking	Redundancy	Uniqueness	Synergy
Increasing ratio	$\mathbf{99.7}_{\pm 0.08}$	${96.4_{\pm 1.08}}$	${74.4_{\pm 2.92}}$
None	$99.6_{\pm 0.22}$	$93.9_{\pm 2.70}$	$50.7_{\pm 0.38}$
25% masking	$99.7_{\pm 0.14}$	$96.6_{\pm 1.39}$	$74.8_{\pm 1.25}$
50% masking	$99.5_{\pm 0.08}$	$\mathbf{97.4}_{\pm 0.63}$	$75.5_{\pm 1.09}$
75% masking	$98.6_{\pm 0.74}$	$96.3_{\pm 0.64}$	$\mathbf{76.9}_{\pm 1.99}$

Table 3: Linear probing accuracy (in %) of COrAL on Trifeature after 100 epochs of self-supervised learning, comparing the effects of asymmetric masking with increasing amounts, static masking percentages, and no masking. Best results are marked in bold.

Masking	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD
Increasing ratio	$67.4_{\pm 0.22}$	$70.4_{\pm 0.21}$	$67.9_{\pm 0.70}$	$63.5_{\pm 1.04}$	$64.4_{\pm 1.76}$
25% masking	$67.3_{\pm 0.29}$	$70.2_{\pm 0.35}$	$65.6_{\pm 0.64}$	$64.0_{\pm 0.41}$	$61.4_{\pm 2.16}$
50% masking	$67.3_{\pm 0.60}$	$70.2_{\pm 0.43}$	$66.2_{\pm 1.07}$	$63.6_{\pm 0.68}$	$60.4_{\pm 2.23}$
75% masking	$67.4_{\pm 0.39}$	$70.4_{\pm 0.39}$	$65.3_{\pm 1.15}$	$63.6_{\pm 0.84}$	$61.3_{\pm 2.47}$

Table 4: Linear probing accuracy (in %) on MultiBench datasets after 100 epochs of self-supervised learning, comparing our adaptive masking approach with static masking ratios.

Figure 4: Sensitivity analysis of COrAL: Impact of loss weight variations on linear probing accuracy across MultiBench datasets after 100 epochs. For each of the three loss components

\mathcal{L}_{\text{orthogonal}}

\mathcal{L}_{\text{shared}}

, and

\mathcal{L}_{\text{unique}}

we plot the changes in accuracy for different values of their

\lambda

, which weigh their influence on the overall loss

\mathcal{L}_{\text{COrAL}}

Unique Path Architecture	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD	Average
MLP + pool (half size) (ours)	$67.4_{\pm 0.22}$	$70.4_{\pm 0.21}$	${67.9}_{\pm 0.70}$	${63.5}_{\pm 1.04}$	${64.4}_{\pm 1.76}$	${66.7}_{\pm 2.79}$
MLP + pool (same size)	${67.3}_{\pm 0.52}$	$70.3_{\pm 0.43}$	${66.1}_{\pm 0.49}$	${63.6}_{\pm 1.32}$	${62.0}_{\pm 1.34}$	${65.9}_{\pm 2.89}$
Transformer (half size)	${67.6}_{\pm 0.46}$	$70.4_{\pm 0.24}$	${66.6}_{\pm 1.82}$	${63.9}_{\pm 0.87}$	${63.2}_{\pm 1.69}$	${{66.3}_{\pm 2.61}}$

Table 5: Linear evaluation accuracy (in %) on MultiBench datasets after 100 epochs of self-supervised learning, evaluating the impact of different architecture designs and embedding sizes for the unique pathway. We compare two architecture options for the unique pathway: a lightweight two-layer MLP with attention pooling (MLP + pool) and a Transformer-based approach, along with different embedding sizes relative to the shared pathway embedding.

Figure 5: Visualization of the density of the shared and unique representations of visual and textual information from the MOSI dataset after projection into the embedding space using UMAP. Contour lines for training data density are shown in lighter colors, and contour lines for test data density are shown in darker colors.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 154 papers

Oct 8, 2026

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

Qun Dai, Liangjian Wen, Jiang Duan +7

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5

Multimodal SSL Multimodal Learning

Ask PDF

Sep 26, 2026

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a

+5.98\%

gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.

Yavuz Yarici, Ghassan AlRegib

OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

Cross-Modal Representation Learning Multimodal Contrastive Learning Multimodal SSL

Ask PDF

Sep 28, 2026

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Wanting Huang, Sanvesh Srivastava, Weiran Wang

Department of Computer Science University of Iowa · Department of Statistics University of Iowa

Multimodal Disentangled Representation Learning Multimodal Learning

Ask PDF

Abstract

Multimodal Embedding

Explore similar work

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Structured Latent Modeling for Supervised Multimodal Information Decomposition