Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Carolin Cissee; Raneen Younis; Zahra Ahmadi

cs.LGFeb 16, 2026

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Authors: Carolin Cissee, Raneen Younis, Zahra Ahmadi

Organizations: Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

Abstract

Multimodal learning seeks to integrate information from heterogeneous sources, where signals may be shared across modalities, specific to individual modalities, or emerge only through their interaction. While self-supervised multimodal contrastive learning has achieved remarkable progress, most existing methods predominantly capture redundant cross-modal signals, often neglecting modality-specific (unique) and interaction-driven (synergistic) information. Recent extensions broaden this perspective, yet they either fail to explicitly model synergistic interactions or learn different information components in an entangled manner, leading to incomplete representations and potential information leakage. We introduce COrAL, a principled framework that explicitly and simultaneously preserves redundant, unique, and synergistic information within multimodal representations. COrAL employs a dual-path architecture with orthogonality constraints to disentangle shared and modality-specific features, ensuring a clean separation of information components. To promote synergy modeling, we introduce asymmetric masking with complementary view-specific patterns, compelling the model to infer cross-modal dependencies rather than rely solely on redundant cues. Extensive experiments on synthetic benchmarks and diverse MultiBench datasets demonstrate that COrAL consistently matches or outperforms state-of-the-art methods while exhibiting low performance variance across runs. These results indicate that explicitly modeling the full spectrum of multimodal information yields more stable, reliable, and comprehensive embeddings.

Figures & tables

Figure 1: COrAL Architecture for multimodal input $X=(X_{1},\ldots,X_{n})$ . Each modality is processed by a shared pathway $F^{SR}(\cdot)$ for an embedding $Z^{SR}$ , which captures synergistic and redundant information, and by a modality-specific unique pathway $F^{U}(\cdot)$ to produce $Z^{U}_{1},\ldots,Z^{U}_{n}$ . At inference, neither augmentation nor masking is applied.

Figure 2: COrAL asymmetric masking strategy for two views

X^{\prime}

and

X^{\prime\prime}

of a bimodal input

X=(X_{1},X_{2})

after encoding and latent conversion

E^{SR}_{i}(\cdot)

Figure 3: Schematic of the loss calculations between the embeddings of two views

X^{\prime}

and

X^{\prime\prime}

for bimodal input, as defined by the loss components of COrAL.

Z^{SR\prime}

and

Z^{SR\prime\prime}

are the embeddings containing information shared between modalities and

Z^{U\prime}_{1}

Z^{U\prime}_{2}

Z^{U\prime\prime}_{1}

, and

Z^{U\prime\prime}_{2}

are the embeddings of modality-unique information. Arrows with

\text{CEL}(\cdot)

indicate that cosine embedding loss is calculated between embeddings, and arrows with

\hat{I}_{\text{NCE}}(\cdot)

refer to the InfoNCE estimator of mutual information.

Model	Redundancy	Uniqueness	Synergy
CLIP [ 25 ] *	$\mathbf{100.0}$	$11.6$	$50.0$
FactorCL [ 20 ] *	$99.8$	$62.5$	$46.5$
CoMM [ 9 ] *	$99.9$	$87.8$	$71.9$
InfMasking [ 33 ] **	${99.9}_{\pm 0.09}$	$90.6_{\pm 2.31}$	$\mathbf{77.0}_{\pm 4.22}$
COrAL (ours)	$99.7_{\pm 0.08}$	$\mathbf{96.4}_{\pm 1.08}$	${74.4_{\pm 2.92}}$

Table 1: Linear probing accuracy (in %) on the bimodal Trifeature dataset. The best results are marked in bold. * denotes results are taken from [ 9 ] . ** denotes results are taken from [ 33 ] .

Model	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD	Average
CLIP [ 25 ]	$66.0_{\pm 0.39}$	$64.2_{\pm 0.45}$	$50.6_{\pm 0.75}$	$57.4_{\pm 1.25}$	$65.5_{\pm 2.71}$	$60.7_{\pm 5.94}$
FactorCL [ 20 ]	$65.1_{\pm 0.21}$	$\mathbf{71.7}_{\pm 1.38}$	$52.5_{\pm 1.74}$	$59.8_{\pm 0.77}$	$54.2_{\pm 2.40}$	$60.7_{\pm 7.08}$
CoMM [ 9 ]	$65.7_{\pm 0.42}$	$69.2_{\pm 0.48}$	$66.0_{\pm 1.67}$	$\underline{63.0}_{\pm 1.08}$	$\underline{65.3}_{\pm 2.21}$	$65.8_{\pm 1.98}$
InfMasking [ 33 ]	$\mathbf{67.7}_{\pm 0.37}$	$69.7_{\pm 0.23}$	$\underline{67.2}_{\pm 1.00}$	$62.8_{\pm 1.22}$	$\mathbf{65.5}_{\pm 0.62}$	$\underline{66.6}_{\pm 2.32}$
COrAL (ours)	$\underline{67.4}_{\pm 0.22}$	$\underline{70.4}_{\pm 0.21}$	$\mathbf{67.9}_{\pm 0.70}$	$\mathbf{63.5}_{\pm 1.04}$	$64.4_{\pm 1.76}$	${\mathbf{66.7}_{\pm 2.79}}$

Table 2: Linear evaluation accuracy (in %) after 100 epochs of self-supervised learning on MultiBench datasets. The best results are marked in bold, the second-best results are underlined.

Masking	Redundancy	Uniqueness	Synergy
Increasing ratio	$\mathbf{99.7}_{\pm 0.08}$	${96.4_{\pm 1.08}}$	${74.4_{\pm 2.92}}$
None	$99.6_{\pm 0.22}$	$93.9_{\pm 2.70}$	$50.7_{\pm 0.38}$
25% masking	$99.7_{\pm 0.14}$	$96.6_{\pm 1.39}$	$74.8_{\pm 1.25}$
50% masking	$99.5_{\pm 0.08}$	$\mathbf{97.4}_{\pm 0.63}$	$75.5_{\pm 1.09}$
75% masking	$98.6_{\pm 0.74}$	$96.3_{\pm 0.64}$	$\mathbf{76.9}_{\pm 1.99}$

Table 3: Linear probing accuracy (in %) of COrAL on Trifeature after 100 epochs of self-supervised learning, comparing the effects of asymmetric masking with increasing amounts, static masking percentages, and no masking. Best results are marked in bold.

Masking	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD
Increasing ratio	$67.4_{\pm 0.22}$	$70.4_{\pm 0.21}$	$67.9_{\pm 0.70}$	$63.5_{\pm 1.04}$	$64.4_{\pm 1.76}$
25% masking	$67.3_{\pm 0.29}$	$70.2_{\pm 0.35}$	$65.6_{\pm 0.64}$	$64.0_{\pm 0.41}$	$61.4_{\pm 2.16}$
50% masking	$67.3_{\pm 0.60}$	$70.2_{\pm 0.43}$	$66.2_{\pm 1.07}$	$63.6_{\pm 0.68}$	$60.4_{\pm 2.23}$
75% masking	$67.4_{\pm 0.39}$	$70.4_{\pm 0.39}$	$65.3_{\pm 1.15}$	$63.6_{\pm 0.84}$	$61.3_{\pm 2.47}$

Table 4: Linear probing accuracy (in %) on MultiBench datasets after 100 epochs of self-supervised learning, comparing our adaptive masking approach with static masking ratios.

Figure 4: Sensitivity analysis of COrAL: Impact of loss weight variations on linear probing accuracy across MultiBench datasets after 100 epochs. For each of the three loss components

\mathcal{L}_{\text{orthogonal}}

\mathcal{L}_{\text{shared}}

, and

\mathcal{L}_{\text{unique}}

we plot the changes in accuracy for different values of their

\lambda

, which weigh their influence on the overall loss

\mathcal{L}_{\text{COrAL}}

Unique Path Architecture	MIMIC	MOSEI	MOSI	UR-FUNNY	MUsTARD	Average
MLP + pool (half size) (ours)	$67.4_{\pm 0.22}$	$70.4_{\pm 0.21}$	${67.9}_{\pm 0.70}$	${63.5}_{\pm 1.04}$	${64.4}_{\pm 1.76}$	${66.7}_{\pm 2.79}$
MLP + pool (same size)	${67.3}_{\pm 0.52}$	$70.3_{\pm 0.43}$	${66.1}_{\pm 0.49}$	${63.6}_{\pm 1.32}$	${62.0}_{\pm 1.34}$	${65.9}_{\pm 2.89}$
Transformer (half size)	${67.6}_{\pm 0.46}$	$70.4_{\pm 0.24}$	${66.6}_{\pm 1.82}$	${63.9}_{\pm 0.87}$	${63.2}_{\pm 1.69}$	${{66.3}_{\pm 2.61}}$

Table 5: Linear evaluation accuracy (in %) on MultiBench datasets after 100 epochs of self-supervised learning, evaluating the impact of different architecture designs and embedding sizes for the unique pathway. We compare two architecture options for the unique pathway: a lightweight two-layer MLP with attention pooling (MLP + pool) and a Transformer-based approach, along with different embedding sizes relative to the shared pathway embedding.

Figure 5: Visualization of the density of the shared and unique representations of visual and textual information from the MOSI dataset after projection into the embedding space using UMAP. Contour lines for training data density are shown in lighter colors, and contour lines for test data density are shown in darker colors.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 156 papers

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5
SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation Learning Multimodal Contrastive Learning
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
Structured Latent Modeling for Supervised Multimodal Information Decomposition
Sep 28, 2026Wanting Huang, Sanvesh Srivastava, Weiran WangMultimodal Disentangled Representation Learning Multimodal Learning
Department of Computer Science University of Iowa · Department of Statistics University of Iowa

Oct 8, 2026J/K move · Enter open · S save

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai+6

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd+5

Multimodal SSL Multimodal Learning

Open paper AskPDF arXiv

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

Abstract

Multimodal Embedding

Explore similar work

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Structured Latent Modeling for Supervised Multimodal Information Decomposition