CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

Chunlei Meng; Guanhong Huang; Rong Fu; Runmin Jian; Zhongxue Gan; Chun Ouyang

cs.CVFeb 23, 2026

CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

Authors: Chunlei Meng, Guanhong Huang, Rong Fu, Runmin Jian, Zhongxue Gan, Chun Ouyang

Organizations: Fudan University · Shantou University · University of Macau · Guangzhou Huashang College

Abstract

Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.

Figures & tables

Figure 1: Cross-level semantic asynchrony and CLCR. Top: mixing across levels causes semantic confusion and mismatch. bottom: CLCR structures each modality into three aligned levels and restricts exchange to the learned shared subspace at matched levels, enabling level semantic alignment and reliable aggregation.

Figure 2: Overview of the proposed CLCR framework. Each modality is organized into a three-level semantic hierarchy. IntraCED performs budgeted cross-modal exchange in an explicitly disentangled level-shared subspace, while InterCAD synchronizes and aggregates shared and private streams across levels into the final task representation.

Methods	CREMA-D [ 4 ]		KS [ 1 ]		AVE [ 33 ]		UCF101 [ 26 ]
Methods	Acc(%)	F1(%)	Acc(%)	F1(%)	Acc(%)	F1(%)	Acc(%)	F1(%)
only-Acoustic	57.27	57.89	48.67	48.89	62.16	58.54	-	-
only-Visual	62.17	62.78	52.36	52.67	31.40	29.87	-	-
Concat	58.83	59.43	64.97	65.21	66.15	62.46	80.41	79.40
Grad-Blending	68.81	69.34	67.31	67.68	67.40	63.87	81.73	80.84
OGM-GE [ 24 ]	64.34	64.93	66.35	66.76	65.62	62.97	81.15	80.36

Table 1: Comparison with state-of-the-art methods on the benchmarks (%). Mean and standard deviation are reported over 5 folds.

Methods	CMU-MOSI					CMU-MOSEI
Methods	MAE ( $\downarrow$ )	Corr ( $\uparrow$ )	Acc-2(%)	Acc-7(%)	F1(%)	MAE ( $\downarrow$ )	Corr ( $\uparrow$ )	Acc-2(%)	Acc-7(%)	F1(%)
MulT [ 29 ]	0.846	0.725	81.70	40.05	81.66	0.673	0.677	80.85	48.37	80.86
MISA [ 12 ]	0.788	0.744	82.07	41.27	82.43	0.594	0.724	82.03	51.43	82.13
Self-MM [ 39 ]	0.765	0.764	82.88	42.03	83.04	0.576	0.732	82.43	52.68	82.47
FDMER [ 36 ]	0.760	0.777	83.01	42.88	83.22	0.571	0.743	83.88	53.21	83.35
PMR [ 7 ]	0.895	0.689	79.88	40.60	79.83	0.645	0.689	81.57	48.88	81.56

Table 2: Performance Comparison.

\uparrow

and

\downarrow

indicate that higher and lower value is better. Mean and StDev. are reported over 5 folds.

Importance of Modality
Methods	MOSI		KS		MOSEI
Methods	MAE $\downarrow$	ACC 7	ACC	F1	MAE $\downarrow$	ACC 7
CLCR (Ours)	0.678	49.56	75.41	75.32	0.511	56.32
w/o Language	0.970	36.2	–	–	0.803	42.96
w/o Acoustic	0.705	47.6	52.36	52.67	0.538	54.36
w/o Visual	0.710	47.1	48.67	48.89	0.543	53.86

Table 3: Ablation studies of CLCR on three benchmarks (%).

\ddagger

denotes removing both

\mathcal{L}_{\mathrm{Inter}}

and

\mathcal{L}_{\mathrm{Intra}}

Figure 3: Qualitative analysis: t-SNE of MOSI representations. CLCR yields more compact, better separated clusters than ablated variants, indicating improved semantic alignment.

Figure 4: Visualization of semantic-level importance and learned weights on MOSI and KS. The shallow, mid, and deep levels are used in a complementary rather than redundant way, supporting the necessity of the semantic hierarchy.

Figure 5: Robustness analysis under Gaussian noise. performance curves over more than 10 random runs with different noisy data on MOSI (a) and KS (b), $\downarrow$ indicates that lower values are better.

Figure 6: Sparsity–performance trade-off of IntraCED token budgets. The best appears near $\gamma\!\approx\!1$ ( $r\!\approx\!0.68$ ), indicating that a moderate sparsity level yields optimal cross-modal exchange.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Sensitivity of CLCR to intra-level regularization on MOSI and KS. Each plot varies the global intra-level weight $\lambda_{\mathrm{intra}}$ or the separation factor $\lambda_{\mathrm{sp}}$ in $\mathcal{L}_{\mathrm{Intra}}$ , performance improves once the constraint is activated and remains stable for moderate values.

Figure 8: Sensitivity of CLCR to the internal weights of the inter-level regularizer on CMU-MOSI and KS. Each plot varies one of $\alpha_{\mathrm{pr}}$ , $\alpha_{\mathrm{sp}}$ , or $\alpha_{\mathrm{mix}}$ while fixing the others, removing any component degrades performance, whereas weights in $[0.5,1.5]$ give similar results.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 208 papers

May 12, 2026

Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

Multimodal learning seeks to integrate information across diverse sensory sources, yet current approaches struggle to balance cross-modal generalizability with modality-specific structure. Continuous (implicit) methods preserve fine-grained priors but render generalization challenging, while discrete (explicit) approaches enforce shared prototypes at the expense of modality specificity. We introduce CoDAAR (Cross-modal Discrete Alignment And Reconstruction), a novel framework that resolves this long-standing trade-off by establishing semantic consensus across modality-specific codebooks through index-level alignment. This design uniquely allows CoDAAR to preserve modality-unique structures while achieving generalizable cross-modal representations within a unified discrete space. CoDAAR combines two complementary mechanisms: Discrete Temporal Alignment (DTA), which enables fine-grained temporal quantization, and Cascading Semantic Alignment (CSA), which promotes progressive cross-modal semantic agreement. Together, they establish a competition-free unified representation space. Trained with self-supervised reconstruction objectives on paired multimodal sequences, CoDAAR demonstrates robust cross-modal and cross-domain generalization. Across Cross-Modal Generalization benchmarks, including event classification, localization, video segmentation, and cross-dataset transfer, CoDAAR achieves state-of-the-art performance, establishing a new paradigm for discrete and generalizable multimodal representation learning.

Souptik Sen, Raneen Younis, Zahra Ahmadi

Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

Multimodal Learning Representation Learning Cross-Modal Representation Learning Multimodal Domain Generalization Cross-Modal Alignment Multimodal SSL+3

Ask PDF

Feb 16, 2026

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Multimodal learning seeks to integrate information from heterogeneous sources, where signals may be shared across modalities, specific to individual modalities, or emerge only through their interaction. While self-supervised multimodal contrastive learning has achieved remarkable progress, most existing methods predominantly capture redundant cross-modal signals, often neglecting modality-specific (unique) and interaction-driven (synergistic) information. Recent extensions broaden this perspective, yet they either fail to explicitly model synergistic interactions or learn different information components in an entangled manner, leading to incomplete representations and potential information leakage. We introduce COrAL, a principled framework that explicitly and simultaneously preserves redundant, unique, and synergistic information within multimodal representations. COrAL employs a dual-path architecture with orthogonality constraints to disentangle shared and modality-specific features, ensuring a clean separation of information components. To promote synergy modeling, we introduce asymmetric masking with complementary view-specific patterns, compelling the model to infer cross-modal dependencies rather than rely solely on redundant cues. Extensive experiments on synthetic benchmarks and diverse MultiBench datasets demonstrate that COrAL consistently matches or outperforms state-of-the-art methods while exhibiting low performance variance across runs. These results indicate that explicitly modeling the full spectrum of multimodal information yields more stable, reliable, and comprehensive embeddings.

Carolin Cissee, Raneen Younis, Zahra Ahmadi

Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

Multimodal Embedding Multimodal Contrastive Learning Multimodal Disentangled Representation Learning

Ask PDF

Mar 19, 2026

Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus

Federated learning (FL) enables collaborative model training across distributed devices without sharing raw data, but applying FL to multi-modal settings introduces significant challenges. Clients typically possess heterogeneous modalities and model architectures, making it difficult to align feature spaces efficiently while preserving privacy and minimizing communication costs. To address this, we introduce CoMFed, a Communication-Efficient Multi-Modal Federated Learning framework that uses learnable projection matrices to generate compressed latent representations. A latent-space regularizer aligns these representations across clients, improving cross-modal consistency and robustness to outliers. Experiments on human activity recognition benchmarks show that CoMFed achieves competitive accuracy with minimal overhead.

Mohamed Badi, Chaouki Ben Issaid, Mehdi Bennis

Center for Wireless Communications, University of Oulu, Oulu 90014, Finland

Multimodal Federated Learning Federated Learning Heterogeneous Federated Learning Multimodal Robustness Cross-Modal Representation Learning

Ask PDF

Abstract

Cross-Modal Representation Learning

Explore similar work

Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus