CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

Chunlei Meng; Guanhong Huang; Rong Fu; Runmin Jian; Zhongxue Gan; Chun Ouyang

cs.CVFeb 23, 2026

CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

Authors: Chunlei Meng, Guanhong Huang, Rong Fu, Runmin Jian, Zhongxue Gan, Chun Ouyang

Organizations: Fudan University · Shantou University · University of Macau · Guangzhou Huashang College

Abstract

Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.

Figures & tables

Figure 1: Cross-level semantic asynchrony and CLCR. Top: mixing across levels causes semantic confusion and mismatch. bottom: CLCR structures each modality into three aligned levels and restricts exchange to the learned shared subspace at matched levels, enabling level semantic alignment and reliable aggregation.

Figure 2: Overview of the proposed CLCR framework. Each modality is organized into a three-level semantic hierarchy. IntraCED performs budgeted cross-modal exchange in an explicitly disentangled level-shared subspace, while InterCAD synchronizes and aggregates shared and private streams across levels into the final task representation.

Methods	CREMA-D [ 4 ]		KS [ 1 ]		AVE [ 33 ]		UCF101 [ 26 ]
Methods	Acc(%)	F1(%)	Acc(%)	F1(%)	Acc(%)	F1(%)	Acc(%)	F1(%)
only-Acoustic	57.27	57.89	48.67	48.89	62.16	58.54	-	-
only-Visual	62.17	62.78	52.36	52.67	31.40	29.87	-	-
Concat	58.83	59.43	64.97	65.21	66.15	62.46	80.41	79.40
Grad-Blending	68.81	69.34	67.31	67.68	67.40	63.87	81.73	80.84
OGM-GE [ 24 ]	64.34	64.93	66.35	66.76	65.62	62.97	81.15	80.36

Table 1: Comparison with state-of-the-art methods on the benchmarks (%). Mean and standard deviation are reported over 5 folds.

Methods	CMU-MOSI					CMU-MOSEI
Methods	MAE ( $\downarrow$ )	Corr ( $\uparrow$ )	Acc-2(%)	Acc-7(%)	F1(%)	MAE ( $\downarrow$ )	Corr ( $\uparrow$ )	Acc-2(%)	Acc-7(%)	F1(%)
MulT [ 29 ]	0.846	0.725	81.70	40.05	81.66	0.673	0.677	80.85	48.37	80.86
MISA [ 12 ]	0.788	0.744	82.07	41.27	82.43	0.594	0.724	82.03	51.43	82.13
Self-MM [ 39 ]	0.765	0.764	82.88	42.03	83.04	0.576	0.732	82.43	52.68	82.47
FDMER [ 36 ]	0.760	0.777	83.01	42.88	83.22	0.571	0.743	83.88	53.21	83.35
PMR [ 7 ]	0.895	0.689	79.88	40.60	79.83	0.645	0.689	81.57	48.88	81.56

Table 2: Performance Comparison.

\uparrow

and

\downarrow

indicate that higher and lower value is better. Mean and StDev. are reported over 5 folds.

Importance of Modality
Methods	MOSI		KS		MOSEI
Methods	MAE $\downarrow$	ACC 7	ACC	F1	MAE $\downarrow$	ACC 7
CLCR (Ours)	0.678	49.56	75.41	75.32	0.511	56.32
w/o Language	0.970	36.2	–	–	0.803	42.96
w/o Acoustic	0.705	47.6	52.36	52.67	0.538	54.36
w/o Visual	0.710	47.1	48.67	48.89	0.543	53.86

Table 3: Ablation studies of CLCR on three benchmarks (%).

\ddagger

denotes removing both

\mathcal{L}_{\mathrm{Inter}}

and

\mathcal{L}_{\mathrm{Intra}}

Figure 3: Qualitative analysis: t-SNE of MOSI representations. CLCR yields more compact, better separated clusters than ablated variants, indicating improved semantic alignment.

Figure 4: Visualization of semantic-level importance and learned weights on MOSI and KS. The shallow, mid, and deep levels are used in a complementary rather than redundant way, supporting the necessity of the semantic hierarchy.

Figure 5: Robustness analysis under Gaussian noise. performance curves over more than 10 random runs with different noisy data on MOSI (a) and KS (b), $\downarrow$ indicates that lower values are better.

Figure 6: Sparsity–performance trade-off of IntraCED token budgets. The best appears near $\gamma\!\approx\!1$ ( $r\!\approx\!0.68$ ), indicating that a moderate sparsity level yields optimal cross-modal exchange.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Sensitivity of CLCR to intra-level regularization on MOSI and KS. Each plot varies the global intra-level weight $\lambda_{\mathrm{intra}}$ or the separation factor $\lambda_{\mathrm{sp}}$ in $\mathcal{L}_{\mathrm{Intra}}$ , performance improves once the constraint is activated and remains stable for moderate values.

Figure 8: Sensitivity of CLCR to the internal weights of the inter-level regularizer on CMU-MOSI and KS. Each plot varies one of $\alpha_{\mathrm{pr}}$ , $\alpha_{\mathrm{sp}}$ , or $\alpha_{\mathrm{mix}}$ while fixing the others, removing any component degrades performance, whereas weights in $[0.5,1.5]$ give similar results.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 208 papers

Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiMultimodal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus
Mar 19, 2026Mohamed Badi, Chaouki Ben Issaid, Mehdi BennisMultimodal Federated Learning Federated Learning
Center for Wireless Communications, University of Oulu, Oulu 90014, Finland

May 12, 2026J/K move · Enter open · S save

Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

Souptik Sen, Raneen Younis, Zahra Ahmadi

Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

Multimodal Learning Representation Learning Cross-Modal Representation Learning Multimodal Domain Generalization Cross-Modal Alignment Multimodal SSL+3

Open paper AskPDF arXiv

Multimodal learning seeks to integrate information across diverse sensory sources, yet current approaches struggle to balance cross-modal generalizability with modality-specific structure. Continuous (implicit) methods preserve fine-grained priors but render generalization challenging, while discrete (explicit) approaches enforce shared prototypes at the expense of modality specificity. We introduce CoDAAR (Cross-modal Discrete Alignment And Reconstruction), a novel framework that resolves this long-standing trade-off by establishing semantic consensus across modality-specific codebooks through index-level alignment. This design uniquely allows CoDAAR to preserve modality-unique structures while achieving generalizable cross-modal representations within a unified discrete space. CoDAAR combines two complementary mechanisms: Discrete Temporal Alignment (DTA), which enables fine-grained temporal quantization, and Cascading Semantic Alignment (CSA), which promotes progressive cross-modal semantic agreement. Together, they establish a competition-free unified representation space. Trained with self-supervised reconstruction objectives on paired multimodal sequences, CoDAAR demonstrates robust cross-modal and cross-domain generalization. Across Cross-Modal Generalization benchmarks, including event classification, localization, video segmentation, and cross-dataset transfer, CoDAAR achieves state-of-the-art performance, establishing a new paradigm for discrete and generalizable multimodal representation learning.

Abstract

Cross-Modal Representation Learning

Explore similar work

Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus