Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

Kang He; Yuzhe Ding; Xinrong Wang; Fei Li; Chong Teng; Donghong Ji

cs.CLApr 14, 2026

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

Authors: Kang He, Yuzhe Ding, Xinrong Wang, Fei Li, Chong Teng, Donghong Ji

Organizations: Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

Abstract

Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.

Figures & tables

Figure 1: Illustration of modality imbalance: text tends to dominate learning while weaker modalities are easily suppressed and affected by noisy samples. Our two-stage framework first performs modality disentanglement and compensation (Stage I), then applies an energy-based gradient flow for modality coordination and sample-aware reliability for trusting (Stage II), leading to balanced multimodal optimization and ensure robustness.

Figure 2: The overall architecture of our proposed model EBMC.

Figure 3: T-SNE visualization of features distrubution on CMU-MOSI. The closer color is to red, the more positive sentiment. EBMC demonstrates superior performance in MSA. Please refer to Sec. 4.4 .

Figure 4: The impact of EMC on modality contributions and overall performance. The dual y-axes show the average contribution of each modality (left) and the corresponding Acc-2 score (right). Please refer to Sec. 4.4 . Best viewed in color.

Figure 5: Comparison of EBMC and baseline predictions on the test sets of IEMOCAP. Refer to Sec. 4.5 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Training Time	Params
SelfMM	3.73h	121,835,723
GLoMo	2.29h	129,818,887
EUAR	1.16h	110,436,422
EBMC ${}_{\textrm{online}}$	1.04h	112,320,746
EBMC ${}_{\textrm{offline}}$	5min	6,384,400

Appendix

Table 6: Comparison of training efficiency and parameters across different baselines on CMU-MOSI dataset. Please see Appendix A.4 for details.

Figure 6: Hyperparameter sensitivity analysis of EBMC on the CMU-MOSEI dataset. The effects of varying

\beta

\gamma

\eta

, and

\zeta

are evaluated according to the overall training objective defined in Eq. ( 17 ), while fixing the remaining hyperparameters to the values used in our main experiments. Please refer to Appendix C.1 .

Figure 7: Performance curves on the CMU-MOSI dataset under increasing modality missing rates. The four subplots respectively report F1, Acc-7, Corr, and MAE across all compared models. Note that F1, Acc-7, and Corr are higher-is-better metrics, whereas MAE is a lower-is-better metric. Please refer to Sec. 4.4 (Q3) and Appendix C.3 .

Figure 8: Performance curves on the CMU-MOSEI dataset under increasing modality missing rates. The four subplots respectively report F1, Acc-7, Corr, and MAE across all compared models. Note that F1, Acc-7, and Corr are higher-is-better metrics, whereas MAE is a lower-is-better metric. Please refer to Sec. 4.4 (Q3) and Appendix C.3 .

Explore similar work

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
Sep 24, 2026Menghua Jiang, Haokai Gao, Xiangui Kang +2Multimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
May 16, 2026Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3Missing-Modality Learning Multimodal Robustness
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute

Sep 24, 2026J/K move · Enter open · S save

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Menghua Jiang, Haokai Gao, Xiangui Kang, Haifeng Hu+1

School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University

Multimodal Sentiment Analysis Multimodal Robustness Uncertainty-Aware Multimodal Fusion Missing-Modality Learning

Open paper AskPDF arXiv

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

Abstract

Explore similar work

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data