Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Authors: Kang He, Yuzhe Ding, Xinrong Wang, Fei Li, Chong Teng, Donghong Ji
Organizations: Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute
Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.
Figures & tables
Figure 1: Illustration of modality imbalance: text tends to dominate learning while weaker modalities are easily suppressed and affected by noisy samples. Our two-stage framework first performs modality disentanglement and compensation (Stage I), then applies an energy-based gradient flow for modality coordination and sample-aware reliability for trusting (Stage II), leading to balanced multimodal optimization and ensure robustness.
Figure 2: The overall architecture of our proposed model EBMC.
Figure 3: T-SNE visualization of features distrubution on CMU-MOSI. The closer color is to red, the more positive sentiment. EBMC demonstrates superior performance in MSA. Please refer to Sec. 4.4 .
Figure 4: The impact of EMC on modality contributions and overall performance. The dual y-axes show the average contribution of each modality (left) and the corresponding Acc-2 score (right). Please refer to Sec. 4.4 . Best viewed in color.
Figure 5: Comparison of EBMC and baseline predictions on the test sets of IEMOCAP. Refer to Sec. 4.5 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Training Time
Params
SelfMM
3.73h
121,835,723
GLoMo
2.29h
129,818,887
EUAR
1.16h
110,436,422
EBMC online
1.04h
112,320,746
EBMC offline
5min
6,384,400
Appendix
Table 6: Comparison of training efficiency and parameters across different baselines on CMU-MOSI dataset. Please see Appendix A.4 for details.
Figure 6: Hyperparameter sensitivity analysis of EBMC on the CMU-MOSEI dataset. The effects of varying β , γ , η , and ζ are evaluated according to the overall training objective defined in Eq. ( 17 ), while fixing the remaining hyperparameters to the values used in our main experiments. Please refer to Appendix C.1 .
Figure 7: Performance curves on the CMU-MOSI dataset under increasing modality missing rates. The four subplots respectively report F1, Acc-7, Corr, and MAE across all compared models. Note that F1, Acc-7, and Corr are higher-is-better metrics, whereas MAE is a lower-is-better metric. Please refer to Sec. 4.4 (Q3) and Appendix C.3 .
Figure 8: Performance curves on the CMU-MOSEI dataset under increasing modality missing rates. The four subplots respectively report F1, Acc-7, Corr, and MAE across all compared models. Note that F1, Acc-7, and Corr are higher-is-better metrics, whereas MAE is a lower-is-better metric. Please refer to Sec. 4.4 (Q3) and Appendix C.3 .
Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.
Menghua Jiang, Haokai Gao, Xiangui Kang +2
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.
Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.
Wenhao Li, Zhibin Wu, Chong Xiao +1
Software School, Shandong University · Shenzhen Loop Area Institute