Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, while overlooking signals that are shared only by certain modality pairs. This limits the expressiveness and discriminative power of multimodal representations. To address this limitation, we propose a Tri-Subspace Disentanglement (TSD) framework that explicitly factorizes features into three complementary subspaces: a common subspace capturing global consistency, submodally-shared subspaces modeling pairwise cross-modal synergies, and private subspaces preserving modality-specific cues. To keep these subspaces pure and independent, we introduce a decoupling supervisor together with structured regularization losses. We further design a Subspace-Aware Cross-Attention (SACA) fusion module that adaptively models and integrates information from the three subspaces to obtain richer and more robust representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that TSD achieves state-of-the-art performance across all key metrics, reaching 0.691 MAE on CMU-MOSI and 54.9% ACC-7 on CMU-MOSEI, and also transfers well to multimodal intent recognition tasks. Ablation studies confirm that tri-subspace disentanglement and SACA jointly enhance the modeling of multi-granular cross-modal sentiment cues.
Figures & tables
Figure 1: An example of a submodally shared cue: the utterance “that’s really great” is delivered with a sarcastic tone and a disdainful facial expression. While the lexical content suggests positive sentiment, the conflicting acoustic and visual cues jointly convey negative affect (sarcasm).
Figure 2: Overview of the proposed TSD framework. Given multimodal inputs, TSD disentangles features into three complementary subspaces and fuses them via a Subspace-Aware Cross-Attention (SACA) module.
Methods
CMU-MOSI
CMU-MOSEI
MAE( ↓ )
ACC 7 (%)
ACC 2 (%)
F1(%)
MAE( ↓ )
ACC 7 (%)
ACC 2 (%)
F1(%)
EF-LSTM*
1.386 / 1.420
33.7 / 31.0
75.3 / 73.6
75.2 / 74.5
0.620 / 0.594
47.4 / 46.3
78.2 / 76.1
77.9 / 75.9
TFN* [ 27 ]
0.953 / 0.995
31.9 / 35.3
78.8 / 76.5
78.9 / 76.6
0.574 / 0.573
50.9 / 50.2
80.4 / 84.2
80.7 / 84.0
LMF* [ 11 ]
0.931 / 0.963
36.9 / 31.1
78.7 / 79.1
78.7 / 79.1
0.564 / 0.565
52.3 / 51.9
84.7 / 83.8
84.5 / 83.9
MFN* [ 3 ]
0.964 / 0.971
35.6 / 34.7
78.4 / 80.0
78.4 / 80.1
0.574 / 0.567
50.8 / 51.3
84.0 / 83.2
84.0 / 83.3
MuLT [ 21 ]
0.936 / 0.933
35.1 / 33.2
80.0 / 80.3
80.1 / 80.3
0.572 / 0.556
52.3 / 53.2
82.7 / 84.0
82.8 / 84.0
Table 1: Performance comparison on CMU-MOSI and CMU-MOSEI. Each cell reports results under aligned/unaligned settings as a/b ( a : aligned, b : unaligned). *: values from EMOE [ 3 ] . The last row lists the ± standard deviation of TSD over n=5 random seeds.
Method
Accuracy
F1
Precision
Recall
MAG-BERT [ 18 ]
70.34
68.19
68.31
69.36
MMIM [ 8 ]
71.21
68.70
69.20
68.90
MuLT [ 21 ]
72.58
69.36
70.73
69.47
MISA [ 6 ]
72.36
70.57
71.24
70.41
CAGC [ 20 ]
73.03
70.62
70.86
70.55
EMOE [ 3 ]
72.58
70.73
72.08
70.86
Table 2: Performance comparison on the MIntRec dataset (%).
Model
MOSI
MOSEI
MIntRec
MAE
ACC 7
MAE
ACC 7
ACC
TSD † (Ours)
0.703
47.9
0.535
54.3
–
TSD (Ours)
0.691
48.0
0.530
54.2
73.67
Importance of Modality
w/o Linguistic
1.010
35.5
0.830
39.8
54.3
w/o Acoustic
0.810
45.0
0.620
51.2
69.3
Table 3: Ablation studies of TSD on the three benchmarks. For MOSI and MOSEI we report MAE ( ↓ ) and ACC 7 (%); for MIntRec we report ACC (%). † denotes the aligned setting.
Figure 3: Qualitative examples illustrating that incorporating the submodally shared subspace enables TSD to better capture cross-modal cues (e.g., sarcasm), producing sentiment predictions closer to the ground truth.
Figure 4: t-SNE visualization of feature distributions on CMU-MOSI. Colors indicate sentiment polarity from negative (dark) to positive (yellow). The full TSD model exhibits a clearer sentiment gradient than the variant without SACA and sub-shared subspaces.
Figure 5: (a) Average fusion weights of each subspace (Common, Private, Sub-Shared) in TSD on CMU-MOSI and CMU-MOSEI. (b) Estimated contribution of each subspace during fusion. Higher values and darker colors denote greater subspace significance.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
TSD
p -value (Holm)
CMU-MOSI (unaligned)
MAE ( ↓ )
0.691 ± 0.005
0.08
ACC 7 (%)
49.0 ± 0.3
0.006
ACC 2 (%)
86.5 ± 0.1
0.001
F1 (%)
86.6 ± 0.3
0.002
CMU-MOSEI (unaligned)
Appendix
Table 4: Statistical significance of TSD. Means and standard deviations for TSD are computed over n=5 independent seeds. p -values are obtained from seed-wise paired t -tests comparing TSD against the strongest prior method on the same metric and setting, followed by Holm correction for multiple comparisons within each block.
Figure 6: Sensitivity analysis of loss coefficients ( λ1,λ2,λ3,λ4 ) on the CMU-MOSI dataset. Model performance (MAE and ACC 7 ) remains stable as each parameter is varied, illustrating the robustness of TSD to hyperparameter selection.
Figure 7: Sensitivity analysis of loss coefficients ( λ1,λ2,λ3,λ4 ) on the CMU-MOSEI dataset. Model performance (MAE and ACC 7 ) remains stable as each parameter is varied, illustrating the robustness of TSD to hyperparameter selection.
Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.
Wenhao Li, Zhibin Wu, Chong Xiao +1
Software School, Shandong University · Shenzhen Loop Area Institute
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.
Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.
Kang He, Yuzhe Ding, Xinrong Wang +3
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute