Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis

Chunlei Meng; Jiabin Luo; Zhenglin Yan; Zhenyu Yu; Rong Fu; Zhongxue Gan; Chun Ouyang

cs.MMFeb 23, 2026

Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis

Authors: Chunlei Meng, Jiabin Luo, Zhenglin Yan, Zhenyu Yu, Rong Fu, Zhongxue Gan, Chun Ouyang

Organizations: Fudan University · Peking University · University of Macau

Abstract

Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, while overlooking signals that are shared only by certain modality pairs. This limits the expressiveness and discriminative power of multimodal representations. To address this limitation, we propose a Tri-Subspace Disentanglement (TSD) framework that explicitly factorizes features into three complementary subspaces: a common subspace capturing global consistency, submodally-shared subspaces modeling pairwise cross-modal synergies, and private subspaces preserving modality-specific cues. To keep these subspaces pure and independent, we introduce a decoupling supervisor together with structured regularization losses. We further design a Subspace-Aware Cross-Attention (SACA) fusion module that adaptively models and integrates information from the three subspaces to obtain richer and more robust representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that TSD achieves state-of-the-art performance across all key metrics, reaching 0.691 MAE on CMU-MOSI and 54.9% ACC-7 on CMU-MOSEI, and also transfers well to multimodal intent recognition tasks. Ablation studies confirm that tri-subspace disentanglement and SACA jointly enhance the modeling of multi-granular cross-modal sentiment cues.

Figures & tables

Figure 1: An example of a submodally shared cue: the utterance “that’s really great” is delivered with a sarcastic tone and a disdainful facial expression. While the lexical content suggests positive sentiment, the conflicting acoustic and visual cues jointly convey negative affect (sarcasm).

Figure 2: Overview of the proposed TSD framework. Given multimodal inputs, TSD disentangles features into three complementary subspaces and fuses them via a Subspace-Aware Cross-Attention (SACA) module.

Methods	CMU-MOSI				CMU-MOSEI
Methods	MAE( $\downarrow$ )	ACC 7 (%)	ACC 2 (%)	F1(%)	MAE( $\downarrow$ )	ACC 7 (%)	ACC 2 (%)	F1(%)
EF-LSTM*	1.386 / 1.420	33.7 / 31.0	75.3 / 73.6	75.2 / 74.5	0.620 / 0.594	47.4 / 46.3	78.2 / 76.1	77.9 / 75.9
TFN* [ 27 ]	0.953 / 0.995	31.9 / 35.3	78.8 / 76.5	78.9 / 76.6	0.574 / 0.573	50.9 / 50.2	80.4 / 84.2	80.7 / 84.0
LMF* [ 11 ]	0.931 / 0.963	36.9 / 31.1	78.7 / 79.1	78.7 / 79.1	0.564 / 0.565	52.3 / 51.9	84.7 / 83.8	84.5 / 83.9
MFN* [ 3 ]	0.964 / 0.971	35.6 / 34.7	78.4 / 80.0	78.4 / 80.1	0.574 / 0.567	50.8 / 51.3	84.0 / 83.2	84.0 / 83.3
MuLT [ 21 ]	0.936 / 0.933	35.1 / 33.2	80.0 / 80.3	80.1 / 80.3	0.572 / 0.556	52.3 / 53.2	82.7 / 84.0	82.8 / 84.0

Table 1: Performance comparison on CMU-MOSI and CMU-MOSEI. Each cell reports results under aligned/unaligned settings as

a/b

(

a

: aligned,

b

: unaligned). *: values from EMOE [ 3 ] . The last row lists the

\pm

standard deviation of TSD over

n=5

random seeds.

Method	Accuracy	F1	Precision	Recall
MAG-BERT [ 18 ]	70.34	68.19	68.31	69.36
MMIM [ 8 ]	71.21	68.70	69.20	68.90
MuLT [ 21 ]	72.58	69.36	70.73	69.47
MISA [ 6 ]	72.36	70.57	71.24	70.41
CAGC [ 20 ]	73.03	70.62	70.86	70.55
EMOE [ 3 ]	72.58	70.73	72.08	70.86

Table 2: Performance comparison on the MIntRec dataset (%).

Model	MOSI		MOSEI		MIntRec
Model	MAE	ACC 7	MAE	ACC 7	ACC
TSD † (Ours)	0.703	47.9	0.535	54.3	–
TSD (Ours)	0.691	48.0	0.530	54.2	73.67
Importance of Modality
w/o Linguistic	1.010	35.5	0.830	39.8	54.3
w/o Acoustic	0.810	45.0	0.620	51.2	69.3

Table 3: Ablation studies of TSD on the three benchmarks. For MOSI and MOSEI we report MAE (

\downarrow

) and ACC 7 (%); for MIntRec we report ACC (%).

\dagger

denotes the aligned setting.

Figure 3: Qualitative examples illustrating that incorporating the submodally shared subspace enables TSD to better capture cross-modal cues (e.g., sarcasm), producing sentiment predictions closer to the ground truth.

Figure 4: t-SNE visualization of feature distributions on CMU-MOSI. Colors indicate sentiment polarity from negative (dark) to positive (yellow). The full TSD model exhibits a clearer sentiment gradient than the variant without SACA and sub-shared subspaces.

Figure 5: (a) Average fusion weights of each subspace (Common, Private, Sub-Shared) in TSD on CMU-MOSI and CMU-MOSEI. (b) Estimated contribution of each subspace during fusion. Higher values and darker colors denote greater subspace significance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	TSD	$p$ -value (Holm)
CMU-MOSI (unaligned)
MAE ( $\downarrow$ )	0.691 $\pm$ 0.005	0.08
ACC 7 (%)	49.0 $\pm$ 0.3	0.006
ACC 2 (%)	86.5 $\pm$ 0.1	0.001
F1 (%)	86.6 $\pm$ 0.3	0.002
CMU-MOSEI (unaligned)

Appendix

Table 4: Statistical significance of TSD. Means and standard deviations for TSD are computed over

n=5

independent seeds.

p

-values are obtained from seed-wise paired

t

-tests comparing TSD against the strongest prior method on the same metric and setting, followed by Holm correction for multiple comparisons within each block.

Figure 6: Sensitivity analysis of loss coefficients ( $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ ) on the CMU-MOSI dataset. Model performance (MAE and ACC 7 ) remains stable as each parameter is varied, illustrating the robustness of TSD to hyperparameter selection.

Figure 7: Sensitivity analysis of loss coefficients ( $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ ) on the CMU-MOSEI dataset. Model performance (MAE and ACC 7 ) remains stable as each parameter is varied, illustrating the robustness of TSD to hyperparameter selection.

Multimodal Sentiment Analysis

Multimodal sentiment analysis combines language, voice, and visual cues to infer sentiment, polarity, and emotional intensity.

See all 53 papers

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
May 16, 2026Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3Missing-Modality Learning Multimodal Robustness
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

Sep 24, 2026J/K move · Enter open · S save

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang

Software School, Shandong University · Shenzhen Loop Area Institute

Missing-Modality Learning Multimodal Robustness Cross-Modal Alignment Multimodal Sentiment Analysis

Open paper AskPDF arXiv

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.

Abstract

Multimodal Sentiment Analysis

Explore similar work

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis