Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China
Multimodal Sentiment Analysis (MSA) requires effective modeling of cross-modal interactions and contextual dependencies while remaining computationally efficient. Existing fusion approaches predominantly rely on Transformer-based cross-modal attention, which incurs quadratic complexity with respect to sequence length and limits scalability. Moreover, contextual information from preceding utterances is often incorporated through concatenation or independent fusion, without explicit temporal modeling that captures sentiment evolution across dialogue turns. To address these limitations, we propose CAGMamba, a context-aware gated cross-modal Mamba framework for dialogue-based sentiment analysis. Specifically, we organize the contextual and the current-utterance features into a temporally ordered binary sequence, which provides Mamba with explicit temporal structure for modeling sentiment evolution. To further enable controllable cross-modal integration, we propose a Gated Cross-Modal Mamba Network (GCMN) that integrates cross-modal and unimodal paths via learnable gating to balance information fusion and modality preservation, and is trained with a three-branch multi-task objective over text, audio, and fused predictions. Experiments on three benchmark datasets demonstrate that CAGMamba achieves state-of-the-art or competitive results across multiple evaluation metrics. All codes are available at https://github.com/User2024-xj/CAGMamba.
Figures & tables
Figure 1: Overall Framework of the Proposed CAGMamba Model.
Figure 2: Architecture of the GCMN module. The upper part shows the three-stream gated fusion structure. The lower part illustrates the internal architecture of the BSSM.
Method
CMU-MOSI
CMU-MOSEI
Acc-2
F1
Acc-7
MAE
Corr
Acc-2
F1
Acc-7
MAE
Corr
MISA
81.80/83.40
81.70/83.60
42.30
0.78
0.78
83.60/85.50
83.80/85.30
52.20
0.56
0.76
Self-MM
83.44/85.46
83.36/85.43
46.67
0.71
0.80
83.76/85.15
83.82/84.90
53.87
0.53
0.77
MMIM
84.14/86.06
84.00/85.98
46.65
0.70
0.80
82.24/85.97
82.66/85.94
54.24
0.53
0.77
ConFEDE
84.17/85.52
84.13/85.52
42.27
0.74
0.78
81.65/85.82
82.17/85.83
54.86
0.52
0.78
CLGSI
83.97/86.43
83.63/86.25
49.67
0.70
0.79
84.01/86.32
84.21/86.18
54.56
0.53
0.76
Table 1: Comparison with representative baselines. Acc-2 reports two values: negative/non-negative and negative/positive classification accuracy, respectively. The best results are marked in bold , while the second-best results are underlined.
Method
Acc-5 ↑
Acc-3 ↑
Acc-2 ↑
F1 ↑
MAE ↓
Corr ↑
LMF
40.53
64.68
77.77
77.88
0.441
0.576
TFN
39.30
65.12
78.38
78.62
0.432
0.591
MulT
37.94
64.77
78.56
79.66
0.453
0.564
MISA
-
-
76.54
76.59
0.447
0.563
MAG-BERT
-
-
74.44
71.75
0.492
0.399
Self-MM
41.53
65.47
80.04
80.44
0.425
0.595
Table 2: Comparison with representative baselines. The best results are marked in bold , while the second-best results are underlined.
Experiment
Configuration
Acc-2 ↑
Context
MMML (no context)
86.31
MMML + concat
87.17
GCMN + concat
87.31
GCMN + gated seq. (ours)
88.19
Fusion
Single-path (no gate)
86.21
Dual-path + fixed gate
87.31
Table 3: Ablation study evaluating the impact of contextual modeling and fusion strategies. The best results are marked in bold , while the second-best results are underlined.
(Kt,Ka)
(0, 0)
(1, 0)
(0, 1)
(1, 1)
(1, 2)
(2, 2)
(3, 1)
(2, 1)
Acc-2 ↑
86.33
86.98
86.72
87.54
87.15
87.63
87.95
88.19
Table 4: Ablation study on context window size on CMU-MOSI. Kt and Ka denote the number of preceding utterances used as context for text and audio, respectively. The best result is marked in bold and the second-best is underlined .
Method
Params
FLOPs
Acc-2 ↑
Vision Mamba
1.40M
0.12G
84.97/85.49
Video Mamba
1.41M
0.13G
85.74/86.89
Transformer
4.45M
0.38G
84.74/86.20
MSAmba
1.41M
0.13G
85.99 / 87.43
GCMN (ours)
0.80M
0.10G
88.19 / 88.15
Table 5: Efficiency comparison in terms of model size (Params), computational cost (FLOPs), and performance (Acc-2). The best results are marked in bold , while the second-best results are underlined.
Figure 3: T-SNE visualization of the learned representations on CMU-MOSI. Compared to baseline methods, our model produces more compact intra-class clusters and better inter-class separation, indicating improved representation quality.
In the era of large-scale pre-trained models, effectively adapting general knowledge to specific affective computing tasks remains a challenge, particularly regarding computational efficiency and multimodal heterogeneity. While Transformer-based methods have excelled at modeling inter-modal dependencies, their quadratic computational complexity limits their use with long-sequence data. Mamba-based models have emerged as a computationally efficient alternative; however, their inherent sequential scanning mechanism struggles to capture the global, non-sequential relationships that are crucial for effective cross-modal alignment. To address these limitations, we propose AlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis. Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy, promoting geometric and statistical consistency between modalities without incurring any inference-time overhead. More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts to explicitly handle data heterogeneity during the fusion process. Extensive experiments on four challenging benchmarks, including dynamic time-series (on the CMU-MOSI and CMU-MOSEI datasets) and static image-related tasks (on the NYU-Depth V2 and MVSA-Single datasets), demonstrate that AlignMamba-2 establishes a new state-of-the-art in both effectiveness and efficiency across diverse pattern recognition tasks, ranging from dynamic time-series analysis to static image-text classification.
Yan Li, Yifei Xing, Xiangyuan Lan +3
Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China
Multimodal Sentiment Analysis (MSA) integrates language, visual, and acoustic modalities to infer human sentiment. Most existing methods either focus on globally shared representations or modality-specific features, while overlooking signals that are shared only by certain modality pairs. This limits the expressiveness and discriminative power of multimodal representations. To address this limitation, we propose a Tri-Subspace Disentanglement (TSD) framework that explicitly factorizes features into three complementary subspaces: a common subspace capturing global consistency, submodally-shared subspaces modeling pairwise cross-modal synergies, and private subspaces preserving modality-specific cues. To keep these subspaces pure and independent, we introduce a decoupling supervisor together with structured regularization losses. We further design a Subspace-Aware Cross-Attention (SACA) fusion module that adaptively models and integrates information from the three subspaces to obtain richer and more robust representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that TSD achieves state-of-the-art performance across all key metrics, reaching 0.691 MAE on CMU-MOSI and 54.9% ACC-7 on CMU-MOSEI, and also transfers well to multimodal intent recognition tasks. Ablation studies confirm that tri-subspace disentanglement and SACA jointly enhance the modeling of multi-granular cross-modal sentiment cues.
Chunlei Meng, Jiabin Luo, Zhenglin Yan +4
Fudan University · Peking University · University of Macau
Multimodal sentiment analysis integrates textual, acoustic and visual cues, yet current language-model-based fusion methods typically leave prediction-layer branch allocation implicit. We introduce Branch-Calibrated Multimodal Language Fusion (BC-MLF), which explicitly models prediction-layer branch allocation through a Branch-Calibrated Task Head (BCHead), complemented by Fusion Token Contrastive Learning (FTCL) for sentiment-aware fusion-token regularization. FTCL organizes mean-pooled fusion-token representations according to continuous sentiment affinity, while BCHead combines fusion, text and audiovisual predictions through a lightweight sample-adaptive constrained mixture. Without modifying the fusion backbone, BC-MLF consistently improves the reproduced DeepMLF baseline and achieves the strongest results among the compared methods on CMU-MOSEI and CH-SIMS across classification and regression metrics. The controlled ablations show that sample-adaptive prediction-layer branch allocation consistently outperforms static branch aggregation. Code is available at https://github.com/sunyulin0421/BC-MLF.