Organizations: Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China
In the era of large-scale pre-trained models, effectively adapting general knowledge to specific affective computing tasks remains a challenge, particularly regarding computational efficiency and multimodal heterogeneity. While Transformer-based methods have excelled at modeling inter-modal dependencies, their quadratic computational complexity limits their use with long-sequence data. Mamba-based models have emerged as a computationally efficient alternative; however, their inherent sequential scanning mechanism struggles to capture the global, non-sequential relationships that are crucial for effective cross-modal alignment. To address these limitations, we propose AlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis. Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy, promoting geometric and statistical consistency between modalities without incurring any inference-time overhead. More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts to explicitly handle data heterogeneity during the fusion process. Extensive experiments on four challenging benchmarks, including dynamic time-series (on the CMU-MOSI and CMU-MOSEI datasets) and static image-related tasks (on the NYU-Depth V2 and MVSA-Single datasets), demonstrate that AlignMamba-2 establishes a new state-of-the-art in both effectiveness and efficiency across diverse pattern recognition tasks, ranging from dynamic time-series analysis to static image-text classification.
Figures & tables
Figure 1: The cross-attention mechanism in Transformer (left) and the scanning mechanism in Mamba (right).
Figure 2: The architecture of AlignMamba-2. The framework consists of three main stages: unimodal encoding, training-time dual alignment regularization (using OT and MMD), and a novel modality-aware Mamba fusion module based on a Mixture-of-Experts design.
Figure 4: GPU memory consumption (in MB) as a function of increasing sequence length. AlignMamba-2 demonstrates superior scalability compared to both Transformer-based and explicit-alignment-based Mamba models.
Figure 5: Inference latency (in milliseconds) as a function of increasing sequence length. AlignMamba-2 maintains a linear increase in inference time, showcasing its efficiency for long-sequence processing.
MOSI
MOSEI
NYUDv2
MVSA
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AlignMamba-2
87.0
87.0
86.5
86.5
73.1
71.5
82.7
80.2
w/o dual alignment loss
84.5
84.4
84.1
84.0
71.0
69.5
79.2
78.1
w/o Mixture-of-Experts
85.8
85.7
85.3
85.2
72.0
70.4
80.5
78.2
w/o both
82.3
82.1
81.8
81.4
69.8
68.7
77.8
77.1
w/ Learnable Routing
86.6
86.4
86.1
86.2
72.8
71.2
82.2
79.5
Table 3: Ablation study of AlignMamba-2 on all four datasets. We report the performance (Accuracy and F1-Score) of the full model and its variants. "w/o" stands for "without", and "w/" stands for "with". Best results are in bold.
Training dataset
MOSI
MOSEI
Test dataset
MOSI
MOSEI
MOSEI
MOSI
Model
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Cobra
80.2
80.0
70.5
70.3
79.5
79.2
70.1
70.0
GeminiFusion
83.2
83.1
74.6
74.6
82.6
82.5
75.0
75.1
AlignMamba-1
83.8
83.8
75.6
76.8
83.2
83.1
76.4
76.3
AlignMamba-2
84.0
83.9
77.4
77.5
83.4
83.2
77.0
77.1
Table 4: Cross-dataset generalization performance on a bimodal (audio-text) setup. The table shows the in-domain and out-of-domain (zero-shot) performance for models trained on MOSI and MOSEI, respectively. Best results are in bold.
Figure 6: Sensitivity analysis of model accuracy with respect to the MMD loss weight ( λMMD ) and the OT loss weight ( λOT ) on the MOSI and MOSEI datasets.
Figure 7: Qualitative analysis of a sample from the CMU-MOSI dataset.
Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images. However, multimodal models often fail to consistently outperform strong text-only baselines, with performance varying significantly across fusion strategies. In this work, we identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning, and show through controlled experiments that alignment prior to fusion is often more important than fusion complexity. To address this issue, we propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions, projecting heterogeneous modalities into a shared linguistic space and enabling interpretable text-centric reasoning. To further improve robustness, we introduce a hybrid learning strategy that combines semantic token selection with a batch-level uniformity regularization objective, encouraging a more dispersed and stable global feature space while mitigating noise introduced by VLM-generated descriptions. Experiments on multiple multimodal sentiment and emotion benchmarks show that our method consistently outperforms strong unimodal and multimodal baselines, achieving state-of-the-art performance. Our analysis further highlights the critical role of representation alignment in multimodal affective learning.
Multimodal Sentiment Analysis (MSA) requires effective modeling of cross-modal interactions and contextual dependencies while remaining computationally efficient. Existing fusion approaches predominantly rely on Transformer-based cross-modal attention, which incurs quadratic complexity with respect to sequence length and limits scalability. Moreover, contextual information from preceding utterances is often incorporated through concatenation or independent fusion, without explicit temporal modeling that captures sentiment evolution across dialogue turns. To address these limitations, we propose CAGMamba, a context-aware gated cross-modal Mamba framework for dialogue-based sentiment analysis. Specifically, we organize the contextual and the current-utterance features into a temporally ordered binary sequence, which provides Mamba with explicit temporal structure for modeling sentiment evolution. To further enable controllable cross-modal integration, we propose a Gated Cross-Modal Mamba Network (GCMN) that integrates cross-modal and unimodal paths via learnable gating to balance information fusion and modality preservation, and is trained with a three-branch multi-task objective over text, audio, and fused predictions. Experiments on three benchmark datasets demonstrate that CAGMamba achieves state-of-the-art or competitive results across multiple evaluation metrics. All codes are available at https://github.com/User2024-xj/CAGMamba.
Minghai Jiao, Jing Xiao, Peng Xiao +6
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.
Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)