Organizations: Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China
In the era of large-scale pre-trained models, effectively adapting general knowledge to specific affective computing tasks remains a challenge, particularly regarding computational efficiency and multimodal heterogeneity. While Transformer-based methods have excelled at modeling inter-modal dependencies, their quadratic computational complexity limits their use with long-sequence data. Mamba-based models have emerged as a computationally efficient alternative; however, their inherent sequential scanning mechanism struggles to capture the global, non-sequential relationships that are crucial for effective cross-modal alignment. To address these limitations, we propose AlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis. Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy, promoting geometric and statistical consistency between modalities without incurring any inference-time overhead. More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts to explicitly handle data heterogeneity during the fusion process. Extensive experiments on four challenging benchmarks, including dynamic time-series (on the CMU-MOSI and CMU-MOSEI datasets) and static image-related tasks (on the NYU-Depth V2 and MVSA-Single datasets), demonstrate that AlignMamba-2 establishes a new state-of-the-art in both effectiveness and efficiency across diverse pattern recognition tasks, ranging from dynamic time-series analysis to static image-text classification.
Figures & tables
Figure 1: The cross-attention mechanism in Transformer (left) and the scanning mechanism in Mamba (right).
Figure 2: The architecture of AlignMamba-2. The framework consists of three main stages: unimodal encoding, training-time dual alignment regularization (using OT and MMD), and a novel modality-aware Mamba fusion module based on a Mixture-of-Experts design.
Figure 4: GPU memory consumption (in MB) as a function of increasing sequence length. AlignMamba-2 demonstrates superior scalability compared to both Transformer-based and explicit-alignment-based Mamba models.
Figure 5: Inference latency (in milliseconds) as a function of increasing sequence length. AlignMamba-2 maintains a linear increase in inference time, showcasing its efficiency for long-sequence processing.
MOSI
MOSEI
NYUDv2
MVSA
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AlignMamba-2
87.0
87.0
86.5
86.5
73.1
71.5
82.7
80.2
w/o dual alignment loss
84.5
84.4
84.1
84.0
71.0
69.5
79.2
78.1
w/o Mixture-of-Experts
85.8
85.7
85.3
85.2
72.0
70.4
80.5
78.2
w/o both
82.3
82.1
81.8
81.4
69.8
68.7
77.8
77.1
w/ Learnable Routing
86.6
86.4
86.1
86.2
72.8
71.2
82.2
79.5
Table 3: Ablation study of AlignMamba-2 on all four datasets. We report the performance (Accuracy and F1-Score) of the full model and its variants. "w/o" stands for "without", and "w/" stands for "with". Best results are in bold.
Training dataset
MOSI
MOSEI
Test dataset
MOSI
MOSEI
MOSEI
MOSI
Model
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Cobra
80.2
80.0
70.5
70.3
79.5
79.2
70.1
70.0
GeminiFusion
83.2
83.1
74.6
74.6
82.6
82.5
75.0
75.1
AlignMamba-1
83.8
83.8
75.6
76.8
83.2
83.1
76.4
76.3
AlignMamba-2
84.0
83.9
77.4
77.5
83.4
83.2
77.0
77.1
Table 4: Cross-dataset generalization performance on a bimodal (audio-text) setup. The table shows the in-domain and out-of-domain (zero-shot) performance for models trained on MOSI and MOSEI, respectively. Best results are in bold.
Figure 6: Sensitivity analysis of model accuracy with respect to the MMD loss weight ( λMMD ) and the OT loss weight ( λOT ) on the MOSI and MOSEI datasets.
Figure 7: Qualitative analysis of a sample from the CMU-MOSI dataset.
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)