Organizations: the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. · the Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education, Nanjing 210096, China, and also with the School of Biological Science and Medical Engineering, Southeast University, Nanjing 210096, China. · the Center for Machine Vision and Signal Analysis, University of Oulu, 90570 Oulu, Finland.
Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities. Existing methods often treat modalities as symmetric, overlooking the fact that language provides a dominant semantic prior while nonverbal signals are complementary and noise-prone. To address this, we propose a Text-Anchored Modality Experts with Uncertainty-Guided Routing, termed TAME, that reformulates MSA as an uncertainty-guided refinement paradigm, where the language prior is calibrated and enhanced by selectively trusted nonverbal evidence. Specifically, TAME adopts an asymmetric Mixture-of-Modality-Experts framework, in which textual features act as a fixed and always-activated shared expert, while audio-textual and video-textual cross-modal branches serve as dynamically routed auxiliary experts. Furthermore, we introduce lightweight Prophet Heads to quantify the sample-level predictive uncertainty of each individual modality, and this uncertainty metric is further incorporated into the modality router to realize reliability-aware expert aggregation and fusion. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that TAME achieves state-of-the-art performance on the majority of metrics while reducing parameter count by up to 78.7% and FLOPs by up to 41.2% over the mixture-of-expert baseline.
Figures & tables
Fig. 1: Overview of the TAME architecture. It consists of a Text-Anchored Cross-Modal Attention (TACMA), a Pyramid Attention Pooling (PAP), Uncertainty Estimation via Prophet Heads (UEPH), and an Uncertainty-Guided Expert Router (UGER).
CMU-MOSI
CMU-MOSEI
Aligned
Unaligned
Aligned
Unaligned
Method
Acc2 (%)
F1 (%)
Acc7 (%)
MAE
Acc2 (%)
F1 (%)
Acc7 (%)
MAE
Acc2 (%)
F1 (%)
Acc7 (%)
MAE
Acc2 (%)
F1 (%)
Acc7 (%)
MAE
MISA ∗ [ 2 ]
84.2
84.2
41.8
0.754
83.8
83.9
43.6
0.742
85.3
85.1
52.3
0.543
84.8
84.8
51.0
0.557
Self-MM ∗ [ 4 ]
84.9
84.9
45.3
0.738
83.4
83.6
45.7
0.724
84.5
84.3
53.2
0.540
85.3
84.8
52.9
0.535
TFN [ 7 ]
78.8
78.9
31.9
0.953
76.5
76.6
35.3
0.995
80.4
80.7
50.9
0.574
84.2
84.0
50.2
0.573
LMF [ 8 ]
78.7
78.7
36.9
0.931
79.1
79.1
31.1
0.963
84.7
84.5
52.3
0.564
83.8
83.9
51.9
0.565
TABLE I: Comparison with state-of-the-art methods on CMU-MOSI and CMU-MOSEI. Best results are in bold ; second best are underlined . ∗ indicates vanilla BERT-base language features. § indicates a domain-adapted BERT variant continued-pretrained on MSA data (PAMoE’s MSA-BERT). ‡ indicates RoBERTa-based language features. † indicates DeBERTa-v3-base language features. The symbol ‘–’ indicates ‘Not reported’.
Method
Acc-2 (%)
F1 (%)
Acc-3 (%)
Acc-5 (%)
MAE
Corr
LF-DNN [N] [ 39 ]
78.87
79.87
66.91
41.62
0.420
0.612
MFN [N] [ 48 ]
77.90
77.88
65.73
39.47
0.435
0.582
LMF [N] [ 8 ]
77.77
77.88
64.68
40.53
0.441
0.576
TFN [N] [ 7 ]
78.38
78.62
65.12
39.30
0.432
0.591
Self-MM [N] [ 4 ]
80.04
80.44
65.47
41.53
0.425
0.595
LF-DNN [U] [ 39 ]
82.28
82.52
69.06
38.03
0.406
0.674
TABLE II: Comparison with state-of-the-art methods on CH-SIMS. Best results are in bold ; second best are underlined . [N] / [U] indicate that the model is trained without or with additional unimodal sentiment labels, respectively. (B) / (E) indicate that the textual embedding is extracted by BERT-base-Chinese or ERNIE-3.0-base, respectively. The asterisk ( ∗ ) on baseline names follows the original SDRS convention denoting an improved fine-tuned variant.
Dataset
Method
Params (M)
FLOPs (G)
CMU-MOSI (Aligned)
EMOE
317.30
1.4
TAME
195.80
1.069
CMU-MOSEI (Aligned)
EMOE
361.15
1.531
TAME
198.59
1.286
CH-SIMS
EMOE
537.78
1.489
TAME
114.80
0.876
TABLE III: Parameter and computational efficiency of TAME against EMOE on CMU-MOSI (Aligned), CMU-MOSEI (Aligned), and CH-SIMS.
Variant
Acc-2 (%) ↑
F1 (%) ↑
Acc-7 (%) ↑
MAE ↓
T1: TAME
88.22
88.15
56.49
0.4966
T2: +A ↔ V branch
86.90
86.95
54.75
0.5144
T3: Text as Query
87.20
87.29
56.13
0.5056
T4: T2 + T3
86.46
86.50
54.17
0.5190
TABLE IV: Ablation study on Text-Anchored Cross-Modal Attention of TAME.
Variant
Acc-2 (%) ↑
F1 (%) ↑
Acc-7 (%) ↑
MAE ↓
TAME
88.22
88.15
56.49
0.4966
w/o UEPH
86.76
86.85
54.60
0.5109
w/o URGE
87.12
87.09
55.76
0.5098
w/o PAP
87.64
87.71
53.94
0.5239
TABLE V: Ablation study of TAME. Each row removes one component from the TAME model. Removing UEPH also disables URGE due to architectural dependency.
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China · Nanyang Technological University, Singapore, 639798 · School of Electronics and Information Technology, Sun Yat-sen University, Guangzhou 510006, China +4
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, China · Zhejiang University Jinhua Research Institute, Jinhua, 321000, China · College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, China