cs.AINov 3, 2026

DGC-Net: Cross-Modal Dynamic Routing with Global-Guided Compensation in Multimodal Sentiment Analysis

Authors: Linan Zhu, Huaijing Wu, Xiao Han, Yuhao Wang

Organizations: College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China · AIDC, Alibaba Group, China · Department of Data Science, City University of Hong Kong, Hong Kong SAR, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

User-generated videos, including opinion vlogs and video reviews, provide a rich source of people's attitudes toward products, services, and social topics. Understanding the sentiment expressed by speakers is important for analyzing online opinions, as transcripts alone may not capture the attitude and intensity conveyed through vocal tone and facial expressions. Multimodal sentiment analysis (MSA) addresses this need by integrating textual, acoustic, and visual cues. However, these cues are often subtle, noisy, and inconsistent across modalities. Existing local affective cue modeling methods typically operate within individual modalities and overlook global contexts from other modalities. This may cause cross-modal affective cue inconsistency. Hierarchical global-local fusion methods integrate global semantics with fine-grained local affective cues. However, they commonly determine the relative contributions of the two representations implicitly, preventing the global-local contributions from adapting to the sample-dependent reliability of local cues and resulting in either insufficient use of informative local cues or excessive interference from unreliable ones. To address these issues, we propose DGC-Net, a Dynamic Gating and Global-Guided Compensation Network for MSA. DGC-Net mitigates cross-modal affective cue inconsistency through cross-modal contribution-aware dynamic expert routing. This mechanism uses auxiliary global contexts to guide local expert selection. Moreover, global-guided dynamic compensation fusion alleviates sample-dependent global-local contribution imbalance while preserving global semantic as the primary anchor and adaptively incorporating expert-refined local cues. Experiments on five multimodal benchmark datasets demonstrate the effectiveness and general applicability of DGC-Net, and ablation studies verify the contribution of each proposed component.

Figures & tables

Figure 1. (a) Independent intra-modal affective cue selection may produce inconsistent sentiment cues across modalities. (b) Static global–local fusion cannot adapt to the varying reliability of local affective cues, resulting in either insufficient local compensation or excessive local interference.
Figure 2. Overall architecture of DGC-Net, consisting of unimodal global–local representation, cross-modal contribution-aware dynamic expert routing, and global-guided dynamic compensation fusion.
 CMU-MOSICMU-MOSEI
MethodMAE ↓\downarrowCorr ↑\uparrowAcc-2 ↑\uparrowF1 ↑\uparrowAcc-7 ↑\uparrowMAE ↓\downarrowCorr ↑\uparrowAcc-2 ↑\uparrowF1 ↑\uparrowAcc-7 ↑\uparrow
TFN0.9010.698–/80.8–/80.734.90.5930.700–/82.5–/82.150.2
LMF0.9170.695–/82.5–/82.433.20.6230.677–/82.0–/82.148.0
MFM0.8770.706–/81.7–/81.635.40.5680.717–/84.4–/84.351.3
GFN0.7360.790–/84.3–/84.347.00.6110.774–/85.0–/85.051.8
MulT0.7670.799–/83.7–/83.741.50.6250.775–/84.7–/84.650.7
Table 1. Results on CMU-MOSI and CMU-MOSEI. For Acc-2 and F1, the values before/after “/” correspond to the negative/non-negative and negative/positive settings, respectively. Bold and underlined values denote the best and second-best results. †\dagger denotes our reproduced results with checkpoints selected by validation MAE.
 CMU-MOSICMU-MOSEI
VariantAcc-2 ↑\uparrowF1 ↑\uparrowAcc-2 ↑\uparrowF1 ↑\uparrow
Role of GDCF
w/o DGL-Fusion83.5183.2886.3586.26
Static Equal Ratio85.1985.1885.9485.94
Static Global-heavy Ratio83.0583.1285.9185.87
No Global Weight Constraint83.9783.9685.8885.85
Table 2. Ablation results on CMU-MOSI and CMU-MOSEI under the negative/positive setting. All values are percentages.
Figure 3. Sample-dependent text expert routing on CMU-MOSI with n=3n=3 experts and Top- k=2k=2 selection. Green and gray boxes indicate active and inactive experts, respectively. Numbers beside experts denote gating weights. Context ratios follow the local-text, global-audio, and global-vision order.
Global weightsyyy^\hat{y}Error
Case 130: but its really well done
(0.750, 0.750, 0.750)(0.750,\,0.750,\,0.750)2.202.090.11
Case 125: it made no sense
(0.739, 0.735, 0.737)(0.739,\,0.735,\,0.737)-2.00-2.040.04
Case 108: two out of five stars
(0.750, 0.750, 0.750)(0.750,\,0.750,\,0.750)-1.601.493.09
Table 3. Selected CMU-MOSI cases. yy and y^\hat{y} denote true and predicted scores. Error is ∣y−y^∣|y-\hat{y}| . Weights follow text/audio/vision order.
Figure 4. Joint t-SNE visualization on CMU-MOSI. Panels (a)–(c) share the same coordinates and display all, global, and local representations, respectively. GG and LL denote global and local representations; tt , aa , and vv denote text, audio, and vision. Three horizontally arranged scatter plots. Circular markers represent global features, and triangular markers represent local features. Colors distinguish the textual, acoustic, and visual modalities.
Figure 5. Joint t-SNE visualization on CMU-MOSEI. Panels (a)–(c) share coordinates within this dataset and display all, global, and local representations, respectively. Notation follows Figure 4 . Three horizontally arranged scatter plots. The first contains all six representation groups. The second displays global features, and the third displays local features at their original projected positions.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetTaskLabel spaceMain metric
CMU-MOSISentiment[−3,3][-3,3]MAE/Corr
CMU-MOSEISentiment[−3,3][-3,3]MAE/Corr
UR-FUNNYHumorBinaryAcc-2
MUStARDSarcasmBinaryAcc-2
CHERMAEmotionSeven classesAcc/Overall-F1
Appendix
Table 4. Tasks and principal evaluation metrics of the five multimodal benchmarks.
DatasetBatchLLTTddLearning rate
CMU-MOSI240460484×10−54\times 10^{-5}
CMU-MOSEI643801921×10−51\times 10^{-5}
UR-FUNNY2203801122×10−52\times 10^{-5}
MUStARD643701602×10−52\times 10^{-5}
CHERMA4007502562×10−52\times 10^{-5}
Appendix
Table 5. Dataset-specific settings. LL , TT , and dd denote encoder layers, maximum sequence length, and hidden dimension. All datasets use n=3n=3 local representations, three experts per modality, Top- k=2k=2 , and λmoe=10−2\lambda_{\mathrm{moe}}=10^{-2} .
MethodUR-FUNNY ↑\uparrowMUStARD ↑\uparrow
MISA (BERT)69.6266.18
MISA (ALBERT)69.8266.18
MAG-BERT (ALBERT)67.2069.12
MAG-BERT (XLNet)72.4376.47
AGM (BERT)65.97–
MIL (BERT)–76.36
Appendix
Table 6. Acc-2 results on UR-FUNNY and MUStARD. Baseline values are taken from the GLoMo appendix. “–” indicates that the corresponding result was not reported. Bold and underlined values indicate the best and second-best results, respectively.
MethodOverall-F1 ↑\uparrow
TFN68.37
LMF68.23
EFT68.72
LFT69.05
MulT69.24
PMR69.53
Appendix
Table 7. Overall-F1 results on CHERMA. Baseline values are taken from the GLoMo appendix. All values are percentages. Bold and underlined values indicate the best and second-best results, respectively.
VariantUR-FUNNY Acc-2 ↑\uparrowMUStARD Acc-2 ↑\uparrowCHERMA Acc ↑\uparrow
Role of GDCF
w/o DGL-Fusion70.7279.4170.56
Static Equal Ratio70.9382.3571.47
Static Global-heavy Ratio72.0382.3571.81
No Global Weight Constraint71.9380.8871.99
Role of Dynamic Modality-Specific MoEs
Appendix
Table 8. Extended ablation results on UR-FUNNY, MUStARD, and CHERMA. Acc-2 is reported for UR-FUNNY and MUStARD, and overall accuracy for CHERMA. All values are percentages. Results use seeds 87, 102, and 20, respectively. Bold values indicate the highest accuracy in each column, including ties.
Global weightsyyy^\hat{y}Error
Case 3918: but it a great color I love
(0.819, 0.737, 0.729)(0.819,\,0.737,\,0.729)2.002.150.15
Case 2771: it just so stupid.
(0.827, 0.724, 0.732)(0.827,\,0.724,\,0.732)-3.00-2.820.18
Case 2567: enjoy this one I did not it has the same
(0.752, 0.736, 0.729)(0.752,\,0.736,\,0.729)-1.330.051.38
Appendix
Table 9. Selected CMU-MOSEI cases. yy and y^\hat{y} denote true and predicted scores. Error is ∣y−y^∣|y-\hat{y}| . Weights follow text/audio/vision order.
More in this topic

Multimodal Sentiment Analysis

Multimodal sentiment analysis combines language, voice, and visual cues to infer sentiment, polarity, and emotional intensity.

See all 56 papers

Explore similar work

CardsList
Oct 20, 2026

Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities. Existing methods often treat modalities as symmetric, overlooking the fact that language provides a dominant semantic prior while nonverbal signals are complementary and noise-prone. To address this, we propose a Text-Anchored Modality Experts with Uncertainty-Guided Routing, termed TAME, that reformulates MSA as an uncertainty-guided refinement paradigm, where the language prior is calibrated and enhanced by selectively trusted nonverbal evidence. Specifically, TAME adopts an asymmetric Mixture-of-Modality-Experts framework, in which textual features act as a fixed and always-activated shared expert, while audio-textual and video-textual cross-modal branches serve as dynamically routed auxiliary experts. Furthermore, we introduce lightweight Prophet Heads to quantify the sample-level predictive uncertainty of each individual modality, and this uncertainty metric is further incorporated into the modality router to realize reliability-aware expert aggregation and fusion. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that TAME achieves state-of-the-art performance on the majority of metrics while reducing parameter count by up to 78.7% and FLOPs by up to 41.2% over the mixture-of-expert baseline.

Xiaohua Huang, Bingrui Lei, Xiuyan Shao +4

the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. +2

Multimodal Sentiment AnalysisModality Reliance in Multimodal ModelsUncertainty-Aware Multimodal FusionMixture-of-Experts Models
AskPDF
Aug 31, 2026

Multimodal Adaptive Expert Selection with Text Routing and Ordinal Prototype Optimization for Sentiment Analysis

Multimodal Sentiment Analysis (MSA) is a fundamental component of affective computing that aims to decipher complex emotional states by integrating verbal content with non-verbal cues including vocal intonation and facial micro-expressions. While recent disentanglement-based approaches have advanced the field, their potential is hindered by two methodological challenges. First, static computation graphs process all samples indiscriminately regardless of semantic complexity, which leads to suboptimal representation for diverse emotional expressions and contextual scenarios. Second, generic contrastive objectives often neglect the intrinsic ordinal hierarchy of sentiment intensities. To systematically address these limitations, we introduce Multimodal Adaptive Expert Selection with Text Routing and Ordinal prototype optimization (MAESTRO), a novel framework designed to dynamically orchestrate and refine multimodal representations. Drawing inspiration from an orchestra conductor, we design a Text-Guided Hybrid Mixture-of-Experts (MoE) mechanism. Unlike static fusion, this module utilizes linguistic context as a routing signal to dynamically activate specific audio-visual experts, thereby resolving cross-modal ambiguity through adaptive feature enhancement. Furthermore, to capture fine-grained sentiment gradations, we propose an Ordinal-aware Prototype Contrastive Learning (O-PCL). By incorporating distance-based penalties into the prototype learning objective, O-PCL enforces a structured latent space that preserves the natural order of emotion. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MAESTRO achieves state-of-the-art performance, and qualitative analysis further confirms the interpretability of our dynamic routing paradigm.

Xiaode Chen, Jiakang Yu, Hongtao Deng +3

Jianghan University Wuhan, China

Representation LearningMixture of ExpertsMultimodal Sentiment AnalysisMultimodal Contrastive LearningMultimodal Fusion
AskPDF
Apr 4, 2026

CAGMamba: Context-Aware Gated Cross-Modal Mamba Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) requires effective modeling of cross-modal interactions and contextual dependencies while remaining computationally efficient. Existing fusion approaches predominantly rely on Transformer-based cross-modal attention, which incurs quadratic complexity with respect to sequence length and limits scalability. Moreover, contextual information from preceding utterances is often incorporated through concatenation or independent fusion, without explicit temporal modeling that captures sentiment evolution across dialogue turns. To address these limitations, we propose CAGMamba, a context-aware gated cross-modal Mamba framework for dialogue-based sentiment analysis. Specifically, we organize the contextual and the current-utterance features into a temporally ordered binary sequence, which provides Mamba with explicit temporal structure for modeling sentiment evolution. To further enable controllable cross-modal integration, we propose a Gated Cross-Modal Mamba Network (GCMN) that integrates cross-modal and unimodal paths via learnable gating to balance information fusion and modality preservation, and is trained with a three-branch multi-task objective over text, audio, and fused predictions. Experiments on three benchmark datasets demonstrate that CAGMamba achieves state-of-the-art or competitive results across multiple evaluation metrics. All codes are available at https://github.com/User2024-xj/CAGMamba.

Minghai Jiao, Jing Xiao, Peng Xiao +6

School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China

Multimodal Sentiment AnalysisMultimodal FusionState Space Models
AskPDF