cs.MMMar 7, 2026
SaveAMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Organizations: College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
Abstract
Multimodal dialogue emotion recognition captures emotional cues by fusing text, visual, and audio modalities. However, existing approaches still suffer from notable limitations in modeling emotional dependencies and learning multimodal representations. On the one hand, they are unable to effectively filter out redundant or noisy signals within multimodal features, which hinders the accurate capture of the dynamic evolution of emotional states across and within speakers. On the other hand, during multimodal feature learning, dominant modalities tend to overwhelm the fusion process, thereby suppressing the complementary contributions of non-dominant modalities such as speech and vision, ultimately constraining the overall recognition performance. To address these challenges, we propose an Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network (AMB-DSGDN). Concretely, we first construct modality-specific subgraphs for text, speech, and vision, where each modality contains intra-speaker and inter-speaker graphs to capture both self-continuity and cross-speaker emotional dependencies. On top of these subgraphs, we introduce a differential graph attention mechanism, which computes the discrepancy between two sets of attention maps. By explicitly contrasting these attention distributions, the mechanism cancels out shared noise patterns while retaining modality-specific and context-relevant signals, thereby yielding purer and more discriminative emotional representations. In addition, we design an adaptive modality balancing mechanism, which estimates a dropout probability for each modality according to its relative contribution in emotion modeling.
Figures & tables
Figure 1: An authentic and representative segment illustrating the dynamic evolution of dialogue from the IEMOCAP dataset (Ses01F_impro01).
Figure 2: This architecture includes four core modules: first, the utterance-level encoder extracts unimodal features through OpenSmile (audio), RoBERTa (text), DenseNet (video), and after Transformer combining speaker embedding (Spk Emb) and position embedding (Pos Emb) encoding, obtains the text feature , video feature , audio feature for the -th utterance; second, the differential graph attention module constructs subgraphs including ”intra-speaker subgraph (Intraspeaker)” and ”inter-speaker subgraph (Interspeaker)” for each modality, computes differences between two groups of attention distributions through differential graph attention, eliminates cross-modal shared noise and retains modality-specific emotional signals; then, the adaptive modality balancing module computes dropout probabilities ( ) for each modality based on batch-level performance, performs dynamic dropout on dominant modalities, while scaling retained features through gradient compensation to maintain information balance; finally, through multimodal fusion and classification module, fuses the balanced features via linear layers and inputs into the classifier to obtain emotion recognition results.
| Dataset | Convs | Utterances | Speakers | Classes | Neutral | Happy Joy | Sadness | Angry | Excited Surprise | Frustrated | Disgust | Fear |
| IEMOCAP | 120 / 31 | 5810 / 1623 | 2 | 6 | 1708 / 490 | 1636 / 456 | 1084 / 266 | 1103 / 290 | 1041 / 265 | 1848 / 324 | - | - |
| MELD | 1153 / 280 | 11098 / 2610 | 9 | 7 | 5180 / 1256 | 1940 / 368 | 794 / 208 | 1243 / 364 | 1205 / 431 | - | 293 / 68 | 268 / 50 |
Table 1: Statistical information of the IEMOCAP and MELD datasets, including the number of conversations, utterances, speakers, total emotion classes, and the number of utterances for each emotion class. The values are reported as Train+Validation / Test splits.
| Models | IEMOCAP | |||||||||||||
| happy | sad | neutral | angry | excited | frustrated | wa-ACC | wa-F1 | |||||||
| ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | |||
| DialogueRNN | 25.00 | 34.95 | 82.86 | 84.58 | 54.43 | 57.66 | 61.76 | 64.42 | 90.97 | 76.30 | 62.20 | 59.55 | 65.43 | 64.29 |
| DialogueGCN | 64.29 | 29.03 | 80.86 | 64.37 | 43.14 | 50.96 | 68.49 | 63.29 | 71.85 | 68.19 | 57.68 | 62.41 | 62.07 | 58.19 |
| MMGCN | 32.64 | 39.66 | 72.65 | 76.89 | 65.10 | 62.81 | 73.53 | 71.43 | 77.93 | 75.40 | 65.09 | 63.43 | 66.62 | 66.25 |
| MM-DFN | 44.44 | 44.44 | 77.55 | 80.00 | 71.35 | 66.99 | 75.88 | 70.88 | 74.25 | 76.42 | 58.27 | 61.67 | 67.84 | 67.85 |
Table 2: The table evaluates the performance of all models on the IEMOCAP (six emotion categories) datasets using F1 scores, while presenting their overall performance across three datasets with weighted accuracy (wa-ACC) and weighted F1 score (wa-F1) as metrics, where the best results are bolded and the second-best are underlined.
| Models | MELD | |||||||||||||||
| neutral | surprise | fear | sadness | joy | disgust | anger | wa-ACC | wa-F1 | ||||||||
| ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1 | |||
| DialogueRNN | 82.17 | 76.56 | 46.62 | 47.64 | 0.00 | 0.00 | 21.15 | 24.65 | 49.50 | 51.49 | 0.00 | 0.00 | 48.41 | 46.01 | 60.27 | 57.95 |
| MMGCN | 84.32 | 76.96 | 47.33 | 49.63 | 2.00 | 3.64 | 14.90 | 20.39 | 56.97 | 53.76 | 1.47 | 2.82 | 42.61 | 45.23 | 61.34 | 58.41 |
| MM-DFN | 79.06 | 75.80 | 53.02 | 50.42 | 0.00 | 0.00 | 17.79 | 23.72 | 59.20 | 55.48 | 0.00 | 0.00 | 50.43 | 48.27 | 60.96 | 58.72 |
| TS-GCL | 78.10 | 80.60 | 56.70 | 56.40 | 6.80 | 5.20 | 42.30 | 43.70 | 68.30 | 66.30 | 2.30 | 2.60 | 43.80 | 48.50 | 64.40 | 64.10 |
Table 3: The table evaluates the performance of all models on the MELD (seven emotion categories) datasets using F1 scores, while presenting their overall performance across three datasets with wa-ACC and wa-F1 as metrics, where the best results are bolded and the second-best are underlined.
| Modality Setting | IEMOCAP | MELD | ||
| wa-ACC | wa-F1 | wa-ACC | wa-F1 | |
| A | 62.60 | 60.70 | 34.70 | 37.70 |
| T | 69.02 | 68.63 | 65.61 | 65.78 |
| V | 39.80 | 35.60 | 23.16 | 31.27 |
| A+T | 73.43 | 72.87 | 65.49 | 65.65 |
| V+T | 69.71 | 69.16 | 65.77 | 65.86 |
Table 4: Comparison of performance of different modal combinations on IEMOCAP and MELD datasets.
Figure 3: Experimental results of different window sizes on IEMOCAP and MELD datasets. The purple line represents wa-ACC, the light yellow bar represents wa-F1; the left subplot corresponds to IEMOCAP dataset, the right subplot to MELD dataset. Note: Window size determines the semantic association capture range of the graph convolutional network and needs to be adjusted based on dataset characteristics.
| Variants | IEMOCAP | MELD | ||
| wa-F1 | wa-ACC | wa-F1 | wa-ACC | |
| w/o Rel Subgraph + DiffRGCN | 68.00 | 68.82 | 65.07 | 65.10 |
| w/o DiffRGCN | 73.44 | 73.56 | 65.62 | 64.80 |
| Full Model | 75.64 | 76.09 | 66.07 | 66.18 |
Table 5: Ablation Study of Semantic Graph Differential Network on IEMOCAP and MELD Datasets.
Figure 4: Sensitivity analysis results of attention head numbers on IEMOCAP and MELD datasets. The blue line represents wa-F1 on IEMOCAP dataset, the green line represents wa-F1 on MELD dataset, horizontal axis “nhead Values” indicates number of attention heads, vertical axis “wa-F1 Score” indicates corresponding weighted average F1 score.
| Variants | IEMOCAP | MELD | ||
| wa-F1 | wa-ACC | wa-F1 | wa-ACC | |
| w/o MD + Rel Subgraph + DiffRGCN | 67.83 | 68.78 | 64.87 | 64.80 |
| w/o MD + DiffRGCN | 72.99 | 73.56 | 65.56 | 65.71 |
| w/o MD | 74.77 | 75.29 | 66.00 | 66.05 |
| Full Model | 75.64 | 76.09 | 66.07 | 66.18 |
Table 6: Ablation Study of Modality Balancing on IEMOCAP and MELD Datasets.
Figure 5: Performance dynamics in the warm-up phase. The horizontal axis represents the warm-up period (i.e., the number of training rounds before the modality dropout mechanism is gradually activated), the vertical axis is model performance metrics; different color curves correspond to weighted average F1 scores and accuracies on IEMOCAP and MELD datasets respectively, showing the impact of different warm-up periods on model performance.
| IEMOCAP | |||
| 0.1 | 74.88 | 75.62 | 75.2 |
| 0.3 | 75.37 | 75.64 | 75.06 |
| 0.5 | 75.07 | 74.96 | 75.12 |
| MELD | |||
Table 7: Sensitivity analysis results of modal discard probability parameter.
Figure 6: Performance comparison of the proposed method with other baseline methods on the long sequence dialogue subset of the IEMOCAP dataset, intuitively showing the performance advantages of the proposed method in experiments.
| Module | Dataset | Inference Time (ms) | Throughput (samples/s) | Batch Time (ms) | FLOPs/s (T) | ||||
| Baseline | (Change) | Baseline | (Change) | Baseline | (Change) | Baseline | (Change) | ||
| DiffRGCN | IEMOCAP | 0.0769 | +0.0090 (+11.70%) | 13169.51 | 1693.29 ( 12.86%) | 62.14 | +10.7027 (+17.22%) | 0.0805 | 0.0157 ( 19.50%) |
| MELD | 0.4271 | +0.1026 (+24.03%) | 2384.02 | 404.02 ( 16.95%) | 60.33 | +11.4914 (+19.05%) | 0.0500 | 0.00236 ( 4.72%) | |
| Adaptive Modality Balancing | IEMOCAP | 0.0769 | +0.0165 (+21.45%) | 13169.51 | 980.73 ( 7.45%) | 62.14 | +6.52 (+10.50%) | 0.0805 | 0.0082 ( 10.19%) |
| MELD | 0.4271 | +0.0190 (+4.45%) | 2384.02 | 137.90 ( 5.79%) | 60.33 | +2.1635 (+3.59%) | 0.0500 | 0.00384 ( 7.68%) | |
Table 8: Computational overhead analysis of DiffRGCN compared to the standard GAT, and of the Adaptive Modality Balancing module.
Figure 7: Full-modality noise robustness test results of the model on the IEMOCAP and MELD datasets. Each cell in the figure shows the model’s wa-ACC and wa-F1 under different noise intensities. The noise intensity ranges from 0 to 0.7, representing the standard deviation multiples of Gaussian noise added to the textual, visual, and audio features during testing. The color encoding indicates performance levels, with darker colors representing higher performance; the color bar on the right provides a reference for the corresponding percentage values. This heatmap intuitively illustrates the model’s stability and performance variations under different noise conditions.
| Method | Params (M) | IEMOCAP Runtime (s) | MELD Runtime (s) |
| DialogueGCN | 12.92 | 58.1 | 127.5 |
| RGAT | 15.28 | 68.5 | 146.3 |
| LR-GCN | 15.77 | 87.7 | 142.3 |
| MMGCN | 0.46 | 93.7 | 75.3 |
| DER-GCN | 78.59 | 125.5 | 189.7 |
| AMB-DSGDN (ours) | 13.13 | 179.95 | 146.4 |
Table 9: Model Parameters and Runtime Comparison
Figure 8: This figure presents the emotion recognition confusion matrices for the IEMOCAP (left) and MELD (right) datasets under the extreme modality imbalance experiment: the left matrix corresponds to the 6 emotion categories of IEMOCAP, while the right one corresponds to the 7 emotion categories of MELD. The ”True” axis represents the ground-truth emotion labels, and the ”Predicted” axis represents the model’s predicted labels; the values inside the cells indicate the number of samples for the corresponding category. The dark-colored blocks along the matrix diagonal reflect the high recognition accuracy for most emotions, while the light-colored blocks off the diagonal reflect the misclassification between different emotions.