Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations

Tao Meng; Weilun Tang; Yuntao Shou; Yilong Tan; Jun Zhou; Wei Ai; Keqin Li

cs.AIMar 22, 2026

Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations

Authors: Tao Meng, Weilun Tang, Yuntao Shou, Yilong Tan, Jun Zhou, Wei Ai, Keqin Li

Organizations: College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha, China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA

Abstract

Multimodal emotion recognition in conversations (MERC) aims to identify and understand the emotions expressed by speakers during utterance interaction from multiple modalities (e.g., text, audio, images, etc.). Existing studies have shown that GCN can improve the performance of MERC by modeling dependencies between speakers. However, existing methods usually use fixed parameters to process multimodal features for different emotion types, ignoring the dynamics of fusion between different modalities, which forces the model to balance performance between multiple emotion categories, thus limiting the model's performance on some specific emotions. To this end, we propose a dynamic fusion-aware graph convolutional neural network (DF-GCN) for robust recognition of multimodal emotion features in conversations. Specifically, DF-GCN integrates ordinary differential equations into graph convolutional networks (GCNs) to {capture} the dynamic nature of emotional dependencies within utterance interaction networks and leverages the prompts generated by the global information vector (GIV) of the utterance to guide the dynamic fusion of multimodal features. This allows our model to dynamically change parameters when processing each utterance feature, so that different network parameters can be equipped for different emotion categories in the inference stage, thereby achieving more flexible emotion classification and enhancing the generalization ability of the model. Comprehensive experiments conducted on two public multimodal conversational datasets {confirm} that the proposed DF-GCN model delivers superior performance, benefiting significantly from the dynamic fusion mechanism introduced.

Figures & tables

Figure 1: Illustration of the difference between traditional static fusion and dynamic fusion processes. (a) Previous studies usually use fixed network parameters to fuse multimodal features. (b) Our proposed method generates modal parameters through GIV, thereby achieving dynamic multimodal feature fusion.

Figure 2: The overall architecture of the proposed DF-GCN model. The framework consists of three main components: (i) Graph Construction, where multimodal utterances from text (RoBERTa), audio (OpenSMILE), and vision (DenseNet) are encoded and integrated through an attention layer and Bi-GRU to form discourse-level interaction graphs; (ii) Global Information Block, which employs stacked Transformer layers and global average pooling to generate GIV that guide context-aware fusion; and (iii) Prompts Generation Network, where DGCODE blocks dynamically generate weight prompts for adaptive multimodal feature integration. SGCODEcaptures structural dependencies, while DGCODE models temporal and contextual dynamics. Skip connections and batch normalization enhance training stability, and the final fused representations are used for emotion classification.

Method	Happy		Sad		Neutral		Angry		Excited		Frustrated		Average(w)
Method	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
A-DMN [ 62 ]	43.1	50.6	69.4	76.8	63.0	62.9	63.5	56.5	88.3	77.9	53.3	55.7	64.6	64.3
RGAT [ 66 ]	42.3	51.6	70.1	77.3	61.6	65.4	64.2	63.0	69.3	68.0	57.8	61.2	66.5	65.2
QMNN [ 61 ]	41.4	39.7	72.9	68.3	54.1	55.3	65.4	62.6	66.0	66.7	55.6	62.2	60.8	59.9
CoMPM [ 63 ]	55.4	60.7	78.3	82.2	64.5	63.0	66.2	59.9	77.4	78.2	57.3	59.5	64.2	67.3
EmoBERTa [ 56 ]	56.9	56.4	79.1	83.0	64.0	61.5	70.6	69.6	86.0	78.0	63.8	68.5	67.3	67.3

Table 1: The performance of different methods on IEMOCAP Dataset. The values reported in each cell represent the Acc/F1. The best result in each column is in bold.

Method	Neutral		Surprise		Fear		Sadness		Joy		Disgust		Anger		Average(w)
Method	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
A-DMN [ 62 ]	78.8	78.9	55.4	55.3	8.7	6.8	24.7	23.9	24.6	22.7	3.5	3.2	41.3	40.9	55.5	55.4
RGAT [ 66 ]	77.6	78.1	43.2	41.5	5.6	2.4	29.8	30.7	59.1	58.6	3.4	2.2	45.2	44.6	60.2	59.5
QMNN [ 61 ]	71.2	77.0	45.8	49.8	-	-	24.3	16.5	53.5	52.1	-	-	42.9	43.2	60.8	58.0
CoMPM [ 63 ]	79.1	82.0	51.5	49.2	5.2	2.9	33.6	32.3	62.3	61.5	8.4	2.8	42.7	45.8	64.3	63.0
EmoBERTa [ 56 ]	78.9	82.5	50.2	50.2	1.8	1.9	33.3	31.2	72.1	61.7	9.1	2.5	43.3	46.4	64.1	63.3

Table 2: The performance of different methods on MELD Dataset. The values reported in each cell represent the Acc/F1. The best result in each column is in bold.

Figure 3: Verifying the effectiveness of multimodal features.

Panel of Figure 4 — Figure 4: Sensitivity analysis of DF-GCN with respect to key graph construction hyperparameters. (a) Effect of the similarity threshold $\theta$ on WF1. (b) Effect of the context window size $w$ on WF1.

Model	# Params.	IEMOCAP	MELD
Model	# Params.	Inference Time	Inference Time
MMGCN	5.63M	53.7s	75.3s
RGAT	15.28M	68.5s	146.3s
DER-GCN	78.59M	125.5s	189.7s
M3Net	7.88M	57.0s	87.6s
DF-GCN (Ours)	9.19M	60.2s	94.1s

Table 3: Comparison of model parameters and inference time on IEMOCAP and MELD datasets.

Panel of Figure 5 — Figure 5: Confusion matrices of emotion classification results on the IEMOCAP and MELD test sets. Subfigures (a) and (b) present the performance of the proposed DF-GCN model on IEMOCAP and MELD, respectively, while subfigures (c) and (d) show the results of the baseline DER-GCN under the same settings. Darker diagonal blocks indicate higher accuracy in correctly identifying specific emotion categories, whereas off-diagonal values represent misclassifications.

Methods	IEMOCAP		MELD
Methods	Acc	F1	Acc	F1
QMNN	60.8 $\pm$ 0.87	59.9 $\pm$ 1.23	60.8 $\pm$ 0.45	58.0 $\pm$ 1.58
CoMPM	64.2 $\pm$ 0.19	67.3 $\pm$ 1.02	64.3 $\pm$ 0.66	63.0 $\pm$ 1.41
EmoBERTa	67.3 $\pm$ 0.33	67.3 $\pm$ 1.15	64.1 $\pm$ 0.92	63.3 $\pm$ 0.54
MMGCN	66.8 $\pm$ 1.37	66.8 $\pm$ 0.28	58.3 $\pm$ 1.49	58.4 $\pm$ 0.71
M3Net	72.5 $\pm$ 1.08	71.1 $\pm$ 1.55	63.9 $\pm$ 0.83	65.8 $\pm$ 1.29

Table 4: Average performance after 10 runs on the IEMOCAP and MELD datasets. The values reported in each cell represent the average Acc/F1

\pm

std. The best result in each column is in bold.

GIV	PGN	DGCODE	IEMOCAP		MELD
GIV	PGN	DGCODE	Acc.	F1	Acc.	F1
+	+	+	73.4	72.2	67.4	67.6
-	+	+	71.3	71.6	67.2	65.8
+	-	+	70.6	70.8	66.9	64.8
+	+	-	69.4	68.7	65.4	64.4
-	-	+	70.6	70.8	66.8	65.0

Table 5: Ablation results of DF-GCN on the IEMOCAP and MELD benchmark dataset.

Panel of Figure 6 — Figure 6: Visualization of the learned embeddings.

Explore similar work

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Mar 7, 2026Yunsheng Wang, Yuntao Shou, Yilong Tan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition
Mar 22, 2026Ying Liu, Yuntao Shou, Wei Ai +2Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA
Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition
Apr 3, 2026Chengling Guo, Yuntao Shou, Tao Meng +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Changsha Hospital for Maternal and Child Health Care · Department of Computer Science, State University of New York

Mar 7, 2026J/K move · Enter open · S save

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition

Yunsheng Wang, Yuntao Shou, Yilong Tan, Wei Ai+2

College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York

Emotion Recognition in Conversations Multimodal Emotion Recognition Modality Imbalance Multimodal Graph Learning

Open paper AskPDF arXiv

Multimodal dialogue emotion recognition captures emotional cues by fusing text, visual, and audio modalities. However, existing approaches still suffer from notable limitations in modeling emotional dependencies and learning multimodal representations. On the one hand, they are unable to effectively filter out redundant or noisy signals within multimodal features, which hinders the accurate capture of the dynamic evolution of emotional states across and within speakers. On the other hand, during multimodal feature learning, dominant modalities tend to overwhelm the fusion process, thereby suppressing the complementary contributions of non-dominant modalities such as speech and vision, ultimately constraining the overall recognition performance. To address these challenges, we propose an Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network (AMB-DSGDN). Concretely, we first construct modality-specific subgraphs for text, speech, and vision, where each modality contains intra-speaker and inter-speaker graphs to capture both self-continuity and cross-speaker emotional dependencies. On top of these subgraphs, we introduce a differential graph attention mechanism, which computes the discrepancy between two sets of attention maps. By explicitly contrasting these attention distributions, the mechanism cancels out shared noise patterns while retaining modality-specific and context-relevant signals, thereby yielding purer and more discriminative emotional representations. In addition, we design an adaptive modality balancing mechanism, which estimates a dropout probability for each modality according to its relative contribution in emotion modeling.

Abstract

Explore similar work

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition

Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition

Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition