Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognition
Authors: Yuntao Shou, Jun Zhou, Tao Meng, Wei Ai, Keqin Li
Organizations: College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
Multimodal Emotion Recognition in Conversations (MERC) aims to predict speakers' emotional states in multi-turn dialogues through text, audio, and visual cues. In real-world settings, conversation scenarios differ significantly in speakers, topics, styles, and noise levels. Existing MERC methods generally neglect these cross-scenario variations, limiting their ability to transfer models trained on a source domain to unseen target domains. To address this issue, we propose a Dual-branch Graph Domain Adaptation framework (DGDA) for multimodal emotion recognition under cross-scenario conditions. We first construct an emotion interaction graph to characterize complex emotional dependencies among utterances. A dual-branch encoder, consisting of a hypergraph neural network (HGNN) and a path neural network (PathNN), is then designed to explicitly model multivariate relationships and implicitly capture global dependencies. To enable out-of-domain generalization, a domain adversarial discriminator is introduced to learn invariant representations across domains. Furthermore, a regularization loss is incorporated to suppress the negative influence of noisy labels. To the best of our knowledge, DGDA is the first MERC framework that jointly addresses domain shift and label noise. Theoretical analysis provides tighter generalization bounds, and extensive experiments on IEMOCAP and MELD demonstrate that DGDA consistently outperforms strong baselines and better adapts to cross-scenario conversations. Our code is available at https://github.com/Xudmm1239439/DGDA-Net.
Figures & tables
Figure 1: (a) Common MERC methods. A well-crafted encoder architecture is used to achieve multimodal emotion recognition without considering out-of-domain distribution differences. (b) Our Proposed Dual-branch Graph Domain Adaptation (DGDA) method. DGDA exploits a dual-branch encoder to explicitly and implicitly extract multimodal features, and constructs a domain adversarial alignment strategy and regularization loss to achieve out-of-domain distribution data generalization and resistance to noise label interference.
Figure 2: An overview of the proposed DGDA framework. The model operates on a labeled source domain and an unlabeled target domain. In both domains, audio, visual, and text features are first extracted and used to construct utterance-level interaction graphs. A dual-branch graph encoder encodes these graphs. For domain alignment, the source domain is adaptively perturbed by a learned noise δ , while a domain discriminator promotes feature invariance across domains. Meanwhile, category-level alignment is enforced by coupling the dual-branch outputs. The final emotion classifier is trained using source labels and pseudo-labeled target samples.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
Tradition
TextCNN
38.79
14.47
54.07
29.98
44.12
39.13
60.65
35.52
54.21
46.73
LSTM
11.17
6.86
57.27
33.37
39.98
51.27
60.89
46.76
46.74
50.68
DialogueRNN
24.02
12.95
63.69
35.23
47.08
37.75
56.47
14.91
59.92
39.46
MMGCN
24.87
0.00
46.98
28.73
35.77
50.95
0.00
54.90
64.13
43.94
Table 1: The performance of different methods is shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Underlines indicate suboptimal performance. Bold results indicate the best performance. We set the noise rate to 10%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
Tradition
TextCNN
38.56
13.11
51.71
29.19
42.51
36.46
54.51
32.58
52.26
43.33
LSTM
10.76
5.70
54.03
32.34
37.82
45.85
55.67
43.47
43.85
46.74
DialogueRNN
22.82
11.40
60.06
34.34
44.52
35.04
51.49
14.60
55.60
36.68
MMGCN
22.62
0.00
45.54
27.99
34.38
46.66
0.00
53.57
59.24
41.59
Table 2: The performance of different methods is shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. Underlines indicate suboptimal performance. We set the noise rate to 20%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
Tradition
TextCNN
34.59
9.87
47.79
26.69
38.83
35.46
52.43
30.87
47.59
40.87
LSTM
9.74
4.57
49.49
29.03
34.48
43.19
51.54
40.71
38.57
43.03
DialogueRNN
20.81
8.62
54.72
31.3
40.41
34.10
49.35
13.84
49.94
34.31
MMGCN
21.64
0.00
41.58
25.73
31.62
42.27
0.00
50.85
53.90
38.59
Table 3: The performance of different methods is shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. Underlines indicate suboptimal performance. We set the noise rate to 30%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
Tradition
TextCNN
33.92
9.04
44.59
24.53
36.51
34.33
48.97
26.65
43.49
37.29
LSTM
9.67
4.61
44.72
26.50
31.41
41.46
48.47
36.83
37.65
40.40
DialogueRNN
20.60
8.08
49.99
28.84
37.30
33.55
47.75
12.01
44.78
31.85
MMGCN
21.46
0.00
38.70
23.59
29.66
40.82
0.00
46.71
51.12
36.14
Table 4: The performance of different methods is shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. Underlines indicate suboptimal performance. We set the noise rate to 40%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
DGDA-HGNN
40.17
11.41
69.41
32.58
53.19
56.46
54.99
42.79
77.15
56.45
DGDA-PathNN
43.59
9.87
62.43
31.68
49.69
52.47
48.31
50.07
67.43
54.49
DGDA/ σHGNN
44.39
7.78
74.95
20.48
55.23
58.44
57.03
45.67
73.41
57.29
DGDA/ σPathNN
48.37
7.65
75.27
18.93
56.01
56.55
62.39
51.65
74.83
60.85
DGDA/AP
28.96
4.25
66.79
25.27
47.74
34.97
42.53
54.78
58.09
50.01
Table 5: The ablation studies are shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. We set the noise rate to 10%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
DGDA-HGNN
36.07
7.77
66.00
29.34
49.64
53.21
51.47
40.09
73.97
53.36
DGDA-PathNN
39.77
6.26
57.87
28.38
45.54
49.27
45.05
47.46
63.93
51.42
DGDA/ σHGNN
39.71
4.72
70.37
15.73
50.73
55.98
53.49
41.75
70.72
53.99
DGDA/ σPathNN
44.56
4.30
71.80
15.75
52.52
53.54
59.50
47.97
72.54
57.80
DGDA/AP
25.02
7.33
63.48
22.15
44.88
32.35
38.62
52.36
54.77
46.98
Table 6: The ablation studies are shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. We set the noise rate to 20%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
DGDA-HGNN
33.28
3.21
62.38
25.83
46.13
50.47
48.93
38.01
69.02
50.34
DGDA-PathNN
35.99
2.61
53.52
23.48
41.30
47.06
42.46
42.80
61.26
48.10
DGDA/ σHGNN
35.73
2.74
67.35
10.94
47.36
53.77
51.10
39.30
66.06
51.01
DGDA/ σPathNN
39.61
1.70
69.56
13.14
49.63
50.95
56.79
43.36
69.06
54.21
DGDA/AP
20.93
5.06
60.07
19.45
41.52
28.86
35.47
49.80
52.39
44.20
Table 7: The ablation studies are shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. We set the noise rate to 30%.
Methods
IEMOCAP → MELD
MELD → IEMOCAP
Joy
Sadness
Neutral
Anger
WF1
Joy
Sadness
Neutral
Anger
WF1
DGDA-HGNN
29.05
2.17
60.09
22.28
43.37
37.82
44.14
35.26
44.37
40.05
DGDA-PathNN
33.60
4.32
51.39
19.75
39.23
33.53
36.77
38.02
48.31
39.77
DGDA/ σHGNN
33.23
6.86
64.70
6.45
45.07
31.58
39.01
34.45
61.35
42.08
DGDA/ σPathNN
35.08
3.19
60.37
8.89
43.02
38.82
31.96
40.72
46.25
39.85
DGDA/AP
17.75
1.72
56.67
16.23
38.20
26.81
31.16
47.53
48.98
41.19
Table 8: The performance of different methods is shown under different noisy rates on the IEMOCAP and MELD datasets. The arrow → means from source to target domains. Bold results indicate the best performance. We set the noise rate to 40%.
Figure 3: Verify the effectiveness of multimodal features.
Figure 4: Hyperparameter sensitivity of threshold ζ and regularization weight λ .
Figure 5: Confusion matrices for multimodal emotion recognition datasets. The matrices provide insights into the model’s classification accuracy, highlighting the challenges and successes in distinguishing between different emotional categories. Top: Results with varying noise levels on the first dataset setting. Bottom: Results with varying noise levels on the second dataset setting.
Figure 6: Visualization of the learned embeddings. Top: Trained on the MELD dataset and tested on the IEMOCAP dataset (with varying noise levels). Bottom: Trained on the IEMOCAP dataset and tested on the MELD dataset (with varying noise levels).
College of Computer and Mathematics, Central South University of Forestry and Technology · Changsha Hospital for Maternal and Child Health Care · Department of Computer Science, State University of New York
College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha, China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA