Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition

Jiaqi Qiao; Yifan Lyu; Xiujuan Xu

cs.AISep 2, 2026

Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition

Authors: Jiaqi Qiao, Yifan Lyu, Xiujuan Xu

Organizations: Dalian University of Technology

Abstract

Multimodal emotion recognition is a key research area in affective computing, with applications in sentiment analysis, intelligent customer service, and human-computer interaction. However, existing methods often rely on single-modal features or simple multimodal fusion, failing to capture the synergy between global and local contexts, which limits model performance and emotion understanding. To address this challenge, we propose Transformer-GAT, a hybrid framework that combines Transformer and the Graph Attention Network to enable cross-modal emotion understanding. The Transformer is used to capture global semantic information, while the Graph Attention Network is employed to model fine-grained relationships between modalities, thereby enhancing the representation of emotional features. Experiments on the IEMOCAP and MELD datasets show that our model achieves weighted F1 scores of 72.45% and 77.37%, outperforming state-of-the-art methods. These results demonstrate that Transformer-GAT effectively integrates multimodal features, balances global and local contexts, and provides deeper emotional insights, offering new directions for multimodal emotion computing.

Figures & tables

Figure 1: A sample dialogue snippet from the MELD dataset.

Figure 2: Transformer-based Graph Attention Network.

Dataset	Partition	No.Uttrs	No.Dials
IEMOCAP	train + val	5810	120
IEMOCAP	test	1623	31
MELD	train + val	11098	1152
MELD	test	2610	280

Table 1: Statistics of the two datasets

Statistics	Train	Dev	Test
Avg. Utt Length	15.15 / 8.03	17.00 / 7.99	20.10 / 8.28
Avg. Emo/Conv	4.86 / 3.30	5.10 / 3.35	4.97 / 3.24
Avg. Utt/Conv	64.68 / 9.61	70.05 / 9.72	70.00 / 9.32

Table 2: IEMOCAP and MELD datasets statistics.

Dataset	Text Dim	Acoustic Dim	Visual Dim
IEMOCAP	768	100	256
MELD	768	300	342

Table 3: IEMOCAP and MELD Feature Dimensions.

Model	Happy	Sad	Neutral	Angry	Excited	Frustrated	Accuracy	Wa-F1
LineConGAT Krishnan et al. (2023)	-	-	-	-	-	-	-	64.58
MPLMM Guo et al. (2024)	-	-	-	-	-	-	67.42	67.22
DualGATs Zhang et al. (2023a)	-	-	-	-	-	-	-	67.68
DAG-ERC Shen et al. (2021)	-	-	-	-	-	-	-	68.03
LR-GCN Ren et al. (2022)	55.50	79.10	63.80	69.00	74.00	68.90	68.50	68.30
EmoBERTa Kim and Vossen (2021)	-	-	-	-	-	-	-	68.57

Table 4: Experimental results on the IEMOCAP dataset

Model	Neutral	Surprise	Fear	Sadness	Joy	Disgust	Anger	Accuracy	Wa-F1
DAG-ERC Shen et al. (2021)	-	-	-	-	-	-	-	-	63.65
EmoCaps Li et al. (2022)	77.12	63.19	3.03	42.52	57.50	7.69	57.54	-	64.00
LR-GCN Ren et al. (2022)	80.80	57.10	0.00	36.90	65.80	11.00	54.70	-	65.60
EmoBERTa Kim and Vossen (2021)	-	-	-	-	-	-	-	-	65.61
DER-GCN Ai et al. (2024)	80.60	51.00	10.40	41.50	64.30	10.30	57.40	66.80	66.10
FacialMMT Zheng et al. (2023)	80.13	59.63	19.18	41.99	64.88	18.18	56.00	-	66.58

Table 5: Experimental results on the MELD dataset

Figure 3: Comparison of the quantity of neutral emotion with other emotions.

GAT layers	IEMOCAP	MELD
1	70.74( $\downarrow$ 1.71)	77.37
2	72.45	73.46( $\downarrow$ 3.91)
3	71.70( $\downarrow$ 0.75)	68.87( $\downarrow$ 8.50)
4	71.65( $\downarrow$ 0.80)	70.28( $\downarrow$ 7.09)

Table 6: Impact of Layer Number on Transformer-GAT Performance in ERC

Modality	IEMOCAP	MELD
v	14.33( $\downarrow$ 58.12)	31.27( $\downarrow$ 46.10)
a	26.37( $\downarrow$ 46.08)	37.48( $\downarrow$ 39.89)
t	71.57( $\downarrow$ 0.88)	75.69( $\downarrow$ 1.68)
va	32.04( $\downarrow$ 40.41)	40.86( $\downarrow$ 36.51)
vt	71.67( $\downarrow$ 0.78)	76.25( $\downarrow$ 1.12)
at	71.73( $\downarrow$ 0.72)	76.40( $\downarrow$ 0.97)

Table 7: The effectiveness of Transformer-GAT for ERC under different modality combinations

Different components	IEMOCAP	MELD
Transformer-GAT	72.45	77.37
w/o Long-term speaker…	71.47( $\downarrow$ 0.98)	76.96( $\downarrow$ 0.41)
w/o Short-term speaker…	70.19( $\downarrow$ 2.26)	75.77( $\downarrow$ 1.60)
w/o Local context…	71.58( $\downarrow$ 0.87)	76.32( $\downarrow$ 1.05)
w/o transformer	70.39( $\downarrow$ 2.06)	76.99( $\downarrow$ 0.38)

Table 8: The impact of different graphs on Transformer-GAT’s performance in ERC

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Emotion label distribution of IEMOCAP.

Figure 7: Emotion label distribution of MELD.

Figure 8: The t-SNE visualization of IEMOCAP.

Figure 9: The t-SNE visualization of MELD.

Explore similar work

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
Sep 4, 2026Xu Lin, Ke Wang, Hui Kang +1Multimodal Emotion Recognition Multimodal Fusion
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China
Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition
Apr 3, 2026Chengling Guo, Yuntao Shou, Tao Meng +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Changsha Hospital for Maternal and Child Health Care · Department of Computer Science, State University of New York
Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry
Sep 9, 2026Oriol Marín, Roger Marí, Gloria Haro +1Emotion Recognition in Conversations Multimodal Emotion Recognition
Eurecat, Centre Tecnològic de Catalunya, Barcelona, Spain · Universitat Pompeu Fabra, Barcelona, Spain · Serra Húnter Fellow Programme, Universitat Pompeu Fabra, Barcelona, Spain

Sep 4, 2026J/K move · Enter open · S save

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

Xu Lin, Ke Wang, Hui Kang, Xinying Wang

College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China

Multimodal Emotion Recognition Multimodal Fusion

Open paper AskPDF arXiv

Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.

Abstract

Explore similar work

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition

Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry