Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
Authors: Xu Lin, Ke Wang, Hui Kang, Xinying Wang
Organizations: College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China
Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.
Figures & tables
Figure 1: Architecture of the audio feature extraction model
Figure 2: Architecture of the video feature extraction model
Figure 3: Architecture of the multimodal audio-visual emotion recognition model
Emotion Label
Training Set
Development Set
Test Set
anger
1109
153
345
disgust
271
22
68
fear
268
40
50
joy
1743
163
402
neutral
4710
470
1256
sadness
683
111
208
Table 1: Distribution Table of the MELD Dataset
Figure 4: Confusion Matrix of Audio-Based Emotion Recognition
Methods
Training Set
Test Set
RNN [ 8 ]
68.4%
68.3%
2D CNN [ 9 ]
69.2%
69.7%
1D CNN-LSTM [ 10 ]
73.9%
76.5%
Multi-audioFE
78.6%
80.3%
Table 2: Comparison of Different Audio Feature Extraction Algorithms
Methods
Accuracy
Precision
Recall
F1-Score
AlexNet [ 11 ]
68.1%
69.6%
66.7%
67.9%
GoogleNet [ 12 ]
67.3%
66.9%
66.9%
66.7%
ResNet50-BiLSTM
71.7%
72.3%
71.4%
71.8%
Table 3: Comparison of Different Video Feature Extraction Algorithms
Figure 5: ROC Curves of Multimodal Emotion Recognition
Figure 6: Confusion Matrix of Multimodal Emotion Recognition
Emotion Label
Accuracy
Precision
Recall
F1-Score
anger
98.9%
96.5%
98.9%
97.7%
disgust
98.0%
89.2%
98.0%
93.4%
fear
95.0%
85.4%
95.0%
90.0%
joy
94.7%
86.0%
94.7%
90.2%
neutral
90.8%
97.6%
90.8%
94.1%
surprise
97.2%
96.8%
97.7%
97.2%
Table 4: Results Table of Multimodal Emotion Recognition
Model
Fusion Strategy
Accuracy
F1-Score
EF-LSTM
Feature-level Fusion
79.8%
79.1%
BERT
Feature-level Fusion
82.7%
83.6%
CNN-LSTM
Decision-level Fusion
82.9%
84.1%
Ours
Feature-level Fusion
90.3%
88.5%
Table 5: Comparative Results of Multimodal Emotion Recognition Models
Model
Fusion Strategy
Accuracy
F1-Score
Non-fused Feature Layer
Concatenated Vector
82.9%
84.1%
Fused Feature Layer
Attention-based Fusion
93.7%
93.5%
Table 6: Ablation Study Results for Multimodal Emotion Recognition
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA