Recognizing emotions from speech using machine learning has become an active research area due to its importance in building human-centered applications. However, while many studies have been conducted in English, German, and other European and Asian languages, research in Arabic remains scarce because of the limited availability of annotated datasets. In this paper, we present an Arabic Speech Emotion Recognition (SER) system based on a hybrid CNN-Transformer architecture. The model leverages convolutional layers to extract discriminative spectral features from Mel-spectrogram inputs and Transformer encoders to capture long-range temporal dependencies in speech. Experiments were conducted on the EYASE (Egyptian Arabic speech emotion) corpus, and the proposed model achieved 97.8% accuracy and a macro F1-score of 0.98. These results demonstrate the effectiveness of combining convolutional feature extraction with attention-based modeling for Arabic SER and highlight the potential of Transformer-based approaches in low-resource languages.
Figures & tables
Study
Dataset
Method
Main Contribution / Focus
Al-Qatab et al. (2019)
KSUEmotions (acted)
CNN+BLSTM (attention)
Introduced hybrid deep model combining CNN, BLSTM, and attention for acted Arabic SER
Al-Qatab et al. (2019)
KSUEmotions (acted)
Deep CNN
Demonstrated effectiveness of deep CNNs for Arabic SER on acted data
Al-Onazi et al. (2022)
KSUEmotions (acted)
SVM, KNN
Explored classical machine learning classifiers for acted Arabic SER
Hussein et al. (2020)
TV Broadcast (natural)
SMO Classifier
Focused on recognizing emotions in natural, real-world Arabic speech
Rakan et al. (2021)
Egyptian Arabic (semi-natural)
Prosodic + Spectral features (SVM/KNN)
Investigated handcrafted acoustic features for semi-natural Egyptian Arabic SER
Proposed (This Work)
EYASE (semi-natural)
CNN–Transformer
Combines CNN for spectral feature extraction with Transformer for long-range temporal modeling in Arabic SER
TABLE I: Summary of Major Arabic SER Studies
Fig. 1: Overview of the proposed CNN–Transformer model for Arabic Speech Emotion Recognition.
Fig. 2: An example of a Mel-spectrogram extracted from an Arabic emotional utterance. The vertical axis corresponds to Mel frequency bins, and the horizontal axis represents time.
Dataset
Language/Dialect
Type
Emotions
Samples
EYASE
Egyptian Arabic
Semi-natural
Anger, Happiness, Sadness, Neutral
461
TABLE II: Summary of the EYASE Dataset
Hyperparameter
Value
Number of Mel filters
128
CNN kernel size
3×3
CNN layers
3 convolutional + pooling
Transformer encoder layers
4
Attention heads per layer
8
Embedding dimension ( dmodel )
256
TABLE III: Summary of Key Hyperparameters
Model
Accuracy (%)
Macro F1-score
SVM (with MFCCs)
68.7
0.65
MLP (with MFCCs)
71.4
0.69
CNN baseline
77.9
0.75
CNN–Transformer (proposed)
97.8
0.98
TABLE IV: Performance Comparison on the Test Set
Emotion
Precision
Recall
F1-score
Anger
0.98
0.97
0.97
Happiness
0.97
0.98
0.97
Sadness
0.98
0.98
0.98
Neutral
0.98
0.98
0.98
Macro Avg.
0.98
0.98
0.98
TABLE V: Class-wise Performance of the CNN–Transformer
Fig. 3: Training and validation loss/accuracy curves of the CNN–Transformer model.
Fig. 4: Confusion matrix of the CNN–Transformer on the test set.
aADASCA Laboratory – Advanced Data Science and Cognitive Applications, Université des Sciences et de la Technologie d’Oran Mohamed Boudiaf (USTO-MB), Oran, Algeria