Speech is a natural means of conveying emotions, making it an effective method for understanding and representing human feelings. Reliable speech emotion recognition (SER) is central to applications in human-computer interaction, healthcare, education, and customer service. However, most SER methods depend on heavy backbone models or hand-crafted features that fail to balance accuracy and efficiency, particularly for low-resource languages like Bangla. In this work, we present SpectroFusion-ViT, a lightweight SER framework built utilizing EfficientViT-b0, a compact Vision Transformer architecture equipped with self-attention to capture long-range temporal and spectral patterns. The model contains only 2.04M parameters and requires 0.1 GFLOPs, enabling deployment in resource-constrained settings without compromising accuracy. Our pipeline first performs preprocessing and augmentation on raw audio, then extracts Chroma and Mel-frequency cepstral coefficient (MFCC) features. These representations are fused into a complementary time-frequency descriptor that preserves both fine-grained spectral detail and broader harmonic structure. Using transfer learning, EfficientViT-b0 is fine-tuned for multi-class emotion classification. We evaluate the system on two benchmark Bangla emotional speech datasets, SUBESCO and BanglaSER, which vary in speaker diversity, recording conditions, and acoustic characteristics. The proposed approach achieves 92.56% accuracy on SUBESCO and 82.19% on BanglaSER, surpassing existing state-of-the-art methods. These findings demonstrate that lightweight transformer architectures can deliver robust SER performance while remaining computationally efficient for real-world deployment.
Figures & tables
Fig. 1 : Overview of the proposed SpectroFusion-ViT framework for speech emotion recognition. Raw audio is first normalized and augmented, after which Chroma and MFCC features are extracted. These complementary representations are fused into a joint time–frequency descriptor and fed into the EfficientViT-b0 backbone for transfer learning–based emotion classification.
Property
Value
Speakers
20 (10 male, 10 female)
Sentences per speaker
10
Emotional classes
7 (anger, disgust, fear,
happiness, neutral, sadness, surprise)
Repetitions per sentence
5
Total utterances
7,000
TABLE I : Properties of the SUBESCO dataset [ 26 ]
Property
Value
No. of Speakers
34 (17 male, 17 female)
Age Range
19–47 years
No. of Utterances
1,467
No. of Classes
5 (angry, happy, sad, surprise, neutral)
Samples per Class
306 (neutral: 243)
Average Duration
3–4 seconds
TABLE II : Properties of the BanglaSER dataset [ 29 ]
BanglaSER
SUBESCO
Architecture
Chroma
MFCC
Combined
Chroma
MFCC
Combined
Inception_v3
53.24
55.91
58.39
59.42
62.07
65.17
MobileNetv2
60.74
65.58
68.06
62.81
66.32
67.29
ResNet50
64.38
67.12
69.30
63.55
65.41
67.58
ResNet34
68.21
70.43
72.84
72.19
74.05
76.50
DenseNet121
65.42
68.37
71.68
77.51
80.26
83.39
TABLE III : Performance comparison of alternative backbone architectures evaluated within the proposed SpectroFusion-ViT framework on the BanglaSER and SUBESCO datasets.
BanglaSER
SUBESCO
Class
Precision
Recall
F1-Score
Precision
Recall
F1-Score
Angry
89
90
90
92
94
93
Happy
78
74
76
90
90
90
Neutral
84
88
86
94
93
93
Sad
78
79
79
90
90
90
Surprise
77
76
77
90
92
91
TABLE IV : Class-wise precision, recall, and F1-score (%) for BanglaSER and SUBESCO datasets
Accuracy (%)
Architecture
BanglaSER
SUBESCO
Chakraborty et al. [ 5 ]
79.00
75.00
Sultana et al. [ 14 ]
–
86.86
Aziz et al. [ 4 ]
78.00
90.00
Billah et al. [ 40 ]
–
90.14
Biswas et al. [ 41 ]
–
90.50
TABLE V : Comparison with state-of-the-art works on BanglaSER and SUBESCO datasets
Fig. 2 : t-SNE visualizations of learned feature embeddings for (a) SUBESCO and (b) BanglaSER datasets.
Recognizing emotions from speech using machine learning has become an active research area due to its importance in building human-centered applications. However, while many studies have been conducted in English, German, and other European and Asian languages, research in Arabic remains scarce because of the limited availability of annotated datasets. In this paper, we present an Arabic Speech Emotion Recognition (SER) system based on a hybrid CNN-Transformer architecture. The model leverages convolutional layers to extract discriminative spectral features from Mel-spectrogram inputs and Transformer encoders to capture long-range temporal dependencies in speech. Experiments were conducted on the EYASE (Egyptian Arabic speech emotion) corpus, and the proposed model achieved 97.8% accuracy and a macro F1-score of 0.98. These results demonstrate the effectiveness of combining convolutional feature extraction with attention-based modeling for Arabic SER and highlight the potential of Transformer-based approaches in low-resource languages.
Speech Emotion Recognition (SER) is an important component in a wide range of human-centered applications, including healthcare, customer service, and human-omputer interaction. In medical and decision-support settings, there is increasing interest in models that not only achieve accurate emotion recognition but also support transparent predictions and efficient deployment. However, many existing SER approaches rely on complex deep learning architectures that limit interpretability and increase computational cost. This paper presents an explainable and lightweight speech emotion recognition framework based on a compact convolutional neural network architecture. The proposed approach utilizes log-Mel spectrogram representations to capture spectro-temporal speech characteristics and employs attentive statistics pooling to emphasize emotionally salient temporal segments. To improve model transparency, gradient-based class activation mapping (Grad-CAM) is incorporated to visualize the time-frequency regions that influence the model's predictions. Experimental evaluation on the SAVEE emotional speech dataset demonstrates that the proposed framework achieves competitive recognition performance while maintaining a compact architecture with significantly fewer parameters than many existing SER models. The results indicate that efficient convolutional architectures combined with interpretable analysis can provide a practical balance between recognition accuracy, computational efficiency, and model transparency.
Nelly Elsayed
School of Information Technology University of Cincinnati Ohio, United States
Speech Emotion Recognition (SER) aims to identify a speaker's emotional state from audio signals. While recent advances in deep learning have significantly improved SER performance in Indo-European languages, Arabic SER remains underexplored and challenging due to dialectal diversity, limited annotated datasets, and the difficulty of modeling both local spectral cues and long-range temporal dependencies. To address these limitations, this study investigates whether hybrid architectures that jointly model spatial and contextual information can improve emotion recognition in Arabic speech. We propose and evaluate a comparative framework involving three architectures: a CNN-LSTM model, a CNN-Transformer model, and a fine-tuned wav2vec 2.0 model. The first two models leverage MFCC and spectrogram-based representations, while wav2vec 2.0 operates directly on raw audio through self-supervised representations. Experiments conducted on the EYASE and BAVED datasets demonstrate that the proposed CNN-Transformer architecture significantly outperforms the other models, achieving an accuracy of 98.1 percent. This result highlights the effectiveness of combining convolutional feature extraction with Transformer-based global context modeling. The main contribution of this work lies in providing a systematic comparison of hybrid and self-supervised approaches for Arabic SER, and in demonstrating that CNN-Transformer architectures offer a robust solution for capturing both spectral and long-range dependencies in low-resource and dialectally diverse settings.
Youcef Soufiane Gheffari, Samiya Silarbi
aADASCA Laboratory – Advanced Data Science and Cognitive Applications, Université des Sciences et de la Technologie d’Oran Mohamed Boudiaf (USTO-MB), Oran, Algeria