Speech is a natural means of conveying emotions, making it an effective method for understanding and representing human feelings. Reliable speech emotion recognition (SER) is central to applications in human-computer interaction, healthcare, education, and customer service. However, most SER methods depend on heavy backbone models or hand-crafted features that fail to balance accuracy and efficiency, particularly for low-resource languages like Bangla. In this work, we present SpectroFusion-ViT, a lightweight SER framework built utilizing EfficientViT-b0, a compact Vision Transformer architecture equipped with self-attention to capture long-range temporal and spectral patterns. The model contains only 2.04M parameters and requires 0.1 GFLOPs, enabling deployment in resource-constrained settings without compromising accuracy. Our pipeline first performs preprocessing and augmentation on raw audio, then extracts Chroma and Mel-frequency cepstral coefficient (MFCC) features. These representations are fused into a complementary time-frequency descriptor that preserves both fine-grained spectral detail and broader harmonic structure. Using transfer learning, EfficientViT-b0 is fine-tuned for multi-class emotion classification. We evaluate the system on two benchmark Bangla emotional speech datasets, SUBESCO and BanglaSER, which vary in speaker diversity, recording conditions, and acoustic characteristics. The proposed approach achieves 92.56% accuracy on SUBESCO and 82.19% on BanglaSER, surpassing existing state-of-the-art methods. These findings demonstrate that lightweight transformer architectures can deliver robust SER performance while remaining computationally efficient for real-world deployment.
Figures & tables
Fig. 1 : Overview of the proposed SpectroFusion-ViT framework for speech emotion recognition. Raw audio is first normalized and augmented, after which Chroma and MFCC features are extracted. These complementary representations are fused into a joint time–frequency descriptor and fed into the EfficientViT-b0 backbone for transfer learning–based emotion classification.
Property
Value
Speakers
20 (10 male, 10 female)
Sentences per speaker
10
Emotional classes
7 (anger, disgust, fear,
happiness, neutral, sadness, surprise)
Repetitions per sentence
5
Total utterances
7,000
TABLE I : Properties of the SUBESCO dataset [ 26 ]
Property
Value
No. of Speakers
34 (17 male, 17 female)
Age Range
19–47 years
No. of Utterances
1,467
No. of Classes
5 (angry, happy, sad, surprise, neutral)
Samples per Class
306 (neutral: 243)
Average Duration
3–4 seconds
TABLE II : Properties of the BanglaSER dataset [ 29 ]
BanglaSER
SUBESCO
Architecture
Chroma
MFCC
Combined
Chroma
MFCC
Combined
Inception_v3
53.24
55.91
58.39
59.42
62.07
65.17
MobileNetv2
60.74
65.58
68.06
62.81
66.32
67.29
ResNet50
64.38
67.12
69.30
63.55
65.41
67.58
ResNet34
68.21
70.43
72.84
72.19
74.05
76.50
DenseNet121
65.42
68.37
71.68
77.51
80.26
83.39
TABLE III : Performance comparison of alternative backbone architectures evaluated within the proposed SpectroFusion-ViT framework on the BanglaSER and SUBESCO datasets.
BanglaSER
SUBESCO
Class
Precision
Recall
F1-Score
Precision
Recall
F1-Score
Angry
89
90
90
92
94
93
Happy
78
74
76
90
90
90
Neutral
84
88
86
94
93
93
Sad
78
79
79
90
90
90
Surprise
77
76
77
90
92
91
TABLE IV : Class-wise precision, recall, and F1-score (%) for BanglaSER and SUBESCO datasets
Accuracy (%)
Architecture
BanglaSER
SUBESCO
Chakraborty et al. [ 5 ]
79.00
75.00
Sultana et al. [ 14 ]
–
86.86
Aziz et al. [ 4 ]
78.00
90.00
Billah et al. [ 40 ]
–
90.14
Biswas et al. [ 41 ]
–
90.50
TABLE V : Comparison with state-of-the-art works on BanglaSER and SUBESCO datasets
Fig. 2 : t-SNE visualizations of learned feature embeddings for (a) SUBESCO and (b) BanglaSER datasets.
aADASCA Laboratory – Advanced Data Science and Cognitive Applications, Université des Sciences et de la Technologie d’Oran Mohamed Boudiaf (USTO-MB), Oran, Algeria