SETEAB: Multiscale approach with Squeeze-and-Excitation Temporal Enhanced Aware Block for Speech Emotion Recognition

Duy Vo; Kiet Anh Hoang; Hao Do

eess.ASSep 5, 2026

SETEAB: Multiscale approach with Squeeze-and-Excitation Temporal Enhanced Aware Block for Speech Emotion Recognition

Authors: Duy Vo, Kiet Anh Hoang, Hao Do

Organizations: Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · UNEY, Switzerland

Abstract

This paper proposes a novel lightweight multiscale architecture for speech emotion recognition (SER) with three key innovations. First, a depthwise convolution-based subsampling module is introduced to reduce model size and computation while preserving salient emotional cues. Second, a Squeeze-and-Excitation block is integrated to enhance channel-wise recalibration and improve representation robustness. Third, a new Temporal Enhanced Aware Block is designed to strengthen temporal dependency modeling and produce more discriminative emotion-aware features. The proposed model is explicitly designed to jointly improve compactness, recognition performance, and generalizability. Experiments on benchmark SER datasets show that our method achieves higher accuracy with reduced computational complexity, while also delivering stronger cross-corpus performance than most recent advanced networks for SER.

Figures & tables

Figure 1: Overview of the proposed SETEAB for speech emotion recognition with four key improvements: depthwise convolution subsampling, SE-Res2Block, the proposed Temporal Enhanced Aware Block (TEAB), and weighted bidirectional fusion (BiF).

Model	Comp. (M/G)	EMOVO		IEMOCAP		RAVDESS		CREMA-D		MELD		Avg
Model	Comp. (M/G)	UA	F1	UA	F1	UA	F1	UA	F1	UA	F1	UA	F1
wav2vec 2.0 base [ 8 ]	$\sim$ 95 / 33.53	31.07	27.24	58.27	57.83	54.33	53.99	61.95	61.75	20.06	20.04	45.14	44.17
TIM-Net [ 15 ]	$\sim$ 0.12 / 0.05	21.94	15.71	54.91	52.57	50.67	49.84	66.00	65.68	17.60	15.37	42.22	39.83
MS-SENet [ 16 ]	$\sim$ 0.15 / 0.13	22.11	13.77	55.48	54.07	54.62	53.77	65.10	64.74	17.55	14.73	42.97	40.22
SETEAB ( $R=2$ )	$\sim$ 0.5 / 0.12	35.20	31.61	58.07	56.26	54.95	53.26	67.29	67.04	22.97	17.51	47.69	45.14
SETEAB ( $R=4$ )	$\sim$ 0.4 / 0.06	30.27	24.94	58.94	57.12	59.57	59.64	66.73	66.55	22.54	17.89	47.61	45.23

Table 1: Comparison of UA and F1 (%) across five SER benchmarks. Model complexity is reported as the number of parameters(M) and FLOPs(G), computed on 5-second input utterances. The best and second-best results in each column are highlighted in bold and underlined , respectively.

R

denotes the subsampling rate. This convention is followed in all subsequent tables.

Figure 2: Comparison of performance–efficiency trade-off in terms of FLOPs(G) and UA% across baseline and our models.

Setting	FLOPs(G)	$\mathbf{UA_{avg}}$	$\mathbf{F1_{avg}}$
DW-Sub ( $R=2$ ) + BiF + SE-Res2	0.12	47.69	45.14
DW-Sub ( $R=4$ ) + BiF + SE-Res2	0.06	47.61	45.23
BiF	0.14	45.89	43.68
BiF + SE-Res2	0.16	46.22	43.20
Bi-DW-Sub + BiF + SE-Res2	0.17	46.62	43.96

Table 2: Ablation study of the proposed model variants on SER datasets, reported in terms of average UA (%) and F1 (%).

Test	Training Set
Test	wav2vec 2.0 base				SETEAB (ours)
	I	M	R	S	I	M	R	S
I	$\backslash$	29.78	18.25	28.84	$\backslash$	47.08	37.50	30.42
M	22.50	$\backslash$	31.39	35.24	35.00	$\backslash$	27.08	30.00
R	27.15	23.20	$\backslash$	33.77	48.33	44.17	$\backslash$	47.08
S	31.34	29.19	21.36	$\backslash$	44.17	27.50	32.08	$\backslash$

Table 3: Cross-dataset evaluation (WA%) across four datasets (IEMOCAP (I), MELD (M), RAVDESS (R), SAVEE (S)).

Model name	WA
wav2vec 2.0 base	27.67 $\pm$ 5.27
TIM-Net	32.22 $\pm$ 5.18
MS-SENet	34.31 $\pm$ 4.35
SETEAB (ours)	37.53 $\pm$ 8.20

Table 4: Overall cross-dataset WA (%) of different models, reported as mean

\pm

standard deviation.

Explore similar work

Explainable Lightweight Compact Deep Models for Speech Emotion Recognition
Jul 18, 2026Nelly ElsayedSpeech Emotion Recognition Convolutional Neural Networks
School of Information Technology University of Cincinnati Ohio, United States
Speech Emotion Recognition using Attention-based LSTM-Network with Residual Connection
Jun 2, 2026Daniil Krasnoproshin, Maxim VashkevichSpeech Emotion Recognition Long Short-Term Memory Networks
Embedded System Department of Belarusian State University of Informatics and Radioelectronics 6, P. Brovky st., 220013, Minsk, Belarus
AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge Distillation Speech Emotion Recognition
Department of Electrical Engineering, The City College of New York · National Taiwan University · Wyze Inc. +1

Jul 18, 2026J/K move · Enter open · S save

Explainable Lightweight Compact Deep Models for Speech Emotion Recognition

Nelly Elsayed

School of Information Technology University of Cincinnati Ohio, United States

Speech Emotion Recognition Convolutional Neural Networks Lightweight CNNs

Open paper AskPDF arXiv

Speech Emotion Recognition (SER) is an important component in a wide range of human-centered applications, including healthcare, customer service, and human-omputer interaction. In medical and decision-support settings, there is increasing interest in models that not only achieve accurate emotion recognition but also support transparent predictions and efficient deployment. However, many existing SER approaches rely on complex deep learning architectures that limit interpretability and increase computational cost. This paper presents an explainable and lightweight speech emotion recognition framework based on a compact convolutional neural network architecture. The proposed approach utilizes log-Mel spectrogram representations to capture spectro-temporal speech characteristics and employs attentive statistics pooling to emphasize emotionally salient temporal segments. To improve model transparency, gradient-based class activation mapping (Grad-CAM) is incorporated to visualize the time-frequency regions that influence the model's predictions. Experimental evaluation on the SAVEE emotional speech dataset demonstrates that the proposed framework achieves competitive recognition performance while maintaining a compact architecture with significantly fewer parameters than many existing SER models. The results indicate that efficient convolutional architectures combined with interpretable analysis can provide a practical balance between recognition accuracy, computational efficiency, and model transparency.

Abstract

Explore similar work

Explainable Lightweight Compact Deep Models for Speech Emotion Recognition

Speech Emotion Recognition using Attention-based LSTM-Network with Residual Connection

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition