SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

Eunseo Choi; Hyunku Kang; Chanwoo Kim

cs.SDAug 31, 2026

SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

Authors: Eunseo Choi, Hyunku Kang, Chanwoo Kim

Organizations: Korea Univiersity, South Korea

Abstract

Speech emotion recognition (SER) faces two fundamental challenges: scarcity of labeled data and inter-speaker variability, both of which hinder generalization of emotion recognition systems. While prior adversarial approaches address speaker variability, they fall short in leveraging powerful pre-trained representations. We propose SISER (Speaker-Invariant Speech Emotion Recognition), integrating wav2vec 2.0 as a feature encoder and ECAPA-TDNN as a speaker discriminator within an entropy-based adversarial training scheme. wav2vec 2.0 provides rich self-supervised representations that alleviate dependency on large labeled datasets, while ECAPA-TDNN enables suppression of speaker identity via a stronger adversarial signal than shallow classifiers. Evaluated on IEMOCAP, SISER achieves a UA of 60.63%, outperforming the baseline (51.15%) and wav2vec 2.0 without speaker suppression (56.46%), with ablation emphasizing that the choice of speaker classifier architecture is a key factor.

Figures & tables

Figure 1: Overview of the proposed framework. Red and blue arrows denote the two alternating training steps: red for Step 1 (SC update with ENC and EC frozen) and blue for Step 2 (ENC and EC update with SC frozen).

Feature Extractor	Data Aug	Emotion Classifier	Speaker Classifier	SI Method	Valid		Test
Feature Extractor	Data Aug	Emotion Classifier	Speaker Classifier	SI Method	UA(%) $\uparrow$	WA(%) $\uparrow$	UA(%) $\uparrow$	WA(%) $\uparrow$
CNN + GRUs [ 12 ]	O	FC	FC	Entropy	60.24	58.85	59.91	\textcolor blue58.62
CNN + GRUs (baseline) [ 12 ]	X	FC	FC	Entropy	54.51	54.20	51.15	50.14
wav2vec 2.0 (vanila)	X	FC	-	-	59.22	59.17	56.46	54.45
wav2vec 2.0	X	FC	ECAPA-TDNN + Linear	GRL	58.93	58.69	56.95	56.01
wav2vec 2.0 (ours)	X	FC	ECAPA-TDNN + Linear	Entropy	\textcolor blue62.37	\textcolor blue61.64	\textcolor blue60.63	58.53

Table 1: Performance comparison on IEMOCAP under a speaker-independent 10-fold cross-validation protocol (UA/WA in %). Five systems are compared: the baseline reproduced from [ 12 ] with and without data augmentation (O/X), a wav2vec 2.0 encoder without speaker disentanglement (vanilla), and wav2vec 2.0 with ECAPA-TDNN trained via Gradient Reversal Layer (GRL) or entropy maximization (SISER). Results are reported on both validation and test sets. Best results per set are highlighted in bold.

Panel of Figure 2 — Figure 2: t-SNE visualizations of utterance-level embeddings for two speakers in Session 1. Color indicates emotion category and marker shape indicates speaker identity ( $\circ$ : 01F, $\triangle$ : 01M).

	FC	ECAPA-TDNN	$\Delta$ (Ecapa gain)
CNN+GRU	55.09	61.64	+7.13
wav2vec 2.0	56.24	62.73	+6.49
$\Delta$ (wav2vec 2.0 gain)	+1.73	+1.09

Table 2: Ablation study on Fold 2 (UA in %). Two encoder types (CNN+GRU vs. wav2vec 2.0) and two speaker classifier configurations (FC vs. ECAPA-TDNN) are compared.

\Delta

values indicate the gain from replacing each component.

N-Fold	Baseline	Vanila	Ours	Total $\Delta$
Fold 1	49.11 (-5.09)	58.63 (-0.59)	61.14 (-1.59)	7.27
\textcolor blue Fold 2	55.09 (+0.89)	57.22 (-2.00)	63.05 (+0.32)	\textcolor blue 3.21
Fold 3	58.81 (+4.61)	58.09 (-1.13)	55.90 (-6.83)	12.57
Fold 4	57.14 (+2.94)	62.08 (+2.86)	60.63 (-2.10)	7.90
Fold 5	50.19 (-4.01)	54.02 (-5.20)	55.73 (-7.00)	16.21
Fold 6	53.82 (-0.38)	57.71 (-1.51)	57.22 (-5.51)	7.40

Table 3: 10-fold experiment results

Explore similar work

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition
Jun 4, 2026Jinyi Mi, Ding Ma, Tomoki TodaZero-Shot Learning Cross-Lingual Representation Alignment
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge Distillation Speech Emotion Recognition
Department of Electrical Engineering, The City College of New York · National Taiwan University · Wyze Inc. +1
SIGMA: Saliency-Guided Sparse Mask Attacks for Speech Emotion Recognition
Jun 29, 2026Qiyang Sun, Yi Chang, Zixing Zhang +1Adversarial Attacks Speech Emotion Recognition
GLAM, Department of Computing, Imperial College London, UK · College of Computer Science and Electronic Engineering, Hunan University, China · Shenzhen Research Institute, Hunan University, China +4

Jun 4, 2026J/K move · Enter open · S save

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition

Jinyi Mi, Ding Ma, Tomoki Toda

Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan

Zero-Shot Learning Cross-Lingual Representation Alignment Speech Emotion Recognition Invariant Representation Learning

Open paper AskPDF arXiv

Zero-shot cross-lingual speech emotion recognition (SER) remains challenging due to distribution mismatches across languages and the lack of emotion annotations in target language. Under such conditions, models trained solely on source-language data frequently suffer from degraded generalization when evaluated on unseen target languages. To address this limitation, we propose an emotion-discriminative representation learning method that integrates supervised contrastive learning and speaker adversarial learning. The contrastive learning promotes cross-lingual emotion alignment, while speaker adversarial learning suppresses speaker-related cues to encourage speaker-invariant representations. Experimental results under a zero-shot cross-lingual SER setting demonstrate that the proposed method significantly improves SER performance over conventional training strategies.

Abstract

Explore similar work

Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

SIGMA: Saliency-Guided Sparse Mask Attacks for Speech Emotion Recognition