Koopman Regularized Deep Speech Disentanglement for Speaker Verification

Nikos Chazaridis; Mohammad Belal; Rafael Mestre; Timothy J. Norman; Christine Evers

cs.SDMar 5, 2026

Koopman Regularized Deep Speech Disentanglement for Speaker Verification

Authors: Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers

Organizations: School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, U.K. · Marine Geosciences, National Oceanography Centre, SO14 3ZH Southampton, U.K.

Abstract

Human speech contains both linguistic content and speaker dependent characteristics making speaker verification a key technology in identity critical applications. Modern deep learning speaker verification systems aim to learn speaker representations that are invariant to semantic content and nuisance factors such as ambient noise. However, many existing approaches depend on labelled data, textual supervision or large pretrained models as feature extractors, limiting scalability and practical deployment, raising sustainability concerns. We propose Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE), a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics. Quantitative experiments across multiple datasets demonstrate that DKSD-AE achieves improved or competitive speaker verification performance compared to state-of-the-art baselines while maintaining high content EER, confirming effective disentanglement. These results are obtained with substantially fewer parameters and without textual supervision. Moreover, performance remains stable under increased evaluation scale, highlighting representation robustness and generalization. Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.

Figures & tables

Fig. 1: The DKSD-AE framework architecture. Speech utterances are processed with VAD and then mel-spectrograms are extracted. Following, the input mel-spectrograms $\mathbf{X}$ are fed to the dynamics encoder $f_{\text{dyn}}$ , to learn the Koopman operator $\mathbf{K}$ and the speaker identity representation $\mathbf{Z}_{s}$ . Concurrently, $\mathbf{X}$ is fed to $f_{c}$ , the content encoder, to learn a content representation $\mathbf{Z}_{c}$ via instance normalization. Finally, $\mathbf{Z}_{s}$ and $\mathbf{Z}_{c}$ are concatenated and fed to the decoder, $q_{\text{dec}}$ , to generate reconstructed mel-spectrograms $\mathbf{\widehat{X}}$ .

Fig. 2: Processing steps to estimate the multi-step prediction loss

\mathcal{L}_{\text{pred}}

. From top to bottom, first a prefix

\mathbf{Z}_{s}^{\mathrm{pre}}

is extracted from the full dynamics representation

\mathbf{Z}_{s}

. Then

\mathbf{Z}_{s}^{\mathrm{pre}}

is split into two subsequent time shifted views, used to solve the the Koopman operator regression shown in Eq. ( 7 ).

Component	LSTM Blocks	Residual Blocks
$f_{\text{c}}$	256,128,128,64	64,64
$f_{\text{dyn}}$	256,128	128,128,64,64,64,64,64
$g_{\text{dec}}$	128	64,64,128

TABLE I: DKSD-AE architecture number of layers and hidden node dimensions.

Method	EER Speaker $\downarrow$	EER Content $\uparrow$	Modality Type	Parameters
DKSD-AE (Ours)	$\mathbf{2.77\,\%}$	$\underline{44.0\,\%}$	Mel Spectrograms	3.5M
SpeechTripleNet [ 20 ]	$7.01\,\%$	$\mathbf{45.5}\,\%$	Mel Spectrograms, Pitch and Energy Contours	19.3M
VAE-TP [ 24 ]	$2.90\,\%$	$44.1\,\%$	Mel spectrograms and Text Annotations	399M
UTTS [ 5 ]	$3.90\,\%$	$43.2\,\%$	WavLM [ 4 ] Features and Text Annotations	$>$ 94M

TABLE II: Best run performance of proposed approach against the VCTK baseline models in speaker verification. DKSD-AE performs better in the SV task than competing approaches without the need for text-annotations or additional modalities. Best performance in each EER category is in bold.

Method	EER Speaker $\downarrow$	EER Content $\uparrow$	Parameters
DKSD-AE (Ours)	3.90 %	$\mathbf{45.8}$ %	3.5M
DSVAE [ 13 ]	4.78 %	17.84 %	21M
C-DSVAE [ 23 ]	4.03 %	31.81 %	11M
SKD [ 34 ]	4.46 %	26.78 %	2M
D-DSVAE [ 22 ]	$\mathbf{3.25}$ %	38.83 %	ND

TABLE III: Our model performs on par or better than state-of-the-art approaches, with signficant improvements in the parameter footprint compared to VAE variants.

Dataset	EER Speaker $\downarrow$	EER Content $\uparrow$
VCTK	${3.44}\pm 0.56$ %	${44.70}\pm 1.50$ %
TIMIT Official	$4.68\pm 0.69$ %	$46.04\pm 0.59$ %
TIMIT Full	$5.64\pm 0.27$ %	$45.94\pm 0.62$ %

TABLE IV: DKSD-AE shows consistent EER performance across VCTK, TIMIT and TIMIT-TEST on different random initializations.

Fig. 3: Visual comparison of speaker

\mathbf{Z}_{s}

(left) and content representations

\mathbf{Z}_{c}

(right), after dimensionality reduction with PCA and t-SNE [ 48 ] . Same speaker representations

\mathbf{Z}_{s}

(left) form compact well-separated clusters in 2D, whereas content representations

\mathbf{Z}_{c}

are dispersed without speaker-specific grouping (right). Different colours correspond to different speaker classes.

Dataset	Koopman Learning	EER Speaker	EER Content
	$\mathcal{L}_{\text{total}}$	$\mathbf{3.66}\pm{0.55}$ %	$\mathbf{44.36}\pm 0.93$ %
VCTK	$\mathcal{L}_{\text{pred}}$	$5.27\pm 0.49$ %	$35.04\pm 3.97$ %
	$\mathcal{L}_{\text{rec}}$	$4.65\pm 0.25$ %	$39.04\pm 1.95$ %
	$\mathcal{L}_{\text{total}}$	$\mathbf{4.68}\pm 0.69$ %	$\mathbf{46.04}\pm 0.59$ %
TIMIT	$\mathcal{L}_{\text{pred}}$	$7.15\pm 0.50$ %	$39.82\pm 1.98$ %
	$\mathcal{L}_{\text{rec}}$	$6.47\pm 1.18$ %	$43.39\pm 3.38$ %

TABLE V: The complete objective yields improved SV performance and stronger disentanglement compared to both a reconstruction-only baseline and a Koopman autoencoder trained solely with

\mathcal{L}_{\text{pred}}

across different datasets and different seeds.

Fig. 4: Our multi-step Koopman operator learning module formulation presents improvements in speaker EER when the forecasting horizon

M\in[5,15]

for VCTK and when

M\in[5,13]

for TIMIT although less pronounced. The identified effective

M

-range corresponds to forecasting horizon length between

5\%

and

10\%

of the initial speech utterance duration.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification
Sep 29, 2026Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov +1Speaker Verification
lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia
A Comprehensive Study of Content Representations for Speech Synthesis
Sep 25, 2026Diego Torres, Axel Roebel, Nicolas ObinAudio Representation Learning Discrete Speech Representation Learning
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Stabilizing Short Duration Speaker Verification through Neural Re-scoring with Hybrid Enrollment
Jun 15, 2026Zhiqi Ai, Han Cheng, Shiyi Mu +3Speaker Verification Speech Processing
Shanghai University, China · Hithink RoyalFlush AI Research Institute, China · Xi’an Jiaotong-Liverpool University, China

Sep 29, 2026J/K move · Enter open · S save

ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification

Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov, Grach Mkrtchian

lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia

Speaker Verification

Open paper AskPDF arXiv

Automatic speaker verification must remain reliable across devices, rooms, and compression pipelines. We present ReDimNet2+, which scales training of the compact ReDimNet2 backbone across seven public corpora (63,934 speakers, about 8,675 hours). Analysis of a VoxBlink2 subset reveals a shift in predicted spectral coloration, motivating codec and waveform augmentation alongside this multi-corpus training, large-margin fine-tuning (LMFT), and graph-based retrieval reranking. With random 4-second evaluation windows for all models, ReDimNet2+ LMFT reduces pooled VoxCeleb1 EER from 2.42% to 0.82% and a 26-condition robustness stress-test EER from 7.21% to 1.99%. Under this shared local protocol, it reaches 0.35% EER on VoxCeleb1-O versus 0.787% for the best evaluated WeSpeaker checkpoint. On a VoxBlink2 retrieval subset, reranking improves the final model's Pr@k from 0.7413 to 0.7687.

Abstract

Speaker Verification

Explore similar work

ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification

A Comprehensive Study of Content Representations for Speech Synthesis

Stabilizing Short Duration Speaker Verification through Neural Re-scoring with Hybrid Enrollment