Koopman Regularized Deep Speech Disentanglement for Speaker Verification
Authors: Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers
Organizations: School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, U.K. · Marine Geosciences, National Oceanography Centre, SO14 3ZH Southampton, U.K.
Human speech contains both linguistic content and speaker dependent characteristics making speaker verification a key technology in identity critical applications. Modern deep learning speaker verification systems aim to learn speaker representations that are invariant to semantic content and nuisance factors such as ambient noise. However, many existing approaches depend on labelled data, textual supervision or large pretrained models as feature extractors, limiting scalability and practical deployment, raising sustainability concerns. We propose Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE), a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics. Quantitative experiments across multiple datasets demonstrate that DKSD-AE achieves improved or competitive speaker verification performance compared to state-of-the-art baselines while maintaining high content EER, confirming effective disentanglement. These results are obtained with substantially fewer parameters and without textual supervision. Moreover, performance remains stable under increased evaluation scale, highlighting representation robustness and generalization. Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.
Figures & tables
Fig. 1: The DKSD-AE framework architecture. Speech utterances are processed with VAD and then mel-spectrograms are extracted. Following, the input mel-spectrograms X are fed to the dynamics encoder fdyn , to learn the Koopman operator K and the speaker identity representation Zs . Concurrently, X is fed to fc , the content encoder, to learn a content representation Zc via instance normalization. Finally, Zs and Zc are concatenated and fed to the decoder, qdec , to generate reconstructed mel-spectrograms X .
Fig. 2: Processing steps to estimate the multi-step prediction loss Lpred . From top to bottom, first a prefix Zspre is extracted from the full dynamics representation Zs . Then Zspre is split into two subsequent time shifted views, used to solve the the Koopman operator regression shown in Eq. ( 7 ).
Component
LSTM Blocks
Residual Blocks
fc
256,128,128,64
64,64
fdyn
256,128
128,128,64,64,64,64,64
gdec
128
64,64,128
TABLE I: DKSD-AE architecture number of layers and hidden node dimensions.
Method
EER Speaker ↓
EER Content ↑
Modality Type
Parameters
DKSD-AE (Ours)
2.77%
44.0%
Mel Spectrograms
3.5M
SpeechTripleNet [ 20 ]
7.01%
45.5%
Mel Spectrograms, Pitch and Energy Contours
19.3M
VAE-TP [ 24 ]
2.90%
44.1%
Mel spectrograms and Text Annotations
399M
UTTS [ 5 ]
3.90%
43.2%
WavLM [ 4 ] Features and Text Annotations
> 94M
TABLE II: Best run performance of proposed approach against the VCTK baseline models in speaker verification. DKSD-AE performs better in the SV task than competing approaches without the need for text-annotations or additional modalities. Best performance in each EER category is in bold.
Method
EER Speaker ↓
EER Content ↑
Parameters
DKSD-AE (Ours)
3.90 %
45.8 %
3.5M
DSVAE [ 13 ]
4.78 %
17.84 %
21M
C-DSVAE [ 23 ]
4.03 %
31.81 %
11M
SKD [ 34 ]
4.46 %
26.78 %
2M
D-DSVAE [ 22 ]
3.25 %
38.83 %
ND
TABLE III: Our model performs on par or better than state-of-the-art approaches, with signficant improvements in the parameter footprint compared to VAE variants.
Dataset
EER Speaker ↓
EER Content ↑
VCTK
3.44±0.56 %
44.70±1.50 %
TIMIT Official
4.68±0.69 %
46.04±0.59 %
TIMIT Full
5.64±0.27 %
45.94±0.62 %
TABLE IV: DKSD-AE shows consistent EER performance across VCTK, TIMIT and TIMIT-TEST on different random initializations.
Fig. 3: Visual comparison of speaker Zs (left) and content representations Zc (right), after dimensionality reduction with PCA and t-SNE [ 48 ] . Same speaker representations Zs (left) form compact well-separated clusters in 2D, whereas content representations Zc are dispersed without speaker-specific grouping (right). Different colours correspond to different speaker classes.
Dataset
Koopman Learning
EER Speaker
EER Content
Ltotal
3.66±0.55 %
44.36±0.93 %
VCTK
Lpred
5.27±0.49 %
35.04±3.97 %
Lrec
4.65±0.25 %
39.04±1.95 %
Ltotal
4.68±0.69 %
46.04±0.59 %
TIMIT
Lpred
7.15±0.50 %
39.82±1.98 %
Lrec
6.47±1.18 %
43.39±3.38 %
TABLE V: The complete objective yields improved SV performance and stronger disentanglement compared to both a reconstruction-only baseline and a Koopman autoencoder trained solely with Lpred across different datasets and different seeds.
Fig. 4: Our multi-step Koopman operator learning module formulation presents improvements in speaker EER when the forecasting horizon M∈[5,15] for VCTK and when M∈[5,13] for TIMIT although less pronounced. The identified effective M -range corresponds to forecasting horizon length between 5% and 10% of the initial speech utterance duration.
Automatic speaker verification must remain reliable across devices, rooms, and compression pipelines. We present ReDimNet2+, which scales training of the compact ReDimNet2 backbone across seven public corpora (63,934 speakers, about 8,675 hours). Analysis of a VoxBlink2 subset reveals a shift in predicted spectral coloration, motivating codec and waveform augmentation alongside this multi-corpus training, large-margin fine-tuning (LMFT), and graph-based retrieval reranking. With random 4-second evaluation windows for all models, ReDimNet2+ LMFT reduces pooled VoxCeleb1 EER from 2.42% to 0.82% and a 26-condition robustness stress-test EER from 7.21% to 1.99%. Under this shared local protocol, it reaches 0.35% EER on VoxCeleb1-O versus 0.787% for the best evaluated WeSpeaker checkpoint. On a VoxBlink2 retrieval subset, reranking improves the final model's Pr@k from 0.7413 to 0.7687.
Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.
Diego Torres, Axel Roebel, Nicolas Obin
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Short-duration speaker verification (SDSV) is crucial for personalized keyword spotting, where test utterances are typically shorter than three seconds. Limited speech duration results in unstable speaker representations and increased sensitivity to noise and phoneme variations, thereby degrading performance. To investigate this issue, we construct VoxPhrase, a large-scale SDSV corpus automatically segmented from the VoxCeleb dataset. Our analysis shows that text-dependent (TD) enrollment is constrained by duration and yields unstable speaker representations. In contrast, although text-independent (TI) enrollment introduces content mismatch, its representations become more stable as the enrollment duration increases. Accordingly, we propose a hybrid-enrollment neural re-scoring framework that combines TD and TI enrollment and performs frame-level comparison via parallel cross-attention. Experiments on VoxPhrase demonstrate consistent improvements across multiple speaker models.
Zhiqi Ai, Han Cheng, Shiyi Mu +3
Shanghai University, China · Hithink RoyalFlush AI Research Institute, China · Xi’an Jiaotong-Liverpool University, China