Input Representation, Not Model Size or Pretraining Scale, Shapes Speaker Verification Robustness to Temporal Degradation

Erdem Baha Topbas; Srikanth Madikeri; Elisa Pellegrino; Volker Dellwo

cs.SDOct 8, 2026

Input Representation, Not Model Size or Pretraining Scale, Shapes Speaker Verification Robustness to Temporal Degradation

Authors: Erdem Baha Topbas, Srikanth Madikeri, Elisa Pellegrino, Volker Dellwo

Organizations: Department of Computational Linguistics, University of Zurich

Abstract

Speech remains intelligible to listeners under heavy temporal degradation, but whether speaker identity survives is largely unexplored. We evaluate six automatic speaker verification (ASV) models, trained on temporally intact speech, and tested under three degradations that remove different combinations of temporal cues. The models form three pairs, one per architecture family (convolutional, transformer, conformer), each pairing a waveform-input with a mel filterbank-input model. All models degrade, but robustness tracks input representation rather than pretraining scale or model size. In all nine family-by-condition comparisons the filterbank-input model degrades less, with eight of nine 95% confidence intervals excluding zero, even when it is smaller or has less pretraining than its waveform counterpart. In the controlled convolutional pair, where the models differ only in input representation, the filterbank-input model's equal error rate rises from 1.1% to 17.1% under the heaviest tested degradation, against 2.9% to29.7% for its waveform counterpart.

Figures & tables

Figure 1: Temporal decomposition in a

15\text{\,}\mathrm{ms}

window of an utterance from the VoxCeleb1 corpus [ 21 ] .

				Unmodified		PER		PER+ENV		PER+TFS
Model	Input	Params	Pretrain	EER	minDCF	EER	minDCF	EER	minDCF	EER	minDCF
Convolutional
ResNet1D*	Waveform	6.20M	–	2.90	0.193	29.74	0.965	28.08	0.930	19.84	0.802
ResNet34*	Filterbank	6.63M	–	1.06	0.072	17.09	0.801	16.68	0.785	10.42	0.579
Transformer
WavLM	Waveform	319.1M	94k	5.48	0.327	36.47	0.971	35.86	0.961	18.36	0.764

Table 1: Speaker verification performance on unmodified and temporally degraded speech, grouped by architecture family. Pretrain gives pretraining data in hours; the convolutional models were trained from scratch (*) and have none. Lower EER and minDCF indicate better performance; the best score in each column is bolded, and evaluations where minDCF exceeds

0.95

are italicized. Within every pair, the waveform-input model degrades more than its filterbank-input counterpart.

Figure 2: The plot visualizes EER (%) by architecture family (Convolutional, Transformer and Conformer), evaluation condition (Unmodified, PER, PER+ENV and PER+TFS), and input representation (Mel filterbank and Waveform). Unmodified is the original VoxCeleb1-O trial audio. The conditions PER+ENV and PER+TFS are parallel restorations from PER, not successive steps. The plot shows that the performance of waveform-input models is more strongly degraded than that of filterbank-input models as temporal cues are removed. Error bars represent

95\text{\,}\mathrm{\%}

speaker-level paired bootstrap confidence intervals over the

40

test speakers.

				PER		PER+ENV		PER+TFS
Model	Input	Params	Pretrain	$\Delta$ EER	$\Delta_{\mathrm{H}}$ EER	$\Delta$ EER	$\Delta_{\mathrm{H}}$ EER	$\Delta$ EER	$\Delta_{\mathrm{H}}$ EER
Convolutional
ResNet1D*	Waveform	6.20M	–	26.84	57.0	25.18	53.5	16.94	36.0
ResNet34*	Filterbank	6.63M	–	16.03	32.8	15.63	31.9	9.36	19.1
Transformer
WavLM	Waveform	319.1M	94k	30.99	69.6	30.39	68.3	12.88	28.9

Table 2: Degradation relative to unmodified speech, grouped by architecture family.

\Delta

EER is the absolute increase over each model’s unmodified EER.

\Delta_{\mathrm{H}}

EER (%) reports headroom normalized scores defined in Eq. 4 . Pretrain gives pretraining data in hours; the convolutional models were trained from scratch (*) and have none. Lower is better; the smallest degradation in each column is bolded. Within every pair, the waveform-input model degrades more than its filterbank-input counterpart.

Figure 3: The waveform-input models degrade more in every family

\times

condition cell. Only one of the nine contrasts has a confidence interval that includes

0

. Difference in headroom-normalized degradation (mel filterbank

-

waveform) is plotted. Negative values indicate waveform models degraded more. The convolutional pair in the shaded area is the controlled comparison: identical training data and schedule, differing only in input representation. Whiskers are

95\text{\,}\mathrm{\%}

speaker-level paired bootstrap intervals.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

Enhancing Speaker Verification with Whispered Speech via Post-Processing
Apr 22, 2026Magdalena Gołębiowska, Piotr SygaAudio Representation Learning Speaker Verification
Department of Artificial Intelligence, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego 27, Wroclaw, 50-370, Poland
TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models
Oct 4, 2026Junjie Li, Zheng Liang, Zhe Li +2Speaker Verification Audio-Language Models
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore
Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker Embedding Speaker Verification
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

Apr 22, 2026J/K move · Enter open · S save

Enhancing Speaker Verification with Whispered Speech via Post-Processing

Magdalena Gołębiowska, Piotr Syga

Department of Artificial Intelligence, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego 27, Wroclaw, 50-370, Poland

Audio Representation Learning Speaker Verification Speech Processing

Open paper AskPDF arXiv

Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonated speech to protect privacy, disrupt others, or when the lack of full vocalization is dictated by a disease. In this paper we propose a model with a training recipe to obtain more robust representations against whispered speech hindrances. The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone, optimized jointly using cosine similarity--based classification and triplet loss. We gain relative improvement of 22.26% compared to the baseline (baseline 6.77% vs ours 5.27%) in normal vs whispered speech trials, achieving AUC of 98.16%. In tests comparing whispered to whispered, our model attains an EER of 1.88% with AUC equal to 99.73%, which represents a 15% relative enhancement over the prior leading ReDimNet-B2. We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech. Additionally, we evaluate how these models perform under noisy audios, obtaining that generally the same relative level of noise degrades the performance of speaker verification more significantly on whispered speech than on normal speech.

Abstract

Speaker Verification

Explore similar work

Enhancing Speaker Verification with Whispered Speech via Post-Processing

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation