Speech remains intelligible to listeners under heavy temporal degradation, but whether speaker identity survives is largely unexplored. We evaluate six automatic speaker verification (ASV) models, trained on temporally intact speech, and tested under three degradations that remove different combinations of temporal cues. The models form three pairs, one per architecture family (convolutional, transformer, conformer), each pairing a waveform-input with a mel filterbank-input model. All models degrade, but robustness tracks input representation rather than pretraining scale or model size. In all nine family-by-condition comparisons the filterbank-input model degrades less, with eight of nine 95% confidence intervals excluding zero, even when it is smaller or has less pretraining than its waveform counterpart. In the controlled convolutional pair, where the models differ only in input representation, the filterbank-input model's equal error rate rises from 1.1% to 17.1% under the heaviest tested degradation, against 2.9% to29.7% for its waveform counterpart.
Figures & tables
Figure 1: Temporal decomposition in a 15ms window of an utterance from the VoxCeleb1 corpus [ 21 ] .
Unmodified
PER
PER+ENV
PER+TFS
Model
Input
Params
Pretrain
EER
minDCF
EER
minDCF
EER
minDCF
EER
minDCF
Convolutional
ResNet1D*
Waveform
6.20M
–
2.90
0.193
29.74
0.965
28.08
0.930
19.84
0.802
ResNet34*
Filterbank
6.63M
–
1.06
0.072
17.09
0.801
16.68
0.785
10.42
0.579
Transformer
WavLM
Waveform
319.1M
94k
5.48
0.327
36.47
0.971
35.86
0.961
18.36
0.764
Table 1: Speaker verification performance on unmodified and temporally degraded speech, grouped by architecture family. Pretrain gives pretraining data in hours; the convolutional models were trained from scratch (*) and have none. Lower EER and minDCF indicate better performance; the best score in each column is bolded, and evaluations where minDCF exceeds 0.95 are italicized. Within every pair, the waveform-input model degrades more than its filterbank-input counterpart.
Figure 2: The plot visualizes EER (%) by architecture family (Convolutional, Transformer and Conformer), evaluation condition (Unmodified, PER, PER+ENV and PER+TFS), and input representation (Mel filterbank and Waveform). Unmodified is the original VoxCeleb1-O trial audio. The conditions PER+ENV and PER+TFS are parallel restorations from PER, not successive steps. The plot shows that the performance of waveform-input models is more strongly degraded than that of filterbank-input models as temporal cues are removed. Error bars represent 95% speaker-level paired bootstrap confidence intervals over the 40 test speakers.
PER
PER+ENV
PER+TFS
Model
Input
Params
Pretrain
Δ EER
ΔH EER
Δ EER
ΔH EER
Δ EER
ΔH EER
Convolutional
ResNet1D*
Waveform
6.20M
–
26.84
57.0
25.18
53.5
16.94
36.0
ResNet34*
Filterbank
6.63M
–
16.03
32.8
15.63
31.9
9.36
19.1
Transformer
WavLM
Waveform
319.1M
94k
30.99
69.6
30.39
68.3
12.88
28.9
Table 2: Degradation relative to unmodified speech, grouped by architecture family. Δ EER is the absolute increase over each model’s unmodified EER. ΔH EER (%) reports headroom normalized scores defined in Eq. 4 . Pretrain gives pretraining data in hours; the convolutional models were trained from scratch (*) and have none. Lower is better; the smallest degradation in each column is bolded. Within every pair, the waveform-input model degrades more than its filterbank-input counterpart.
Figure 3: The waveform-input models degrade more in every family × condition cell. Only one of the nine contrasts has a confidence interval that includes 0 . Difference in headroom-normalized degradation (mel filterbank − waveform) is plotted. Negative values indicate waveform models degraded more. The convolutional pair in the shaded area is the controlled comparison: identical training data and schedule, differing only in input representation. Whiskers are 95% speaker-level paired bootstrap intervals.
Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonated speech to protect privacy, disrupt others, or when the lack of full vocalization is dictated by a disease. In this paper we propose a model with a training recipe to obtain more robust representations against whispered speech hindrances. The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone, optimized jointly using cosine similarity--based classification and triplet loss. We gain relative improvement of 22.26% compared to the baseline (baseline 6.77% vs ours 5.27%) in normal vs whispered speech trials, achieving AUC of 98.16%. In tests comparing whispered to whispered, our model attains an EER of 1.88% with AUC equal to 99.73%, which represents a 15% relative enhancement over the prior leading ReDimNet-B2. We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech. Additionally, we evaluate how these models perform under noisy audios, obtaining that generally the same relative level of noise degrades the performance of speaker verification more significantly on whispered speech than on normal speech.
Magdalena Gołębiowska, Piotr Syga
Department of Artificial Intelligence, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego 27, Wroclaw, 50-370, Poland
Audio large language models (ALLMs) can understand speech content, yet their ability to use speaker identity for verification remains limited. We propose TS-SP (Two-Stage Speaker Preservation), a parameter-efficient framework for learning speaker-preserving representations and making them accessible to an ALLM's language-model component. We instantiate and evaluate TS-SP on Qwen2.5-Omni-7B. First, we adapt the audio encoder with speaker identity supervision. We then freeze the adapted encoder and train the language model to compare speakers. Both stages use low-rank adaptation (LoRA), keeping the pretrained base weights fixed. On Vox1-O, TS-SP reduces the equal error rate (EER) from 7.01% for the Paired Loss Adaptation Baseline to 4.37%. EER remains within 4.31--4.79% under unseen prompts. Cross-domain evaluation on CN-Celeb yields a similar EER to the baseline, but lower accuracy at the native decision threshold. These findings support two-stage adaptation for improving speaker verification on the evaluated backbone.
Junjie Li, Zheng Liang, Zhe Li +2
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore
Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.
Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA