Authors: Chanwoo Hwang, Sunpill Kim, Yong Kiam Tan, Tianchi Liu, Seunghun Paik, Dongsoo Kim, Mondal Soumik, Khin Mi Mi Aung, Jae Hong Seo
Organizations: Hanyang University Seoul, Republic of Korea · I2R, A*STAR & NTU Singapore · National University of Singapore Singapore · I2R, A*STAR Singapore
Advances in deep learning have enabled the widespread deployment of speaker recognition systems (SRSs), yet they remain vulnerable to score-based impersonation attacks. Existing attacks that operate directly on raw waveforms require a large number of queries due to the difficulty of optimizing in high-dimensional audio spaces. Latent-space optimization within generative models offers improved efficiency, but these latent spaces are shaped by data distribution matching and do not inherently capture speaker-discriminative geometry. As a result, optimization trajectories often fail to align with the adversarial direction needed to maximize victim scores. To address this limitation, we propose an inversion-based generative attack framework that explicitly aligns the latent space of the synthesis model with the discriminative feature space of SRSs. We first analyze the requirements of an inverse model for score-based attacks and introduce a feature-aligned inversion strategy that geometrically synchronizes latent representations with speaker embeddings. This alignment ensures that latent updates directly translate into score improvements. Moreover, it enables new attack paradigms, including subspace-projection-based attacks, which were previously infeasible due to the absence of a faithful feature-to-audio mapping. Experiments show that our method significantly improves query efficiency, achieving competitive attack success rates with on average 10x fewer queries than prior approaches. In particular, the enabled subspace-projection-based attack attains up to 91.65% success using only 50 queries. These findings establish feature-aligned inversion as a key tool for evaluating the robustness of modern SRSs against score-based impersonation threats.
Figures & tables
Method
Victim Voice Required
Query Output
#Queries
( Kinnunen et al., 2019 )
Yes
–
0
( Yoon et al., 2020 )
–
0
( Pani et al., 2023 )
–
0
SV2TTS ( Jia et al., 2018 )
–
0
Voxstructor ( Lu et al., 2021 )
Feature vector
≥100K
AdvTTS ( Zuo et al., 2024 )
–
0
Table 1. Comparison of speaker impersonation attacks against SRSs. The reported number of queries corresponds to the cost of performing the attack task as defined in each original work. Ours-NES and Ours-SP denote two optimization strategies used in our attack: Natural Evolution Strategies (NES) and Subspace Projection (SP), respectively. Details of these optimization methods are described in Section 6.1 . †: We consider FakeBob applied to speaker verification here.
Figure 1. Distributions of round-trip similarity scores sL(v) (red) evaluated on the local speaker recognition model FL . Positive (blue) and negative (orange) reference distributions are shown for comparison. Best viewed in color.
Figure 2. Distributions of round-trip similarity scores sT(v) (red) evaluated on the FT=T1 . Positive (blue) and negative (orange) reference distributions are shown for comparison. Best viewed in color.
Figure 3. Optimization trajectories of score-based impersonation attacks on various search spaces. Curves show the mean of the best cosine distance with 95% CI. Best viewed in color.
Figure 4. Fixed-text Strategy for Fine Tuning Inverse Model.
Figure 5. ID-Constraints Loss LIC . Given an audio batch Abatch , the cosine similarities for each sample pair are computed using the set of synthesized samples Abatch generated through the process illustrated in Figure 4 .
Figure 6. ID-Constraints Loss LIC . Given an audio batch Abatch , the cosine similarities for each sample pair are computed using the set of synthesized samples Abatch generated through the process illustrated in Figure 4 .
Model
Architecture
Trainset
τE
τM
T1
Redim-S ( Yakovlev et al., 2024 )
VoxBlink2
0.6605
0.5518
T2
Redim-M ( Yakovlev et al., 2024 )
VoxBlink2
0.6624
0.5437
T3
SimAMResNet34 ( Qin et al., 2022 )
VoxBlink2
0.6256
0.5060
T4
SimAMResNet100 ( Qin et al., 2022 )
VoxBlink2
0.6135
0.4932
T5
Titanet-L ( Koluguri et al., 2022 )
VoxCeleb1,2+ α
0.6654
0.5215
L
H/ASP ( Heo et al., 2020 )
VoxCeleb1,2
0.5778
0.4667
Table 2. Specifications of target SR models ( T1 – T5 ) and local model ( L ), including architectures, trainsets, and thresholds defined on EER, minDCF (resp. τE , τM ). Detailed EER and minDCF performance metrics are in Table 13 .
Figure 7. Attack success rate as a function of the number of score queries for NES-based impersonation attacks. Results are shown for two decision thresholds, τE (left) and τM (right), across five target speaker recognition models ( T1 – T5 ).
Method
T1
T2
T3
T4
T5
Avg.
τtarget=τE
Audio-NES
100.0 % (5.2k)
100.0 % (4.3k)
100.0 % (8.7k)
100.0 % (9.5k)
100.0 % (3.8k)
100.0 % (6.3k)
YourTTS-NES
100.0 % (2.5k)
100.0 % (3.4k)
96.0% (4.1k)
93.0% (4.2k)
96.0% (7.9k)
97.0% (4.4k)
Ours-NES
100.0 % (0.2k)
100.0 % (0.3k)
100.0 % (0.4k)
100.0 % (0.5k)
100.0 % (0.3k)
100.0 % (0.3k)
τtarget=τM
Audio-NES
100.0 % (8.5k)
100.0 % (7.6k)
100.0 % (16.3k)
100.0 % (16.1k)
100.0 % (19.3k)
100.0 % (13.6k)
Table 3. Comparison of ASR and query efficiency under τtarget=τE and τM . The query budget is 50k. Each cell reports the ASR (top) and the average number of queries calculated over successful attacks (bottom).
Method
T1
T2
T3
T4
T5
τE
τM
τE
τM
τE
τM
τE
τM
τE
τM
SV2TTS
1.57%
0.04%
0.87%
0.02%
0.59%
0.00%
0.42%
0.00%
0.02%
0.00%
Voxstructor
0.11%
0.00%
0.02%
0.00%
0.02%
0.00%
0.01%
0.00%
0.00%
0.00%
Audio-GD
4.65%
0.15%
6.37%
0.19%
1.38%
0.02%
0.30%
0.00%
34.36%
4.89%
YourTTS
15.99%
2.70%
11.62%
1.57%
10.58%
1.72%
7.99%
1.57%
0.28%
0.00%
Ours-SP
91.65%
59.83%
85.47%
44.33%
76.23%
34.54%
62.23%
19.86%
68.46%
17.03%
Table 4. ASRs comparing the proposed method against alternative baseline approaches attacking T1 – T5 . Target thresholds ( τtarget ) are derived from EER ( τE ) and minDCF ( τM ) for each target model.
Attack Method
F−1
Target SR models
T1
T2
T3
T4
T5
NES
YourTTS
93.0% (8.9k)
82.0% (11.5k)
72.0% (8.9k)
79.0% (11.3k)
37.0% (18.8k)
+LIC
100% (1.7k)
100% (1.9k)
100% (2.5k)
99.0% (2.8k)
6.0% (1.3k)
+LIC+LSC
100% (0.5k)
100% (0.6k)
100% (0.9k)
100% (1.0k)
100% (0.8k)
SP
YourTTS
15.99%
11.62%
10.58%
7.99%
0.34%
+LIC
71.37%
56.29%
45.58%
29.21%
64.80%
Table 5. Ablation study on inverse-model training objectives. For NES-based, we report ASR (%) at τM together with the average number of score queries over successful attacks (in parentheses). For SP-based, we report ASR (%) at τE . LIC : Identity-Constraint loss, LSC : Structure-Constraint loss.
Model
Architecture
Trainset
τE
τM
C1
CAM++ ( Wang et al., 2023b )
N/A
0.7064
0.4817
C2
ERes2Net ( Chen et al., 2023b )
N/A
0.6350
0.4617
LCN
ResNet34 ( Chung et al., 2020 )
CN-Celeb 1,2
0.7349
0.5043
Table 6. Specifications of CN target SR models ( C1 - C2 ) and local model ( LCN ), including architectures, trainsets, and thresholds defined on EER, minDCF (resp. τE , τM ).
Method
C1
C2
τE
τM
τE
τM
Ours-NES
100% (0.2k)
100% (1.4k)
100% (0.2k)
100% (0.6k)
Ours-SP
71.43%
4.08%
87.76%
16.84%
Table 7. Attack success rates (ASR, %) on Chinese speaker recognition systems. For Ours-NES, the average number of score queries computed over successful attacks is reported in parentheses. Target thresholds are derived from EER ( τE ) and minDCF ( τM ).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Impersonation Type
Target Model
Victim’s Voice Requirement
Strategy
Query to Target Model
Output Format
Number
( Kinnunen et al., 2019 )
Voice Mimicry
Black-box
◯
Human Mimicry
-
0
( Yoon et al., 2020 )
Replay Attack
Black-box
◯
Playback
-
0
( Pani et al., 2023 )
Voice Morphing
Black-box
◯
Identity Morphing
-
0
SV2TTS ( Jia et al., 2018 )
Speech Synthesis
White-box
◯
Training-based
-
0
Voxstructor ( Lu et al., 2021 )
Speech Synthesis
Black-box
◯
Training-based
Feature vector
≥100K
Appendix
Table 8. Comparison of prior attack methods enabling speaker impersonation against SRSs, summarizing their methodologies, requirements (victim voice, target model access), and query strategies. The details of FakeBob are given for its speaker verification setting, which we denote as FakeBob † in this table.
TTS
Input
(Speaker) Encoder
Auxiliary Input
Output
Tacotron2 ( Shen et al., 2018 )
Text
None
None
Mel-spectrogram Waveform
FastSpeech2 ( Ren et al., 2020 )
Text
None
None
Mel-spectrogram Waveform
XTTS ( Casanova et al., 2024 )
Text, Voice Sample
H/ASP ( Heo et al., 2020 ) , Perceiver Conditioner
Speaker Feature Vector Identity, Style, Prosody, etc.
Waveform
EmoTTS ( Osman, 2022 )
Text, Voice Sample
SER model (based on Deep Voice3 ( Ping et al., 2017 ) )
Emotion, Speaker Embedding
Mel-spectrogram
NaturalSpeech3 ( Ju et al., 2024 )
Text, Voice Sample
Convolutional Blocks
Content, Prosody, Timbre, Acoustic Details
Waveform
StyleTTS2 ( Li et al., 2023 )
Text, Voice Sample
Acoustic and Prosodic, Style Encoders
Style Embedding
Waveform
Appendix
Table 9. Other considered TTS models for Inverse Model Candidates.
Scores
SV2TTS
Voxstructor
YourTTS
Pos.
0.7795±0.0771
0.6470±0.1212
0.6549±0.1007
Neg.
0.5766±0.0707
0.0267±0.1119
0.1383±0.1095
ID-const.
0.6944±0.0761
0.0472±0.0879
0.4015±0.0923
Appendix
Table 10. ID-constraint test statistics for selected ZS-TTS on VoxCeleb1 test dataset.
Figure 8. Distributions of round-trip similarity scores sT(v) (red) evaluated on the each FT . Positive (blue) and negative (orange) reference distributions are shown for comparison.
Hyperparameter
Value
Search variable
waveform aud∈R48000
Initialization
N(0,I48000)
Candidate pool size C
100
Selected candidates S
1
Maximum iterations
1000
Samples per draw B
10, 50
Appendix
Table 11. Hyperparameters for Audio-NES
Hyperparameter
Value
Search variable
latent vector z∈SD
Initialization
N(0,ID) + ℓ2 normalization
Candidate pool size C
100
Selected candidates S
1
Maximum iterations
1000
Samples per draw B
10, 50
Appendix
Table 12. Hyperparameters for latent(spherical) NES. (for Ours-NES, YourTTS-NES)
Index
Architecture
Trainset
EER(%)
minDCF
τE
τM
T1
Redim-S
VoxBlink2
1.54
0.1415
0.6605
0.5518
T2
Redim-M
VoxBlink2
1.31
0.1003
0.6624
0.5437
T3
SimAMResNet34
VoxBlink2
1.16
0.1039
0.6258
0.5060
T4
SimAMResNet100
VoxBlink2
0.76
0.0767
0.6137
0.4932
T5
Titanet-L
VoxCeleb1,2, Fisher, Switchboard, LibriSpeech
0.82
0.1353
0.6654
0.5215
C1
CAM++
N/A
4.70
0.3120
0.7064
0.4817
Appendix
Table 13. Specifications and performance of the speaker recognition models used in this paper, including the target models ( T1 – T5 ) and the attacker’s local models ( LSV2TTS , LVox , and LOurs ). Note that ‘ LOurs ’ is referred to as ‘ L ’ in the main paper. Performance is evaluated using EER, minDCF, and the corresponding thresholds τE and τM on the VoxCeleb1 test set. For the Chinese speaker recognition models ( C1 , C2 ) and the local model ( LCN ), performance is evaluated on the CN-Celeb test set.
Methods
T1
T2
T3
T4
T5
Ours
46/50
44/50
37/50
36/50
36/50
FakeBob
43/50
43/50
32/50
29/50
34/50
# of Queries
≈14233
≈14067
≈14073
≈13181
≈11552
Appendix
Table 14. FakeBob attack performance against target models T1 – T5 , showing attack successes (out of 50 trials) and the number of queries required solely for the attack generation phase (excluding threshold estimation).
Figure 10. Flow diagram of the proposed Non-Adaptive Speaker Impersonation Attack applied to SRSs (Ours-SP). Key stages include pre-processing for δ -OVS generation, querying the target SRS, and synthesizing the impersonation audio. Note that the attacker cannot directly access the black-box target T (and FT ). The local models ( FL and FL−1 ) are used as surrogate for the attack.
T
LSV2TTS
LVox
LOurs
T1
0.3304±0.2017
0.0617±0.0479
0.0741±0.0582
T2
0.3305±0.2076
0.0588±0.0461
0.0757±0.0594
T3
0.2999±0.2143
0.0675±0.0517
0.0691±0.0523
T4
0.2935±0.2248
0.0722±0.0528
0.0760±0.0548
T5
0.3461±0.2191
0.0484±0.0387
0.0804±0.0664
Appendix
Table 15. Statistical summary of score discrepancies between local and target models ( T1 – T5 ) on the VoxCeleb1 test set. Each entry reports the mean and std. of the absolute differences in cosine similarity scores across all pairs.
Figure 11. Scatter plots of the cosine similarity values output by local models for all pairs from the Voxceleb1 testset; R is the correlation coefficient between scores output by L different target models ( T1 - T5 ).
F
⟨x′,x⟩
T1
0.3134
T2
0.3112
T3
0.3114
T4
0.3004
T5
0.3151
Appendix
Table 16. Cosine similarity between the recovered feature x′ and original feature x using a fixed pseudo-inversion matrix A constructed from LOurs . The condition number of A is cond(A)=3.8692 and remains constant across all rows.
Feat.
YourTTS
YourTTS +LIC
YourTTS +LIC+LSC
Org.
0.4011±0.0919
0.9026±0.0197
0.8390±0.0285
+10∘
0.3959±0.0922
0.8978±0.0204
0.8349±0.0291
+20∘
0.3804±0.0925
0.8832±0.0225
0.8220±0.0309
+30∘
0.3556±0.0937
0.8559±0.0267
0.7973±0.0346
+40∘
0.3213±0.0949
0.8114±0.0345
0.7555±0.0418
Appendix
Table 17. Analysis of ID-constraint robustness under noise with random noise ( 10∘ to 40∘ ).
Decision
VoxCeleb1 Testset
YourTTS
Ours
Spoof
8.05%
55.90%
12.55%
Bonafide
91.95%
44.10%
87.45%
Appendix
Table 18. Results (%) of the Deepfake detection ( Tak et al., 2022 ) of audio samples classified as ‘Spoof’ or ‘Bonafide’ for genuine VoxCeleb1 audio, audio generated by YourTTS, and audio generated by our proposed method (‘Ours’).
Evaluation
Threshold
ASR(%)
FRR(%)
EER
7.73
0.4246
60.24
(= EER)
minDCF
0.6428
0.6484
0.06
60.26
Appendix
Table 19. Cross-lingual attack performance against a ResNet34 model trained on CNCeleb. Thresholds and baseline metrics were derived using the ASR was computed under these thresholds.
Deep learning-based audio classification systems, including automatic speaker verification, are vulnerable to adversarial attacks. Realistic real-time threat assessment remains difficult because optimization-based methods, such as projected gradient descent (PGD) and Carlini-Wagner, require costly iterative updates in the high-dimensional waveform domain. Generative attacks allow single-shot synthesis but often introduce perceptible artifacts or depend on computationally intensive architectures, while diffusion and autoregressive approaches incur high inference latency. To address this gap, we propose a generative attack framework operating in the continuous latent space of a neural audio codec. A conditional generator synthesizes class-specific perturbations in a single forward pass and decodes them into adversarial waveforms. Our method achieves targeted attack success rates up to 99% with sub-7 ms inference, outperforming generative baselines while reducing latency by 24x.
Sophisticated generative speech technology can undermined the reliability of voice biometrics. While spoofing detection systems excel when assessed under in-domain conditions, generalisation to out-of-domain settings is often poor. In this paper, we show that such issues could be caused by speaker bias, where models learn individual voice traits rather than markers of manipulation or generation. We propose a teacher-student framework for speaker-invariant spoofing detection that disentangles identity without requiring speaker labels. We leverage a pre-trained speaker recognition teacher to guide a student model via a gradient reversal layer. To control the balance between suppressing cues related to voice identity with the preservation of those related to spoofing detection, we integrate a Variational Information Bottleneck. Evaluations across nine datasets show our model achieves a 25.7% relative reduction to the EER compared to the MHFA baseline.
Anh-Tuan Dao, Driss Matrouf, Mickael Rouvier +1
Laboratoire Informatique d’Avignon, Avignon Universite, France · EURECOM, Sophia Antipolis, France
The rapid advancement of generative AI has made audio deepfakes increasingly indistinguishable from authentic human vocals, posing significant threats to persons-of-interest (POI) such as public figures. Current detection systems primarily rely on generic, black-box models that fail to capture speaker-specific idiosyncratic traits and lack interpretability. In this paper, we propose Phoneme-based Voice Profiling (PVP), a novel personalized defense framework. By shifting the detection paradigm from macro-utterance analysis to micro-phonetic modeling, PVP captures the unique acoustic distributions underlying a POI's habitual articulatory patterns. Specifically, our framework models speaker-specific phonetic realizations using lightweight Gaussian Mixture Models (GMMs) estimated solely from bona fide reference speech. This design enables data-efficient profiling and robust generalization to previously unseen spoofing attacks without requiring heavy spoof-specific training. Furthermore, we introduce the first large-scale Chinese POI deepfake dataset to benchmark speaker-specific detection. Experimental results demonstrate that PVP significantly outperforms state-of-the-art generic detectors in POI spoofing scenarios, achieving substantial EER reductions while providing fine-grained, phoneme-level interpretability for forensic analysis. Code and data are available at: https://github.com/JunXue-tech/PVP
Jun Xue, Tong Zhang, Zhuolin Yi +4
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education · School of Cyber Science and Engineering, Wuhan University