Voice anonymization aims to protect a speaker's identity while preserving the linguistic content of speech, enabling voice data to be shared and processed with reduced privacy risk. This work focuses on reversible voice anonymization, where the original voice can be recovered by an authorized user rather than being permanently discarded. However, existing approaches can achieve apparent speaker anonymization by degrading the speech itself, while their effectiveness may also depend on the speaker-verification representation used during training. This creates a need for methods that protect speaker identity without sacrificing speech intelligibility and that remain effective against unseen speaker verifiers. We introduce content preserving training constraints to improve the privacy-utility trade-off by preventing anonymization from being achieved primarily through speech degradation. We further introduce a verifier-generalization strategy that incorporates a stronger speaker-verification representation during training, improving anonymization robustness beyond the verifier used in the original training setup. The resulting approach maintains intelligible speech, strengthens anonymization, and preserves reliable voice restoration, with evaluation on unseen speakers and held-out speaker verification models. On 38 unseen speakers, the approach raises the attacker's equal error rate from a measured floor of 2.8% to 26.3%, and to 27.9% against a held-out verifier, with 11.8% word error rate and 94.9% speaker similarity on key-based restoration. These improvements advance reversible voice anonymization toward practical privacy-preserving speech applications, where anonymized speech can be safely processed or shared while the original voice remains recoverable under authorized conditions.
Figures & tables
Fig. 1: The training objective viewed as constraints on the anonymized spectrogram Xa . The consistency loss acts on a second forward pass conditioned on the true speaker embedding s and never sees Xa . The triplet loss acts only on the speaker embedding of Xa . Neither penalizes changes to its content (red). The two proposed losses act on Xa directly (blue).
EER
WER
mel
max
triplet
(%)
(%)
MSE
mel
loss
Losses off
41.60
117.58
6.551
12.4
0.0039
Losses on
21.20
8.29
1.338
8.5
0.163
Starting point
—
—
1.294
9.2
—
TABLE I: Ablation of the two proposed losses. Both arms share initialization, data, batch size and step count, and are evaluated on a 50-utterance subset of the evaluation set (10 speakers, one key seed), where the ASR floor is 2.01% WER. Real log-mel values on this data reach about 10.2. Both arms use the earlier vocoder (see Section VII ).
Metric
Floor
Ours
McAdams
Rano [ 3 ]
EER, ECAPA (%)
2.83
26.31±0.25
8.44
47.81
EER, ResNet (%)
2.08
27.86
6.37
—
WER (%)
2.48
11.75
3.61
11.91
Simspk (%)
94.90
94.90
n/a
87.75
TABLE II: Main results on 38 unseen speakers and 303 utterances. Floors are measured on clean audio. EER is the mean and standard deviation of three seeded runs.
Case
mean Dkey
Ours (%)
Rano [ 3 ] (%)
Correct key
0.00
95.28
87.75
0.1<Dkey<0.2
0.14
4.09
12.78
0.2<Dkey<0.3
0.24
4.41
9.49
0.3<Dkey<0.4
0.35
4.30
4.66
TABLE III: Restoration quality with an incorrect key, over 303 utterances. For incorrect keys, lower is better. A flat response denies an attacker a search gradient. These runs use the earlier vocoder (see Section VII ).
The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech. It relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder. These embeddings are decoded into an anonymized signal using vector quantization and a HiFi-GAN vocoder, both trained on LibriTTS without any waveform reconstruction loss or speaker embedding mapping. The training objective enforces that embeddings of the anonymized signal match those of the original one. While training, an auxiliary speaker classification branch with a gradient reversal layer is used to discard speakerspecific information. Results show that this straightforward embedding-based approach achieves very low WER (2.53) with an anonymization performance (EER 13.39) ranking within first level for VPC. Notably, emotions are partially preserved (UAR 43.91), even without a supporting training objective, while the anonymized voice is audible without reconstruction loss.
Adrien Schneider, Kacper Zabkowski, Anderson Augusma +3
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER ≈ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering
Voice anonymization masks vocal traits while preserving linguistic content, which may still leak speaker-specific patterns. To assess and strengthen privacy evaluation, we propose a dual-stream attacker that fuses spectral and self-supervised learning features via parallel encoders with a three-stage training strategy. Stage I establishes foundational speaker-discriminative representations. Stage II leverages the shared identity-transformation characteristics of voice conversion and anonymization, exposing the model to diverse converted speech to build cross-system robustness. Stage III provides lightweight adaptation to target anonymized data. Results on the VoicePrivacy Attacker Challenge (VPAC) dataset demonstrate that Stage II is the primary driver of generalization, enabling strong attacking performance on unseen anonymization datasets. With Stage III, fine-tuning on only 10% of the target anonymization dataset surpasses current state-of-the-art attackers in terms of EER.
Ridwan Arefeen, Xiaoxiao Miao, Rong Tong +3
Singapore Institute of Technology, Singapore · Duke Kunshan University, China · NVIDIA AI Technology Centre, Singapore