Your Voice Cloning System is Secretly a Voice Anonymizer

Romolo Muletta; Felix Matthias Saaro; Mark Cieliebak; Jan Deriu

cs.CLAug 27, 2026

Your Voice Cloning System is Secretly a Voice Anonymizer

Authors: Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak, Jan Deriu

Organizations: Centre for Artificial Intelligence ZHAW School of Engineering

Abstract

Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER $\approx$ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.

Figures & tables

Figure 1: Speaker anonymization pipeline using XTTSv2. The original speech is encoded into a codebook representation via VQ-VAE, preserving prosody. The GPT-2 backbone transforms this representation to match the pseudo-speaker identity provided by the Perceiver conditioner.

	Common Voice		MLS
Language	Spkrs	Samples	Spkrs	Samples
German (de)	819	3,003	60	6,863
English (en)	2,538	4,481	84	7,576
Spanish (es)	2,068	6,429	40	4,792
French (fr)	934	3,506	36	4,838
Italian (it)	1,083	6,836	20	2,510

Table 1: Evaluation data: speaker and sample counts across seven languages (dev + test splits combined).

		EER $\uparrow$								WER $\downarrow$								$\Delta$ UTM. $\uparrow$
	Model	de	en	es	fr	it	nl	pt	Avg	de	en	es	fr	it	nl	pt	Avg	Avg
CV	SALT	.37	.35	.38	.38	.37	.34	.38	.37	.21	.37	.18	.32	.23	.17	.35	.26	-0.74
	MultiLingual	.46	.46	.46	.46	.46	.46	.47	.46	.20	.31	.19	.28	.25	.23	.39	.27	-0.62
	XTTSv2 (Ours)	.49	.49	.49	.50	.49	.50	.48	.49	.16	.28	.10	.20	.12	.07	.22	.16	+0.17
MLS	SALT	.35	.29	.32	.35	.27	.16	.36	.30	.09	.12	.06	.12	.17	.23	.13	.13	-1.29
	MultiLingual	.46	.45	.45	.45	.44	.36	.45	.44	.14	.17	.08	.17	.20	.24	.21	.17	-1.23

Table 2: Speaker anonymization results across seven languages (dev+test combined). EER: higher is better (0.5 = optimal privacy). WER: lower is better.

\Delta

UTMOS: naturalness change from original. Best results in bold .

System	CMOS	Std Dev
SALT	$-1.00$	$\pm 1.12$
MultiLingual	$-1.85$	$\pm 0.88$
XTTSv2 (Ours)	$\mathbf{-0.90}$	$\pm 1.39$

Table 3: Comparative Mean Opinion Score (CMOS) on English MLS test samples (30 samples, 9 raters, 270 total comparisons per system). Anonymized speech compared against original recordings. Scale: -3 (much worse) to +3 (much better), 0 (equal quality). Higher values indicate better quality preservation.

Speaker Anonymization

Speaker anonymization changes identifying voice characteristics while preserving speech content and useful traits such as emotion and prosody.

See all 22 papers

NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization
Jul 4, 2026Meiying Melissa Chen, Anastasia Kuznetsova, Zhenyu Wang +1Speaker Anonymization Privacy-Preserving Speech Processing
Department of Electrical and Computer Engineering, University of Rochester Rochester NY, USA
DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Apr 29, 2026Ismail Rasim Ulgen, Zexin Cai, Nicholas Andrews +2Audio Diffusion Models Speech Prosody
Center for Language and Speech Processing, Johns Hopkins University, USA · Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA
Voice "Cloning" is Style Transfer
May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech Generation Trust in AI
Cornell University · TogetherAI · Stanford University

Jul 4, 2026J/K move · Enter open · S save

NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization

Meiying Melissa Chen, Anastasia Kuznetsova, Zhenyu Wang, Zhiyao Duan

Department of Electrical and Computer Engineering, University of Rochester Rochester NY, USA

Speaker Anonymization Privacy-Preserving Speech Processing

Open paper AskPDF arXiv

Advanced neural technologies in speech synthesis and voice conversion (VC) have introduced severe risks to personal privacy, necessitating robust Speaker Anonymization Systems (SAS). Existing SAS approaches modify voice characteristics in the hand-crafted feature space or speaker embedding space, often struggling to provide sufficient identity variance across generated voices. In this paper, we propose NouveauVoice, a novel pseudo-speaker generation framework based on a Hierarchical Deep Variational Autoencoder (NVAE). Integrated as a standalone plug-in module on top of state-of-the-art architectures (FACodec and CosyVoice2), our approach leverages tractable sampling and the Evidence Lower Bound (ELBO) objective to synthesize highly expressive pseudo-speaker embeddings with significantly enhanced speaker diversity. Evaluating our framework under a protocol similar to the VoicePrivacy Challenge alongside Maximum Mean Discrepancy (MMD) analysis, we demonstrate that NouveauVoice achieves strong identity concealment, yielding an Equal Error Rate (EER) exceeding 38% against an automatic speaker verification attacker model. Our system shows a reasonable trade-off between strict anonymity, rich pseudo-speaker diversity, and downstream speech utility, such as intelligibility and emotional expressiveness.

Abstract

Speaker Anonymization

Explore similar work

NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization

DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

Voice "Cloning" is Style Transfer