Toward Complete Speaker Anonymization: a Streaming Cascaded System for Voice Privacy and PII Removal
Authors: Stanisław Kacprzak, Thomas Thebaud, Najim Dehak
Organizations: Center for Language and Speech Processing, Johns Hopkins University, Baltimore, Maryland, USA · AGH University of Krakow, Institute of Electronics, Krakow, Poland · AMIAD, Palaiseau, FR
We present a cascade streaming voice privacy system, consisting of ASR, NLP, and TTS modules. By leveraging recent improvements in all three aspects of speech and language processing, we show that such a system is capable of achieving a result comparable to state-of-the-art offline voice privacy systems, while providing controllable capabilities of content protection beyond voice. With an estimated utterance-onset end-to-end latency of 717 ms, the presented system preserves speech content with 11% WER on Librispeech, while detecting PII with a 69% SLU F1 score and achieving up to 47.66% EER against a semi-informed attacker.
Figures & tables
ASR [ms]
TTS (TTFA) [ms]
E2E [ms]
Tout/Tin
669
41
711
1.33
Table 1: Median of measurements for latency, TTFA and the duration ratio of output and input audio.
System
WER ↓
UAR ↑
dev
test
dev
test
Orig.
1.80
1.84
69.08
71.06
T30 [ 13 ]
5.02
4.78
28.45
29.44
T8-4 [ 30 ]
3.74
3.75
30.10
30.59
Ours (Streaming)
11.11
11.44
29.37
29.22
Table 2: Utility metrics evaluating linguistic preservation (WER) on LibriSpeech and emotion preservation (UAR) on IEMOCAP, on the original speech.
Split
Orig.
T30 [ 13 ]
T8-4 [ 30 ]
B5 [ 28 ]
Ours
dev-f ↑
10.51
44.32
47.58
35.82
44.18
dev-m ↑
0.93
47.52
49.23
32.92
47.66
test-f ↑
8.76
45.07
47.44
33.95
44.37
test-m ↑
0.42
44.77
48.75
34.73
45.25
Table 3: Privacy results are shown as an EER, evaluated on the Librispeech subsets.
SLURP
DialogPII
Model
Clean transcript
ASR chunks
Simulated Streaming
Entity F1 ↑
SLU F1 ↑
Entity F1 ↑
SLU F1 ↑
Entity F1 ↑
SLU F1 ↑
Latency [ms] ↓
Qwen 1.5B
87.50
89.10
51.48
66.51
56.27
64.22
46
Qwen 0.5B
83.92
85.42
48.33
62.73
46.55
56.62
39
spaCy
87.19
88.98
54.82
69.01
53.27
60.60
4
GLiNER Small v2.5
82.55
87.55
50.59
66.93
61.22
69.14
6
Table 4: DialogPII and SLURP PII detection results for fine-tuned models.
Speaker anonymization (SA) aims to conceal speaker identity while preserving linguistic content and emotion, but existing streaming systems often lose emotion information through discrete content bottlenecks or require emotion-specific modeling. We present LiveAnon, a voice-conversion-based streaming SA system that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder, preserving emotion without emotion-specific supervision. To reduce codec-generation latency, we introduce capped RVQ decoding, which preserves dependencies among coarse codebooks while predicting the remaining codebooks in parallel. Under VPC24, LiveAnon improves intelligibility and emotion preservation over fixed-target Stream-Voice-Anon (SVA), a strong prior streaming SA system, matches emotion-prompted SVA in emotion preservation, and achieves the strongest semi-informed privacy among streaming systems. Under VPC26, it ranks second across all metrics among the evaluated anonymization systems, demonstrating a strong privacy-utility balance.
Yejin Kim, Kyungsu Kim, Seungu Han +1
Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER ≈ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering
End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).
Nikita Kuzmin, Tao Zhong, Jiajun Deng +6
Nanyang Technological University, Singapore · A*STAR, Singapore · Huawei Leibniz Research Center +2