Authors: Nikita Kuzmin, Tao Zhong, Jiajun Deng, Yingke Zhu, Tristan Tsoi, Tianxiang Cao, Simon Lui, Kong Aik Lee, Eng Siong Chng
Organizations: Nanyang Technological University, Singapore · A*STAR, Singapore · Huawei Leibniz Research Center · The Chinese University of Hong Kong · The Hong Kong Polytechnic University
End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).
Figures & tables
Figure 1: Speaker privacy ( 1−Linkability ) vs. dialogue turn count. Only discrete encoder variants are shown; continuous encoder Linkability is omitted for clarity. Red lines: no anonymization; green lines: post-anonymization. Without anonymization, both systems drop into the low-privacy zone within a few turns; anonymization lifts privacy into the protected zone, though Moshi + W2W shows gradual degradation over dialogue length.
Figure 2: Overview of the original SALM-Duplex pipeline and proposed anonymization setups. The main diagram shows the ASR-based encoder baseline: an ECAPA-TDNN probe attached to the LLM’s hidden states (red dashed path) reveals substantial speaker identity leakage. The Anon-W2W inset (upper left) prepends Stream-Voice-Anon to anonymize the waveform before the unchanged ASR encoder. The Anon-W2F inset (upper right) replaces the ASR encoder with the Stream-Voice-Anon encoder (anonymization active) and fine-tunes the LLM, eliminating the redundant waveform synthesis step. (Anon-W2F is demonstrated for SALM-Duplex; the Anon-W2W setup is additionally evaluated on Moshi.)
Model
User
Anonymization
Privacy
Quality
Efficiency
Name
Encoder
Type
EER ↑
Lnk ↓
sBLEU ↑
sBERT ↑
RTFx ↑
FRL ↓
TTSR ↑
Int.L. ↓
ISR ↑
S2T
S2S
S2T
S2S
discrete
–
6.4
0.90
7.18
6.85
0.50
0.48
17
0.50
0.85
1.24
0.45
Moshi
discrete
W2W
36.9
0.35
5.20
5.00
0.39
0.38
1.6
0.72
0.72
1.31
0.38
discrete
–
11.2
0.79
4.18
3.15
0.45
0.29
238
0.44
0.85
1.03
0.96
discrete
W2F
41.0
0.23
3.57
1.49
0.39
0.20
2.5
0.50
0.68
1.18
0.93
Table 1: Privacy, dialogue quality, and efficiency on the VPC2024 evaluation set. Privacy: higher EER and lower Linkability (Lnk) = stronger privacy; EER = 50% = perfect anonymization. Linkability for no-anon rows uses the “orig” attacker; for anonymized rows, the “lazy-informed” attacker [ 10 ] . Quality: higher sBLEU / sBERT = better. Efficiency: RTFx = 1/RTF; FRL = First Response Latency (s); TTSR = Turn-Taking Success Rate; Int.L. = Interruption Latency (s); ISR = Interruption Success Rate. Bold = best per column; underline = second best. Shaded rows = anonymized conditions.
System
Encoder
Anon.
Early
Mid
Late
All
discrete
–
7.3
5.6
6.4
6.4
Moshi
discrete
W2W
42.5
37.6
35.2
36.9
discrete
–
7.5
14.0
20.1
11.2
discrete
W2F
43.8
40.5
40.1
41.0
continuous
–
24.6
28.6
32.1
28.5
SALM- [-2pt]Duplex
continuous
W2W
31.5
33.7
35.3
34.6
Table 2: Layer-wise EER (%) of the speaker verification attacker. Higher = better privacy; 50% = chance level. All = mean-pooled over all layers. Bold = best; underline = second best. Shaded rows = anonymized conditions.
We address the challenge of preserving emotional content in streaming speaker anonymization (SA). Neural audio codec language models trained for audio continuation tend to degrade source emotion: content tokens discard emotional information, and the model defaults to dominant acoustic patterns rather than preserving paralinguistic attributes. We propose supervised finetuning with neutral-emotion utterance pairs from the same speaker, combined with frame-level emotion distillation on acoustic token hidden states. All modifications are confined to finetuning, which takes less than 2 hours on 4 GPUs and adds zero inference latency overhead, while maintaining a competitive 180ms streaming latency. On the VoicePrivacy 2024 protocol, our approach achieves a 49.2% UAR (emotion preservation) with 5.77% WER (intelligibility), a +24% relative UAR improvement over the baseline (39.7%->49.2%) and +10% over the emotion-prompt variant (44.6% UAR), while maintaining strong privacy (EER 49.0%). Demo and code are available: https://anonymous3842031239.github.io/
Nikita Kuzmin, Kong Aik Lee, Eng Siong Chng
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore · The Hong Kong Polytechnic University, Hong Kong
Voice anonymization masks vocal traits while preserving linguistic content, which may still leak speaker-specific patterns. To assess and strengthen privacy evaluation, we propose a dual-stream attacker that fuses spectral and self-supervised learning features via parallel encoders with a three-stage training strategy. Stage I establishes foundational speaker-discriminative representations. Stage II leverages the shared identity-transformation characteristics of voice conversion and anonymization, exposing the model to diverse converted speech to build cross-system robustness. Stage III provides lightweight adaptation to target anonymized data. Results on the VoicePrivacy Attacker Challenge (VPAC) dataset demonstrate that Stage II is the primary driver of generalization, enabling strong attacking performance on unseen anonymization datasets. With Stage III, fine-tuning on only 10% of the target anonymization dataset surpasses current state-of-the-art attackers in terms of EER.
Ridwan Arefeen, Xiaoxiao Miao, Rong Tong +3
Singapore Institute of Technology, Singapore · Duke Kunshan University, China · NVIDIA AI Technology Centre, Singapore
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER ≈ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering