Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

Nikita Kuzmin; Tao Zhong; Jiajun Deng; Yingke Zhu; Tristan Tsoi; Tianxiang Cao; Simon Lui; Kong Aik Lee

eess.ASMar 9, 2026

Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

Authors: Nikita Kuzmin, Tao Zhong, Jiajun Deng, Yingke Zhu, Tristan Tsoi, Tianxiang Cao, Simon Lui, Kong Aik Lee,

Organizations: Nanyang Technological University, Singapore · A*STAR, Singapore · Huawei Leibniz Research Center · The Chinese University of Hong Kong ·

Abstract

End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).

Figures & tables

Panel of Figure 1 — Figure 1: Speaker privacy ( $1-\mathrm{Linkability}$ ) vs. dialogue turn count. Only discrete encoder variants are shown; continuous encoder Linkability is omitted for clarity. Red lines: no anonymization; green lines: post-anonymization. Without anonymization, both systems drop into the low-privacy zone within a few turns; anonymization lifts privacy into the protected zone, though Moshi + W2W shows gradual degradation over dialogue length.

Figure 2: Overview of the original SALM-Duplex pipeline and proposed anonymization setups. The main diagram shows the ASR-based encoder baseline: an ECAPA-TDNN probe attached to the LLM’s hidden states (red dashed path) reveals substantial speaker identity leakage. The Anon-W2W inset (upper left) prepends Stream-Voice-Anon to anonymize the waveform before the unchanged ASR encoder. The Anon-W2F inset (upper right) replaces the ASR encoder with the Stream-Voice-Anon encoder (anonymization active) and fine-tunes the LLM, eliminating the redundant waveform synthesis step. (Anon-W2F is demonstrated for SALM-Duplex; the Anon-W2W setup is additionally evaluated on Moshi.)

Model	User	Anonymization	Privacy		Quality				Efficiency
Name	Encoder	Type	EER $\uparrow$	Lnk $\downarrow$	sBLEU $\uparrow$		sBERT $\uparrow$		RTFx $\uparrow$	FRL $\downarrow$	TTSR $\uparrow$	Int.L. $\downarrow$	ISR $\uparrow$
					S2T	S2S	S2T	S2S
	discrete	–	6.4	0.90	7.18	6.85	0.50	0.48	17	0.50	0.85	1.24	0.45
Moshi	discrete	W2W	36.9	0.35	5.20	5.00	0.39	0.38	1.6	0.72	0.72	1.31	0.38
	discrete	–	11.2	0.79	4.18	3.15	0.45	0.29	238	0.44	0.85	1.03	0.96
	discrete	W2F	41.0	0.23	3.57	1.49	0.39	0.20	2.5	0.50	0.68	1.18	0.93

Table 1: Privacy, dialogue quality, and efficiency on the VPC2024 evaluation set. Privacy: higher EER and lower Linkability (Lnk) = stronger privacy; EER = 50% = perfect anonymization. Linkability for no-anon rows uses the “orig” attacker; for anonymized rows, the “lazy-informed” attacker [ 10 ] . Quality: higher sBLEU / sBERT = better. Efficiency: RTFx = 1/RTF; FRL = First Response Latency (s); TTSR = Turn-Taking Success Rate; Int.L. = Interruption Latency (s); ISR = Interruption Success Rate. Bold = best per column; underline = second best. Shaded rows = anonymized conditions.

System	Encoder	Anon.	Early	Mid	Late	All
	discrete	–	7.3	5.6	6.4	6.4
Moshi	discrete	W2W	42.5	37.6	35.2	36.9
	discrete	–	7.5	14.0	20.1	11.2
	discrete	W2F	43.8	40.5	40.1	41.0
	continuous	–	24.6	28.6	32.1	28.5
SALM- [-2pt]Duplex	continuous	W2W	31.5	33.7	35.3	34.6

Table 2: Layer-wise EER (%) of the speaker verification attacker. Higher = better privacy; 50% = chance level. All = mean-pooled over all layers. Bold = best; underline = second best. Shaded rows = anonymized conditions.

Privacy-Preserving Speech Processing

See all 21 papers

StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation
Mar 6, 2026Nikita Kuzmin, Kong Aik Lee, Eng Siong ChngSpeaker Anonymization Privacy-Preserving Speech Processing
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore · The Hong Kong Polytechnic University, Hong Kong
DAST: A Dual-Stream Voice Anonymization Attacker with Staged Training
Mar 13, 2026Ridwan Arefeen, Xiaoxiao Miao, Rong Tong +3Privacy-Preserving Speech Processing Speaker Anonymization
Singapore Institute of Technology, Singapore · Duke Kunshan University, China · NVIDIA AI Technology Centre, Singapore
Your Voice Cloning System is Secretly a Voice Anonymizer
Aug 27, 2026Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1Speaker Anonymization Privacy-Preserving Speech Processing
Centre for Artificial Intelligence ZHAW School of Engineering

Mar 6, 2026J/K move · Enter open · S save

StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation

Nikita Kuzmin, Kong Aik Lee, Eng Siong Chng

Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore · The Hong Kong Polytechnic University, Hong Kong

Speaker Anonymization Privacy-Preserving Speech Processing Neural Audio Codecs

Open paper AskPDF arXiv

We address the challenge of preserving emotional content in streaming speaker anonymization (SA). Neural audio codec language models trained for audio continuation tend to degrade source emotion: content tokens discard emotional information, and the model defaults to dominant acoustic patterns rather than preserving paralinguistic attributes. We propose supervised finetuning with neutral-emotion utterance pairs from the same speaker, combined with frame-level emotion distillation on acoustic token hidden states. All modifications are confined to finetuning, which takes less than 2 hours on 4 GPUs and adds zero inference latency overhead, while maintaining a competitive 180ms streaming latency. On the VoicePrivacy 2024 protocol, our approach achieves a 49.2% UAR (emotion preservation) with 5.77% WER (intelligibility), a +24% relative UAR improvement over the baseline (39.7%->49.2%) and +10% over the emotion-prompt variant (44.6% UAR), while maintaining strong privacy (EER 49.0%). Demo and code are available: https://anonymous3842031239.github.io/

Abstract

Privacy-Preserving Speech Processing

Explore similar work

StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation

DAST: A Dual-Stream Voice Anonymization Attacker with Staged Training

Your Voice Cloning System is Secretly a Voice Anonymizer