LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models
Authors: Yejin Kim, Kyungsu Kim, Seungu Han, Kyogu Lee
Organizations: Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Speaker anonymization (SA) aims to conceal speaker identity while preserving linguistic content and emotion, but existing streaming systems often lose emotion information through discrete content bottlenecks or require emotion-specific modeling. We present LiveAnon, a voice-conversion-based streaming SA system that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder, preserving emotion without emotion-specific supervision. To reduce codec-generation latency, we introduce capped RVQ decoding, which preserves dependencies among coarse codebooks while predicting the remaining codebooks in parallel. Under VPC24, LiveAnon improves intelligibility and emotion preservation over fixed-target Stream-Voice-Anon (SVA), a strong prior streaming SA system, matches emotion-prompted SVA in emotion preservation, and achieves the strongest semi-informed privacy among streaming systems. Under VPC26, it ranks second across all metrics among the evaluated anonymization systems, demonstrating a strong privacy-utility balance.
Figures & tables
Figure 1: Overview of LiveAnon during (a) training and (b) inference. A 3-s reference utterance provides codec tokens for acoustic prompting, and features from a frozen streaming ASR encoder provide content conditioning. Training uses teacher-forced codec sequences, whereas inference pre-fills the reference prompt into the Transformer KV cache and continues autoregressive generation with the capped RVQ delay pattern.
System
Speaker Acc. ↓
Emotion UAR ↑
SVA
32.50
39.95
LiveAnon
28.75
55.17
Table 1: Linear probing of content representations. Lower speaker accuracy and higher emotion UAR are preferred.
System
WER ↓
UAR ↑
EER-L ↑
EER-S ↑
Original
1.83
70.07
5.16
–
DarkStream [ 28 ]
8.75
34.73
47.26
21.83
TVTSyn [ 29 ]
5.35
37.32
47.55
14.57
SVA [ 10 ] cremad-emo-4rnd
6.59
44.59
46.53
18.63
SVA [ 10 ] vctk-1fix
4.54
39.71
47.19
15.92
SVA+ [ 18 ] frame-distill
5.77
49.22
48.98
18.30
Table 2: Comparison with streaming SA systems under the VPC24 protocol.
System
WER ↓
UAR ↑
EER-L ↑
EER-S ↑
Original
1.82
70.07
4.75
–
B2
9.36
54.59
6.89
3.98
B3
4.21
38.09
22.55
15.28
B4
6.03
42.38
17.19
13.21
B5
4.68
38.46
28.30
18.55
LiveAnon
4.30
44.09
26.08
17.99
Table 3: Comparison with VPC26 challenge baselines under the VPC26 evaluation protocol.
Chunk
Align.
Codec
Total
RTF ↓
160 ms
120 ms
60 ms
340 ms
0.34
320 ms
120 ms
60 ms
500 ms
0.17
Table 4: Algorithmic latency and real-time factor (RTF) of LiveAnon.
Figure 2: Privacy–utility–latency trade-off under different capped-delay settings. Delay cap 3 provides the best operating point.
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), A*STAR, Singapore · The Hong Kong Polytechnic University, Hong Kong