Speech anonymization is evaluated with automatic speaker verification attackers whose performance depends on the attribute distribution of both their training and evaluation data, which may bias privacy estimates for some speaker groups. To alleviate this issue, we introduce VoxAnon, a dataset built from VoxCeleb1 and 2 with training and evaluation splits balanced across gender, age, accent, and pitch. We extract and cross-check attributes using two complementary methods, balance trial lists per subgroup, and draw test/enrollment utterances from distinct video sources in order to limit linguistic and session bias. Comparing VoxAnon with unbalanced training and test data across four attackers and three anonymizers, we show that attribute imbalance changes which subgroups appear hardest to re-identify, in directions that depend on the attribute. In the balanced setting, residual disparities persist, mostly tied to accent and pitch, and depend on the anonymizer/attacker pair. Code and trial lists are publicly available.
Figures & tables
Training set
Evaluation set
Attr.
Group
Speak.
Utt.
Dur. (h)
Speak.
Trials
Test
Enrol.
T
NT
Total
All
1,000
137,121
291.48
50
100
500
49,500
50,000
Gender
Male
500
69,538
147.23
25
50
250
24,750
25,000
Female
500
67,583
144.26
25
50
250
24,750
25,000
Age
18–24
250
26,587
54.63
12
25
120
11,880
12,000
Table 1: Number of speakers, utterances and duration in the training set, and number of test and enrollment speakers, and target (T), non-target (NT), and total trials for the evaluation set.
Figure 1: Trial difficulty across subgroups in the balanced test set.
Overall
Gender
Age
Accent
Pitch
Attacker
Cond.
EER
Male
Female
Std
18–24
25–34
35–59
60+
Std
US
English
Indian
Austral.
Std
Low
Mid
High
Std
RedimNet
orig
1.40
1.57
1.13
0.22
1.62
0.84
1.72
1.51
0.35
1.45
1.64
0.77
1.97
0.44
1.84
1.62
0.84
0.43
B3
28.17
27.22
28.80
0.79
28.24
28.01
27.82
28.46
0.24
24.45
31.47
27.55
25.55
2.67
28.26
27.76
27.60
0.28
B4
22.60
23.39
22.01
0.69
22.34
23.03
23.95
21.63
0.86
14.55
26.63
21.01
26.81
5.02
26.98
20.48
19.91
3.21
B5
29.91
32.38
28.37
2.01
29.36
28.43
31.53
31.48
1.35
25.70
32.22
26.76
34.22
3.58
32.83
31.02
26.19
2.80
ResNet
orig
1.60
1.36
1.95
0.29
1.63
0.85
1.53
2.51
0.59
2.27
2.27
0.85
0.73
0.74
1.81
1.22
1.91
0.30
Table 2: Results for systems trained and tested on VoxAnon-balanced. EER (%) by attacker, anonymization condition, and speaker-attribute subgroup, and intra-attribute EER standard deviation (Std.).
Figure 2: Balanced and unbalanced sets comparisons: percentage of attack scenarios for which the undersampled subgroup for each category obtained the highest and lowest EER, corresponding to the hardest and easiest subgroup to re-identify, respectively.
Voice anonymization masks vocal traits while preserving linguistic content, which may still leak speaker-specific patterns. To assess and strengthen privacy evaluation, we propose a dual-stream attacker that fuses spectral and self-supervised learning features via parallel encoders with a three-stage training strategy. Stage I establishes foundational speaker-discriminative representations. Stage II leverages the shared identity-transformation characteristics of voice conversion and anonymization, exposing the model to diverse converted speech to build cross-system robustness. Stage III provides lightweight adaptation to target anonymized data. Results on the VoicePrivacy Attacker Challenge (VPAC) dataset demonstrate that Stage II is the primary driver of generalization, enabling strong attacking performance on unseen anonymization datasets. With Stage III, fine-tuning on only 10% of the target anonymization dataset surpasses current state-of-the-art attackers in terms of EER.
Ridwan Arefeen, Xiaoxiao Miao, Rong Tong +3
Singapore Institute of Technology, Singapore · Duke Kunshan University, China · NVIDIA AI Technology Centre, Singapore
Speech anonymization has focused on the acoustic signal, leaving open how much speaker information the transcribed words themselves retain. In this work, we study pairwise speaker and demographic-attribute verification from transcribed speech. We introduce SPARC (Spoken Profile Authorship Recognition Corpus), a benchmark built from two podcast corpora comprising over 12,500 speakers, 1.5M quality-filtered utterances, and speaker-level inferred or source-provided labels for sex, age, and accent, and evaluate five attribution models and three LLMs on identity- and attribute-verification trials with episode- and metadata-based topic proxies. LLM adversaries distinguish same-speaker pairs from text alone. Attribute-label agreement is harder to verify, but LLMs perform sex- and accent-label verification above chance. Finally, releasing only short probe texts (50 words) degrades the strongest adversaries the most, offering a partial reduction in observed linkage.
Efthymios Tsaprazlis, Bowen Yi, Tiantian Feng +3
University of Southern California · University of Michigan
Speech anonymization is commonly evaluated using averagecase metrics such as the equal error rate, which can hide large disparities in re-identification risks across individuals. In this paper, we conduct a large-scale per-speaker privacy analysis using a linkability-based metric under a worst-case scenario. Nearly 5,000 speakers are evaluated across multiple anonymization systems, attacker architectures, and conversation lengths. While linkability scores are highly polarized at the speaker level, the sets of easy to re-identify and hard to re-identify speakers vary substantially across configurations. We show that no single factor explains speaker vulnerability. Instead, the re-identification risk emerges from the interaction between the attacker, the anonymizer, and the amount of available speech. These results challenge the notion of intrinsic speaker-level privacy risks and emphasize the need for evaluation protocols that are explicitly conditioned on the attacker and anonymizer.
Orane Dufour, Paul Magron, Mickael Rouvier +1
Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France · LIA, Avignon University, F-84911 Avignon, France