Disentangling Speaker Traits for Deepfake Source Verification via Chebyshev Polynomial and Riemannian Metric Learning
Authors: Xi Xuan, Wenxin Zhang, Zhiyu Li, Jennifer Williams, Ville Hautamäki, Tomi H. Kinnunen
Organizations: University of Eastern Finland, Finland · University of Illinois Urbana-Champaign, USA · University of Chinese Academy of Sciences, China · University of Science and Technology of China, China · University of Southampton, UK
Speech deepfake source verification systems aims to determine whether two synthetic speech utterances originate from the same source generator, often assuming that the resulting source embeddings are independent of speaker traits. However, this assumption remains unverified. In this paper, we first investigate the impact of speaker factors on source verification. We propose a speaker-disentangled metric learning (SDML) framework incorporating two novel loss functions. The first leverages Chebyshev polynomial to mitigate gradient instability during disentanglement optimization. The second projects source and speaker embeddings into hyperbolic space, leveraging Riemannian metric distances to reduce speaker information and learn more discriminative source features. Experimental results on MLAAD benchmark, evaluated under four newly proposed protocols designed for source-speaker disentanglement scenarios, demonstrate the effectiveness of SDML framework. The code, evaluation protocols and demo website are available at https://github.com/xxuan-acoustics/RiemannSD-Net.
Figures & tables
Embedding Extractor
Task 1
Task 2
EER (%)
AUC
EER (%)
AUC
Speaker
1.85
0.9954
29.42
0.7845
Source
15.34
0.9123
5.18
0.9737
Table 1: Pilot experiment on the MLAAD dataset. High cross-task performance between Task1 (speaker verification) and Task2 (source verification) reveals strong entanglement of speaker traits and source, motivating speaker disentanglement.
Encoder
Loss Function
Seen Source
Unseen Source
Average
Same Spk (P-I)
Diff Spk (P-II)
Same Spk (P-III)
Diff Spk (P-IV)
(All Sets)
EER(%) ↓
AUC ↑
EER(%) ↓
AUC ↑
EER(%) ↓
AUC ↑
EER(%) ↓
AUC ↑
EER(%) ↓
AUC ↑
ECAPA-TDNN
Baseline
0.94 ( ± 0.05)
0.995 ( ± 0.002)
1.66 ( ± 0.08)
0.994 ( ± 0.003)
6.60 ( ± 0.21)
0.965 ( ± 0.008)
11.56 ( ± 0.35)
0.941 ( ± 0.012)
5.19 ( ± 0.17)
0.974 ( ± 0.006)
ChebySD-AAM
0.74 ( ± 0.04)
0.997 ( ± 0.001)
1.22 ( ± 0.06)
0.994 ( ± 0.002)
5.72 ( ± 0.18)
0.972 ( ± 0.006)
9.12 ( ± 0.28)
0.946 ( ± 0.010)
4.20 ( ± 0.14)
0.977 ( ± 0.005)
RiemannSD-AAM
0.73 ( ± 0.03)
0.998 ( ± 0.001)
1.20 ( ± 0.05)
0.995 ( ± 0.002)
5.53 ( ± 0.16)
0.980 ( ± 0.005)
9.06 ( ± 0.27)
0.952 ( ± 0.008)
4.13 ( ± 0.13)
0.981 ( ± 0.004)
ResNet34
Baseline
0.73 ( ± 0.04)
0.997 ( ± 0.002)
1.38 ( ± 0.07)
0.994 ( ± 0.003)
7.24 ( ± 0.22)
0.971 ( ± 0.007)
9.77 ( ± 0.29)
0.962 ( ± 0.009)
4.78 ( ± 0.15)
0.981 ( ± 0.005)
Table 2: Performance comparison of the AAM-Softmax (baseline) with the proposed ChebySD-AAM and RiemannSD-AAM on four proposed evaluation protocols. The best results are in bold , and the second-best are underlined . Confidence intervals are in parentheses.
Figure 1: Overview of the proposed SDML framework.
Eval.
Source
Speaker
# Total
# Positive
# Negative
Protocol
Pair
Pair
Utterances
Pairs
Pairs
P-I
Seen-Seen
Same
27,530
13,765
13,765
P-II
Seen-Seen
Different
27,530
13,765
13,765
P-III
Unseen-Unseen
Same
27,530
13,765
13,765
P-IV
Unseen-Unseen
Different
27,530
13,765
13,765
Table 3: Statistics for the different evaluation protocols. All protocols are balanced to ensure a 1:1 positive-negative ratio.
Figure 2: t-SNE visualization of embeddings learned using RiemannSD-AAM.
Unseen Source
Same Spk (P-III)
Diff Spk (P-IV)
Method
EER(%) ↓
AUC ↑
EER(%) ↓
AUC ↑
Baseline [ 27 ] ( × )
7.24 ( ± 0.22)
0.971 ( ± 0.007)
9.77 ( ± 0.29)
0.962 ( ± 0.009)
ChebyAAM [ 15 ] ( × )
6.10 ( ± 0.20)
0.973 ( ± 0.006)
8.87 ( ± 0.27)
0.965 ( ± 0.008)
ChebySD-AAM ( ✓ )
5.85 ( ± 0.19)
0.974 ( ± 0.006)
8.24 ( ± 0.25)
0.969 ( ± 0.007)
HAM-Softmax [ 17 ] ( × )
4.53 ( ± 0.15)
0.985 ( ± 0.004)
7.48 ( ± 0.22)
0.970 ( ± 0.007)
Table 4: Ablation and Parameter Sensitivity with ResNet34 source encoder on unseen source protocols. (✓) / ( × ) denotes whether speaker disentanglement (SD) is ablated. Best in bold , second best underlined . ChebySD-AAM ( K=10 , λ=1 ); RiemannSD-AAM ( c=6 , λ=1 ).