Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Advances in synthesis techniques have made deepfake speech increasingly realistic, amplifying voice security threats, including unauthorized voice cloning and impersonation. Although existing deepfake detectors perform well on digitally generated speech, their performance often degrades on physically replayed deepfakes, which constitute a low-cost and practical threat to real-world systems. Existing research focuses primarily on dataset construction and evaluation, while available dataset diversity and replay-aware detector design remain limited. We propose ReplayFactor, a dual-attribute framework that enhances replay evidence by fusing self-supervised learning (SSL) representations with an acoustic prior, while leveraging multi-task learning and supervised contrastive learning to promote robust, generalizable representations. We also construct ReplayFake-CN, a large-scale, multi-device Chinese dataset comprising approximately 185k replayed synthetic utterances to complement public benchmarks. Our framework improves macro-F1 from 60.99% to 76.44% and reduces EER from 11.86% to 3.05% on the EchoFake open set over the baseline, demonstrating strong potential for practical replayed deepfake detection.
Figures & tables
Figure 1: Overview of the proposed framework, comprising (I) Representation extraction with SSL and acoustic-prior fusion; and (II) multi-task and supervised contrastive learning.
6 consumer and 3 high-fidelity loudspeakers, e.g., JBL, HP-speaker, Edifier-MR4, etc.
Table 1: Statistics of ReplayFake-CN.
Model
Test
EER ↓
F1-score ↑
macro.
B
RB
F
RF
XLSR-Linear † Reported in [ 21 ]
Closed
0.27
98.81
99.63
98.16
99.43
98.02
Open
11.86
60.99
78.83
40.13
75.64
49.34
ReplayDF
–
–
–
7.72
–
19.02
XLSR-AASIST
Closed
0.47
98.33
98.68
97.97
98.66
98.02
Open
8.17
56.99
86.69
19.96
71.91
49.40
Table 2: Results (%) on EchoFake and ReplayDF. Closed and Open denote the EchoFake evaluation sets. ReplayDF reports RB and RF F1 scores from EchoFake-trained models.
Method
Overall ACC
Unseen Synthesis
Unseen Devices
Unseen Synth.–Devices
Binary public checkpoint: AntiDeepfake [ 7 ]
MMS-300M
55.43
64.04
46.63
62.98
MMS-AASIST
84.22
93.97
82.59
73.30
Ours
85.44
96.95
83.00
73.75
Table 3: Accuracy (%) on ReplayFake-CN under unseen conditions. Binary and four-class models are reported separately.
Configuration
EER ↓
F1-score ↑
macro-F1
B
RB
F
RF
4-class baseline
7.26
62.27
90.14
25.93
77.99
55.02
Ours (full)
3.05
76.44
93.91
61.12
82.01
68.72
w/o acoustic prior
8.11
73.54
89.07
61.44
84.28
59.37
w/o SupCon
2.70
73.89
93.82
50.77
82.85
68.14
w/o 4-class head
4.31
69.22
94.84
32.97
84.03
65.03
Table 4: Ablation results on the EchoFake open set (%).
Figure 2: t-SNE visualizations on the EchoFake closed set: (a) joint, (b) source, and (c) replay representations.
Figure 3: Spectrograms of the original non-replay utterance (top) and its replays through JBL (middle) and Xiaoai (bottom) loudspeakers, both recorded with a Huawei Mate30Pro phone.