cs.SDSep 29, 2026
SaveLearning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection
Organizations: The Hong Kong Polytechnic University, Hong Kong SAR, China · Communication University of China, Beijing, China · Beijing Institute of Technology, Beijing, China
Abstract
Continual audio deepfake detection requires learning newly emerging deepfake methods while retaining discrimination of previously encountered speech. Existing dataset-incremental evaluation changes both real-speech domains and deepfake mechanisms, making their effects difficult to distinguish. We construct five task organizations over identical training, development, and evaluation pools to study these factors under a controlled sample budget. Our proposed Real-Anchored Mechanism-Incremental (RAMI) protocol reflects the practical setting in which available real speech provides a recurring mixed-domain reference while new deepfake mechanisms arrive incrementally. We further propose RF-Prompt, an asymmetric continual prompt-learning method that preserves reusable real-speech knowledge through a shared real prompt and expands mechanism-specific knowledge through inherited fake experts with orthogonal residuals. Input-adaptive soft fusion combines the accumulated experts into a fixed number of injected tokens without requiring task identity at inference. On RAMI, RF-Prompt achieves 10.110% average EER and 10.370% pooled EER, outperforming all evaluated continual-learning baselines. Across the five controlled protocols, RAMI yields the lowest common-average and pooled EER. Component ablations, limited-data experiments, and cross-backbone evaluations further validate the proposed design.
Figures & tables
Figure 1: Overview of mechanism-incremental audio deepfake detection. RF-Prompt preserves shared real-speech knowledge while expanding fake experts for emerging deepfake mechanisms. Counts below M1–M4 indicate the numbers of generators in the training, development, and evaluation splits.
Figure 2: Five task organizations from identical train, development, and evaluation pools. Columns vary real arrival, while rows vary fake organization. RAMI (Protocol 5) is the proposed protocol; recurring real domains use disjoint utterances across tasks.
Figure 3: Overview of our proposed RF-Prompt. Left: the overall audio deepfake detection pipeline. Middle: continual learning of the shared real prompt and task-specific fake experts. Right: fake expert selection, residual learning, expert construction, and soft fusion at Task 4 as an example.
Figure 4: Acquired-task EER trajectories of Sequential and RF-Prompt on RAMI (Protocol 5). Rows correspond to the checkpoint after each task and columns to acquired deepfake mechanisms; lower EER is better. Purple boxes mark the final checkpoint. Complete trajectories for all continual methods appear in Appendix C .
| Method | Category | Venue | M1 | M2 | M3 | M4 | Avg EER | Pool EER | AF |
|---|---|---|---|---|---|---|---|---|---|
| Sequential | Traditional CL | – | 14.50 | 9.12 | 18.78 | 10.12 | 13.130 | 13.445 | 6.433 |
| EWC ( Kirkpatrick and others, 2017 ) | General CL | PNAS’17 | 13.40 | 8.30 | 17.88 | 9.44 | 12.255 | 12.805 | 5.827 |
| OWM ( Zeng et al., 2019 ) | General CL | Nat. Mach. Intell.’19 | 13.80 | 8.68 | 17.64 | 9.90 | 12.505 | 13.080 | 5.993 |
| RAWM ( Zhang et al., 2023 ) | ADD-specific CL | ICML’23 | 10.22 | 8.02 | 14.96 | 11.98 | 11.295 | 11.730 | 5.753 |
| RWM ( Zhang et al., 2024c ) | ADD-specific CL | AAAI’24 | 13.40 | 8.32 | 17.46 | 9.88 | 12.265 | 12.845 | 5.700 |
| RegO ( Chen and others, 2025 ) | ADD-specific CL | AAAI’25 | 14.46 | 8.22 | 17.72 | 9.64 | 12.510 | 13.000 | 6.060 |
Table 1: Final results on RAMI (Protocol 5) with XLS-R 300M. EER and AF are reported in %; lower is better. Category and venue identify each method’s original scope and publication. All scores are from our matched experiments; marks methods adapted to continual ADD. Offline co-training is excluded when identifying the best continual method.
| Protocol | real arrival | fake organization | Common Avg EER | Pool EER |
|---|---|---|---|---|
| 1 | Dataset-wise | Dataset-wise | 16.400 | 16.430 |
| 2 | Dataset-wise | Mechanism-wise | 15.675 | 15.900 |
| 3 | Source-support-matched | Mechanism-wise | 13.230 | 13.110 |
| 4 | Four-domain mixture | Dataset-wise | 11.035 | 11.330 |
| 5 (RAMI) | Four-domain mixture | Mechanism-wise | 10.110 | 10.370 |
Table 2: Controlled protocol comparison for our RF-Prompt model. The common average uses identical RAMI (Protocol 5) evaluation groups.
| Configuration | Avg EER | Pool EER | AF |
|---|---|---|---|
| RF-Prompt (full) | 10.110 | 10.370 | 4.560 |
| w/o real cosine-anchoring loss | 10.755 | 11.365 | 5.293 |
| w/o residual-orthogonality loss | 12.255 | 12.805 | 7.320 |
| w/o adaptive fusion (uniform mean) | 11.525 | 12.480 | 5.480 |
| w/o orthogonal initialization projection | 11.085 | 11.430 | 5.867 |
Table 3: Ablations on RAMI. Each control retains the other selected settings; the initialization control removes only historical-space projection.
Figure 5: Protection-depth sweeps (left) and limited-fake-data adaptation (right). The other constraint remains active across 24 layers; the right axis counts unique fake clips per task.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Protocol | Task 1 | Task 2 | Task 3 | Task 4 |
|---|---|---|---|---|
| 1 | 4,900/4,505/9,832 | 3,200/3,535/8,525 | 4,800/4,800/10,000 | 6,300/6,360/11,643 |
| 2 | 4,800/4,800/10,000 | 4,800/4,800/10,000 | 4,800/4,800/10,000 | 4,800/4,800/10,000 |
| 3 | 4,704/5,526/10,600 | 5,819/5,019/10,380 | 4,800/4,800/10,000 | 3,877/3,855/9,020 |
| 4 | 4,900/4,505/9,832 | 3,200/3,535/8,525 | 4,800/4,800/10,000 | 6,300/6,360/11,643 |
| 5 (RAMI) | 4,800/4,800/10,000 | 4,800/4,800/10,000 | 4,800/4,800/10,000 | 4,800/4,800/10,000 |
Table 4: Total utterances per task. Each cell lists Train/Dev/Eval counts.
| Mechanism | Train | Dev | Eval |
|---|---|---|---|
| M1 | ASV2019: 2,400 | ASV2019: 2,000; ASV5: 400 | ASV2019: 3,890; ASV5: 1,110 |
| M2 | ASV2019: 100; ASV5: 800; AT-ADD: 1,500 | ASV2019: 105; ASV5: 735; AT-ADD: 1,560 | ASV2019: 942; ASV5: 2,030; AT-ADD: 2,028 |
| M3 | CodecFake: 2,400 | CodecFake: 2,400 | CodecFake: 5,000 |
| M4 | AT-ADD: 2,400 | AT-ADD: 2,400 | ASV5: 385; AT-ADD: 4,615 |
Table 5: Source-dataset composition of the four mechanism-defined fake tasks under RAMI.
| Source | Mechanism | Generator IDs in assignment catalog |
|---|---|---|
| ASV2019 | M1 | A02, A03, A04, A05, A06, A11, A13, A14, A16, A17, A18, A19 |
| ASV2019 | M2 | A01, A07, A08, A09, A10, A12, A15 |
| ASV5 | M1 | A12, A19, A20 |
| ASV5 | M2 | A01, A02, A03, A04, A05, A06, A07, A08, A09, A10, A11, A13, A14, A15, A16, A17, A18, A21, A22, A23, A24, A25, A26, A27, A28, A30, A31, A32 |
| ASV5 | M4 | A29 |
| AT-ADD | M2 | BigVGAN, DiffGANTTS, DiffSpeech_fastdiff, E2TTS, F5TTS, FastDiff, FastPitch_fastdiff, FastSpeech2_fastdiff, GlowTTS, GradTTS, HiFiGAN, Kokoro, MBMelGAN, MelGAN, MeloTTS, OpenVoice2, ParallelWaveGAN, PortaSpeech_normal_fastdiff, SeedVC1, StarGANv2VC, StyleMelGAN, StyleSpeech, StyleTTS2, Tacotron2, VITS, WaveGlow, WaveNet, prodiff_teacher_fastdiff |
Table 6: Generation-mechanism assignment by source dataset.
| Setting | Value |
|---|---|
| Sample rate / input length | 16 kHz / 64,600 samples |
| Tasks / epochs per task | 4 / 50 |
| Batch size / random seed | 32 / 2026 |
| Optimizer | Adam |
| Adam betas / epsilon | (0.9, 0.999) / 1e-8 |
| Weight decay | 5e-4 |
Table 7: Selected XLS-R 300M configuration.
| Method | After Task 1 | After Task 2 | After Task 3 | After Task 4 | Final Avg | Final Pool |
|---|---|---|---|---|---|---|
| Sequential | 4.66 | 8.00/9.38 | 12.84/12.22/9.32 | 14.50/9.12/18.78/10.12 | 13.130 | 13.445 |
| EWC | 4.66 | 8.28/9.68 | 13.58/12.34/9.14 | 13.40/8.30/17.88/9.44 | 12.255 | 12.805 |
| OWM | 4.66 | 9.34/9.80 | 14.08/12.50/8.80 | 13.80/8.68/17.64/9.90 | 12.505 | 13.080 |
| RAWM | 4.66 | 6.18/5.10 | 8.54/6.62/6.18 | 10.22/8.02/14.96/11.98 | 11.295 | 11.730 |
| RWM | 4.66 | 9.24/10.08 | 13.80/12.38/9.10 | 13.40/8.32/17.46/9.88 | 12.265 | 12.845 |
| RegO | 4.66 | 8.82/10.16 | 14.92/12.88/9.34 | 14.46/8.22/17.72/9.64 | 12.510 | 13.000 |
Table 8: Acquired-task EER trajectories on RAMI (Protocol 5). The entry under “After Task ” lists EERs for in order; the last two columns report final average and pooled EER. All entries are percentages.
Figure 6: RF-Prompt acquired-task EER trajectories on Protocols 1–4. Rows denote the checkpoint after each task, columns denote the task-specific evaluation groups, and purple boxes mark the final checkpoint. All values are EER (%); lower is better.
Figure 7: Comparison of real prompt protection and fake expert construction. Shared Prompt Distillation (SPD) aligns audio-token features produced with current and previous real prompts; Prompt cosine directly anchors their parameters. Full-Prompt orthogonality separates complete fake experts, whereas residual orthogonality separates only the newly learned residual while retaining the inherited base.
| real protection | fake construction | Avg | Pool | AF |
|---|---|---|---|---|
| None | None | 12.600 | 12.020 | 5.393 |
| None | Inherited residual | 10.755 | 11.365 | 5.293 |
| Shared Prompt Distillation (SPD) | Inherited residual | 10.910 | 11.180 | 5.467 |
| Prompt cosine | Full-Prompt orthogonality | 11.795 | 12.355 | 6.140 |
| Prompt cosine | Inherited residual | 10.110 | 10.370 | 4.560 |
Table 9: Comparison of real prompt protection and fake expert construction on RAMI. All entries are EER or AF (%).
Figure 8: Geometry of the learned fake experts on RAMI. Panels (a)–(b) show a joint PCA projection of complete-expert and residual tokens; translucent points are layer-wise prompt tokens and stars are mechanism centroids. Panels (c)–(d) report normalized layer-wise subspace overlap, where lower off-diagonal values indicate stronger separation.
| Unique fake/task | Method | Avg EER | Pool EER | AF |
|---|---|---|---|---|
| 100 | Sequential | 17.325 | 17.750 | 6.887 |
| EWC | 17.705 | 18.080 | 6.707 | |
| RWM | 17.075 | 17.645 | 6.067 | |
| RegO | 17.765 | 18.210 | 6.040 | |
| RF-Prompt | 12.590 | 12.375 | 3.907 | |
| 500 | Sequential | 15.220 | 15.715 | 7.367 |
Table 10: Limited-data results on RAMI. All metrics are percentages; lower is better.
Figure 9: Sensitivity to the number of real and fused fake prompt tokens per transformer layer on RAMI. The selected configuration uses five tokens for each prompt type, for ten injected tokens in total.
Figure 10: Cross-backbone comparison of Sequential and RF-Prompt. Each panel reports one final continual-learning metric after four tasks; lower is better. Values above the bars are percentages.