Enhancing Efficiency and Performance in Deepfake Audio Detection through Neuron-level Dropin & Neuroplasticity Mechanisms
Authors: Yupei Li, Shuaijie Shao, Manuel Milling, Björn Schuller
Organizations: Department of Computer Science, University College London, London, United Kingdom · Chair of Health Informatics, Technical University of Munich, Munich, Germany · Chair of Health Informatics and Department of Computing, Technical University of Munich and Imperial College London, Munich and London, Germany and United Kingdom
Current audio deepfake detection has achieved remarkable performance using diverse deep learning architectures such as ResNet, and has seen further improvements with the introduction of large models (LMs) like Wav2Vec. The success of large language models (LLMs) further demonstrates the benefits of scaling model parameters, but also highlights one bottleneck where performance gains are constrained by parameter counts. Simply stacking additional layers, as done in current LLMs, is computationally expensive and requires full retraining. Furthermore, existing low-rank adaptation methods are primarily applied to attention-based architectures, which limits their scope. Inspired by the neuronal plasticity observed in mammalian brains, we propose novel algorithms, dropin and further plasticity, that dynamically adjust the number of neurons in certain layers to flexibly modulate model parameters. We evaluate these algorithms on multiple architectures, including ResNet, Gated Recurrent Neural Networks, and Wav2Vec. Experimental results using the widely recognised ASVSpoof2019 LA, PA, and FakeorReal dataset demonstrate consistent improvements in computational efficiency with the dropin approach and a maximum of around 39% and 66% relative reduction in Equal Error Rate with the dropin and plasticity approach among these dataset, respectively. The code and supplementary material are available at Github link.
Figures & tables
Fig. 1: High-level dropin process: During the dropin phase, we load the original pretrained model weights and keep them frozen. Additional neurons are then introduced into randomly selected layers, and only the connection weights associated with these newly added neurons are trained.
Fig. 2: Convolutional layers, recurrent units, and attention encoders dropin process. These represent the specific dropin techniques adapted to different model architectures. From a mathematical perspective, this involves increasing the kernel size for CNNs, expanding the gate weight dimensions for GRUs, and enlarging the query, key, and value weight dimensions for attention mechanisms.
Fig. 3: Plasticity process: The pipeline is divided into three stages. The first stage is identical to conventional training, where the model is optimised using the standard objective function. In the second stage, new neurons are dropped in, a process analogous to neurogenesis. The whole model is then retrained. After the newly introduced information has been assimilated and distributed across existing neurons. The third stage emulates neuroapoptosis by pruning the added neurons, followed by a final retraining.
θinit,θinit∼N(0,σ2)
Algorithm 1 Neuroplasticity Training
Dataset
Model
Training Strategy
Test EER (%) ↓
Backward Time per step (ms) ↓
Parameters (M) ↓
Trainable Params (M) ↓
ASV LA
ResNet
baseline
14.69
5.46
11.17
11.17
dropin unfrozen
7.60
6.72
14.28
14.28
dropin frozen (ours)
11.98
3.49
14.28
1.47
plasticity (ours)
9.70
/
11.17
/
GRNN
baseline
19.84
343.35
0.06
0.06
dropin unfrozen
16.64
477.79
0.10
0.10
TABLE I: Comparison of different training strategies across ASV LA, PA, and FoR datasets. Dropin unfrozen denotes an enlarged network trained conventionally, while baseline refers to the original model. For plasticity, backward time and trainable parameters are not reported as they vary across stages. The symbol “/” indicates unavailable or non-applicable measurements.
Fig. 4: GradCAM results on Resnet for one case from ASVSpoof 2019 LA.
Fig. 5: EER results for different dropin layers. “3” represents that we dropped in new neurons on the 3rd encoding layer in Wav2vec 2.0 small on ASVSpoof2019 LA.
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
Jianqiao Cui, Haoxian Zhu, Hanbiao Meng +1
Imperial College London, London, UK · Taiyuan University of Technology, Taiyuan, China
Existing audio deepfake detection (ADD) datasets and detectors are primarily built for vocoder-based synthesis, evaluated against traditional post-hoc perturbations such as MP3/AAC compression or additive noise, applied independently of generation. However, recent speech synthesizers, particularly ALM-based systems, use neural audio codecs both for compression and as the resynthesis reconstructing waveforms from generated tokens, producing artifacts distinct from post-hoc compression. Neural codecs thus play a dual role: some are designed for pure compression under low-bandwidth communication, while others serve as resynthesis components. Despite this dual role, robustness to codec-based compression, unlike post-hoc compression, remains largely unexplored. We expose this gap, showing that state-of-the-art (SOTA) ADD models degrade drastically on codec-compressed speech; in particular, systems trained on Codec Resynthesized data as a proxy for codec-based generation prove most vulnerable, with legitimately compressed bona fide speech often misclassified as fake. To investigate this, we construct the Audio Neural Codec-Spoof dataset by applying seven neural codec algorithms to existing ADD benchmarks, isolating codec-induced resynthesis artifacts as a controlled proxy for codec-based generation. As baseline mitigation, we propose PCL-NET (Pairwise Consistency Learned Network), fine-tuning a pretrained XLS-R (300M) encoder with a pairwise consistency objective that minimizes the representation distance between an utterance's uncompressed and codec-compressed versions, disentangling codec artifacts from the real-versus-fake decision. As a result, PCL-NET reduces average EER under neural codec compression from 28.67% to 12.77%, while preserving competitive CoSG-based deepfake detection performance. We will also make the dataset publicly available on Hugging Face upon acceptance.
Abdullah, Awais Khan, Khalid Mahmood Malik
College of Innovation and Technology University of Michigan-Flint Michigan, USA
Advances in speech synthesis have made deepfake speeches increasingly convincing, posing growing threats to security. While self-supervised learning (SSL) based detectors achieve state-of-the-art performance, their computational demands (typically 300M+ parameters) prevent deployment on resource-constrained devices. Existing compression methods, designed mainly for content-centric tasks, struggle to maintain competitive performance when directly adapted to deepfake detection. We propose a Task-Aware Joint Pruning and Distillation framework that combines cross-domain knowledge distillation with movement-guided structured pruning to transfer forgery-discriminative knowledge and preserve critical structures under aggressive compression. Our framework reduces the model to 31.9M parameters with 6.3× FLOPs reduction, with an average performance drop of only 1.30% across multiple datasets compared to the uncompressed baseline, demonstrating strong potential for on-device deployment.
Miao He, Peng Cheng, Zhongjie Ba +4
The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, China · Qingdao Institute of Software College of Computer Science and Technology, China University of Petroleum (East China), China +1