What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection

Jiajun Xu; Menglu Li; Xiao-Ping Zhang

cs.SDSep 27, 2026

What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection

Authors: Jiajun Xu, Menglu Li, Xiao-Ping Zhang

Organizations: Department of Engineering Physics Tsinghua University Beijing, China · Tsinghua Shenzhen International Graduate School Tsinghua University Shenzhen, China

Abstract

The transition from vocoder-based to neural-codec speech synthesis makes generalization more difficult for speech deepfake detectors, particularly those relying on speech-oriented representations. It remains unclear which acoustic representations retain discriminative information when the generation mechanism changes. We therefore compare 12 acoustic representations using a shared low-capacity linear classifier to identify effective evidence under codec shift. The analysis shows that hierarchical XLS-R leads on the pooled test set, while pooled no-vocals residual statistics perform best on the unseen-codec condition, revealing complementary behavior across generation conditions. Building on this finding, we propose MN-P, a dual-view detector that integrates an utterance-level pooled no-vocals representation with token-level XLS-R features through adaptive gating. The proposed MN-P reduces EER by 54.2% overall and by 60.9% on the codec-unseen condition relative to the best-performing retrained state-of-the-art system, with consistent gains across different detector backends. These results indicate that pooled no-vocals residual statistics provide effective complementary evidence for cross-generation speech deepfake detection.

Figures & tables

Figure 1: Speech-centric single-view detection versus the proposed MN-P. (a) A conventional detector relies on a single speech representation. (b) The proposed MN-P detector complements token-level speech features with an utterance-level pooled no-vocals residual obtained after vocal separation, and integrates the two through adaptive gating.

Candidates	EER(%) $\downarrow$	CU EER(%) $\downarrow$	Source F1 $\downarrow$
B6	4.843	$\underline{9.291}$	0.726
B2-XV	$\underline{13.448}$	21.935	0.656
B2-NV	13.859	8.619	0.539
B2-V	25.999	41.201	$\underline{0.588}$
B3	26.766	34.606	0.628
B1	26.956	34.204	0.620

Table 1: Six candidate representations with the lowest overall test-set EER. Source F1 is the closed-set macro-F1 of a linear probe for fake-source identity, where lower values indicate less linearly decodable source information. Bold and underlined denote the best and second-best results.

Figure 2: Architecture of the proposed MN-P detector. A frozen XLS-R branch produces a token-level speech representation, while the no-vocals branch, highlighted in the orange box, aggregates residual statistics over the whole utterance into a single vector. CrossViewGate fuses the two views before the AASIST backend. Frozen and trainable modules are marked by distinct icons.

System	AUC $\uparrow$	EER (%) $\downarrow$	Fake miss (%) $\downarrow$	CU EER (%) $\downarrow$	CU fake miss (%) $\downarrow$
MN-P-AASIST (Ours)	99.638	2.744	6.890	6.136	27.903
MN-P-Nes2Net	$\underline{99.583}$	$\underline{2.927}$	$\underline{9.068}$	$\underline{6.706}$	$\underline{40.182}$
M0-AASIST	98.118	7.010	14.702	18.639	68.729
M0-Nes2Net	98.630	5.869	14.499	15.632	68.092
FAD [ 13 ]	98.507	6.246	14.876	16.562	70.012
W2V2-AASIST with CSAM [ 24 ]	98.647	5.986	15.444	15.692	74.325

Table 2: Performance comparison on the merged test set (%). MN-P is evaluated with AASIST and Nes2Net backends; M0 removes the pooled no-vocals view and CrossViewGate. Selected state-of-the-art systems are retrained under the same protocol. Bold and underlined denote the best and second-best results.

	CU EER (%) $\downarrow$	CU fake miss (%) $\downarrow$
M0	34.830	80.180
MN-S	$\underline{18.168}$	$\underline{57.808}$
MN-P	8.709	14.865

Table 3: Ablation on model structure (%), on an independent train/development/test split. M0 uses XLS-R alone and MN-S feeds the no-vocals residual as a frame-level sequence. Bold and underlined denote the best and second-best results.

Explore similar work

Neural Audio Codec for Robust Audio Deepfake Detection
Sep 30, 2026Jungwoo Kim, Joonyong Park, Junyoung Koh +1Neural Audio Codecs Affectcodec
Yonsei University, Seoul, Republic of Korea · MAAP Lab, Republic of Korea · The University of Tokyo, Tokyo, Japan +1
FlowFake: Liquid Networks for Audio Deepfake Detection
Jun 17, 2026Shivaay Dhondiyal, Divyansh Sharma, Dinesh Kumar VishwakarmaNeural Audio Speaker
Delhi Technological University, New Delhi, India.
Alethia: A Foundational Encoder for Voice Deepfakes
Apr 30, 2026Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram +1Deepfake Detection Opencode
Reality Defender Inc.

Sep 30, 2026, cs.SDJ/K move · Enter open · S save

Neural Audio Codec for Robust Audio Deepfake Detection

Jungwoo Kim, Joonyong Park, Junyoung Koh, Jong-Seok Lee

Yonsei University, Seoul, Republic of Korea · MAAP Lab, Republic of Korea · The University of Tokyo, Tokyo, Japan+1

Neural Audio Codecs Affectcodec Spoofing

Open paper AskPDF arXiv

Audio deepfake detectors are typically evaluated on uncompressed audio, although real-world audio often undergoes low-bitrate coding. In this work, we investigate how audio coding affects deepfake detection across codecs, bitrates, and detectors, finding higher errors at lower rates. A mixed-pair protocol isolates codec-induced changes in bona fide and spoof audio, revealing asymmetric, codec-dependent failures: low-rate DAC and EnCodec mainly degrade bona fide detection, whereas X-Codec shows a stronger spoof-side limitation. Motivated by these, we propose a forensic-preserving neural audio codec (FP-NAC), which fine-tunes a pretrained codec using a detector-guided objective while preserving its native hard quantization path and bitrate. On ASVspoof 2019 LA, FP-NAC reduces EER by up to 49.8~~pp compared with the original DAC at 0.5~~kbps while maintaining comparable reconstruction quality. Although supervised by only one detector, FP-NAC improves performance across multiple detectors, highlighting forensic transparency as a codec design objective alongside perceptual quality. Our codes are available at https://github.com/kjungwoo03/FP-NAC.

Abstract

Explore similar work

Neural Audio Codec for Robust Audio Deepfake Detection

FlowFake: Liquid Networks for Audio Deepfake Detection

Alethia: A Foundational Encoder for Voice Deepfakes