The Affective Bridge: Preserving Speech Representations while Enhancing Deepfake Detection via emotional Constraints

Yupei Li; Chenyang Lyu; Longyue Wang; Weihua Luo; Kaifu Zhang; Björn W. Schuller

cs.SDDec 12, 2025

The Affective Bridge: Preserving Speech Representations while Enhancing Deepfake Detection via emotional Constraints

Authors: Yupei Li, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang, Björn W. Schuller

Abstract

Speech deepfake detection (DFD) has benefited from diverse acoustic and semantic speech representations, many of which encode valuable speech information and are costly to train. Prior work has shown that affective cues improve DFD, yet existing approaches either fuse emotion with other task-specific features in complex pipelines or directly fine-tune representations toward DFD objectives, risking distortion of the original speech representations that support downstream tasks such as speaker verification (SV) or automatic speech recognition (ASR). We propose a simpler approach: fine-tuning speech encoders on emotion recognition alone-without any DFD supervision, and training a lightweight support vector machine (SVM) on the frozen emotion-tuned representations for DFD. This preserves the original representation capacity for downstream tasks such as SV and ASR, while emergently improving DFD performance. Crucially, we find that emotion is uniquely effective as this bridging task: replacing it with speaker identity even degrades DFD performance, demonstrating that the benefit stems from emotion's role as a natural bridge between speech representation and DFD. Experiments on FakeOrReal and In-the-Wild show accuracy improvements of up to 6% and 2% with corresponding EER reductions, while analysis on ASVspoof 2019 LA reveals dataset-specific speaker bias in the real-speech subset. Code is available at GitHub repository.

Figures & tables

Figure 1: Overview of the proposed Affective Bridge framework for speech DFD. In Phase I Emotion-Guided Representation Alignment (EmoBridge), raw speech is transformed into speech representations through two parallel routes: hand-crafted acoustic feature extractors (e.g., OpenSMILE) or encoders pre-trained for other speech tasks such as ASR or speaker verification. For hand-crafted acoustic features, a subset of features containing richer emotion-related cues is retained. For deep representations, an emotion prediction head is attached to the pre-trained encoder and optimized using an emotion recognition objective, guiding the representation to incorporate emotion-related information while preserving its original task-oriented structure. In Phase II deepfake detection, the resulting emotion-aligned representation, either selected acoustic features or outputs from the trained encoder, are frozen and fed into a downstream classifier (e.g., SVM or MLP) to perform DFD, classifying input speech as real or fake.

Metrics	ASR	SV	DL-raw	Emotion	I. Shahin [ 34 ]	S. Hamsa [ 13 ]	Koya S. [ 19 ]
Weighted Acc.	0.728	0.701	0.786	0.732	0.840	0.909	0.917

Table 1: EmoBridge performance. The table reports the weighted accuracy for the four pre-trained models evaluated in this study, with comparison to other models in literature.

Dataset	Model	Pre-trained		EmoBridge		WaveFake FT		In-domain FT
	Model	EER $\downarrow$	Acc $\uparrow$	EER $\downarrow$	Acc $\uparrow$	EER $\downarrow$	Acc $\uparrow$	EER $\downarrow$	Acc $\uparrow$
FoR	openSMILE	.406	.574	.275	.663	–	–	–	–
	Emotion	.139	.854	.078	.913	.110	.875	.211	.632
	SV	.136	.858	.089	.908	.126	.861	.144	.719
	ASR	.082	.887	.045	.946	.091	.855	.218	.702
	DL-raw	.091	.890	.062	.931	.089	.898	.056	.886

Table 2: Comparison of layer-wise averaged EER and accuracy between the pre-trained model, EmoBridge, WaveFake fine-tuned encoders, and in-domain fine-tuned encoders on three datasets.

Source	ASR		SV		DL-raw		Emotion		Static	Concatenation
EmoBridge?	No	Yes	No	Yes	No	Yes	No	Yes	—	—
A07	1.000	0.999	0.999	1.000	0.998	0.999	1.000	0.998	0.948	0.994
A08	0.995	1.000	0.999	0.999	0.949	0.995	0.988	0.999	0.988	0.962
A09	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
A10	0.999	0.996	0.995	0.996	0.842	0.857	0.998	0.977	0.900	0.965
A11	1.000	1.000	1.000	1.000	0.979	0.937	1.000	0.999	0.895	0.997

Table 3: Accuracy comparison across different sources and feature settings before and after the emotion as a bridge strategy. Column Static is from work [ 25 ] (— means not provided or N/A) while Column Emotion is from our strategy using Hubert as original feature. In this table, only the most noteworthy results are emphasised in bold.

Error band	#Spk.	#Samples	#Misclass.	Rate
$\geq$ 50% (severe)	17	1,963	1,285	65.5%
30–50% (moderate)	13	1,556	589	37.9%
10–30% (mild)	35	3,643	725	19.9%
Overall	65	7,162	2,599	36.3%

Table 4: Distribution of per-speaker bona fide error rates on ASVspoof 2019 LA after EmoBridge adaptation (Emotion feature, layer-averaged). Degradation is heavy-tailed: 26% of speakers account for 49% of all misclassifications, while even the least-affected speakers retain a

\sim

10% error floor.

Dataset	Model	Pre-trained		SpeakerBridge
	Model	EER $\downarrow$	Acc $\uparrow$	EER $\downarrow$	Acc $\uparrow$
FoR	Emotion	.139	.854	.148	.857
	SV	.136	.858	.136	.854
	ASR	.082	.887	.135	.884
	DL-raw	.091	.890	.137	.775
ITW	Emotion	.050	.953	.148	.870

Table 5: Comparison of layer-wise averaged EER and accuracy between the Pre-trained model and the model with the speaker as a bridge strategy on three datasets.

Dataset	Model	Pre-trained		EmoBridge
	Model	EER $\downarrow$	Acc $\uparrow$	EER $\downarrow$	Acc $\uparrow$
FoR	openSMILE	.385	.554	.265	.645
	Emotion	.143	.866	.083	.915
	SV	.146	.864	.087	.910
	ASR	.088	.898	.066	.952
	DL-raw	.095	.899	.077	.943

Table 6: EER and accuracy between the Pre-trained model and the model with the EmoBridge strategy on FoR with MLP as classifier.

Task	Model	Training Strategy	Metric $\downarrow$
SV	WavLM-base-SV	Pretrained	5.63%
		EmoBridge fine-tuned	6.78%
		Conventional DFD	35.13%
ASR	SpeechT5-ASR	Pretrained	11.91%
		EmoBridge fine-tuned	16.56%
		Conventional DFD	221.4%

Table 7: Representation preservation evaluated on speaker verification (SV) and automatic speech recognition (ASR). Lower EER and WER indicate better preservation.

Figure 2: t-SNE visualization of ASR (left) and SV (right) representations from selected samples obtained with the original model ( model_ori ), after EmoBridge training ( model_emobridge ), and after conventional DFD fine-tuning (model_wavefake_ft).

Figure 3: Comparison of mean attention values of Whisper layers before (model_ori) and after emotion as a bridge (model_new).

Audio Deepfake Detection

Audio deepfake detection identifies synthetic, cloned, or manipulated speech and other audio to support authentication and media forensics.

See all 204 papers

Time-Frequency Consistency Learning for Robust Speech Deepfake Detection
Jul 20, 2026Jun Xue, Zhuolin Yi, Yanzhen Ren +6Audio Deepfake Detection Frequency-Domain Feature Learning
School of Cyber Science and Engineering, Wuhan University Wuhan, China · Wuhan University Wuhan, China · Tongji University ShanghaiU, China
From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection
Sep 8, 2026Mengzhe Geng, Yujia Lu, Manuela Kunz +1Audio Deepfake Detection Explainable Deepfake Detection
National Research Council Canada · The Chinese University of Hong Kong
Phonetically Explainable Speech Deepfake Detection
Jun 13, 2026Manasi Chhibber, Jagabandhu Mishra, Tomi H. KinnunenAudio Deepfake Detection Explainable Deepfake Detection
School of Computing, University of Eastern Finland, Joensuu, FI-80101, Finland

Jul 20, 2026J/K move · Enter open · S save

Time-Frequency Consistency Learning for Robust Speech Deepfake Detection

Jun Xue, Zhuolin Yi, Yanzhen Ren, Yihuan Huang+5

School of Cyber Science and Engineering, Wuhan University Wuhan, China · Wuhan University Wuhan, China · Tongji University ShanghaiU, China

Audio Deepfake Detection Frequency-Domain Feature Learning Noise Robustness

Open paper AskPDF arXiv

Recently, speech deepfake detection (SDD) has achieved significant progress. However, its robustness evaluation remains largely confined to controlled additive noise scenarios, lacking systematic investigation of the complex distortions introduced by acoustic front-end (AFE) processing pipelines in real-world deployments. In this work, we simulate a unified AFE pipeline comprising acoustic echo cancellation, noise suppression, automatic gain control, and voice activity detection (VAD), and conduct a comprehensive evaluation of current state-of-the-art models. The results show that the nonlinear and time-frequency coupled distortions introduced by AFE significantly degrade detection performance. To address this issue, we propose a Time-Frequency Consistency Learning (TFCL) framework, which aims to learn invariant spoofing representations that remain stable before and after AFE processing. We observe that AFE not only introduces temporal misalignment (e.g., segment-level shifts caused by VAD), but also weakens or distorts critical frequency-domain cues. To this end, TFCL employs an attention-driven soft alignment mechanism to capture cross-temporal dependencies, along with frequency-domain structural consistency constraints to enforce feature invariance. As a result, the model is able to maintain stable representations under both temporal perturbations and spectral distortions. Extensive experimental results demonstrate that the proposed method effectively mitigates the performance degradation caused by AFE processing, significantly improving the robustness of SDD in real-world scenarios. The code is available at https://github.com/JunXue-tech/TFCL.

Abstract

Audio Deepfake Detection

Explore similar work

Time-Frequency Consistency Learning for Robust Speech Deepfake Detection

From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection

Phonetically Explainable Speech Deepfake Detection