Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis

Miao Liu; Fangda Wei; Jing Wang; Xinyuan Qian

cs.CVApr 14, 2026

Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis

Authors: Miao Liu, Fangda Wei, Jing Wang, Xinyuan Qian

Organizations: University of Science and Technology Beijing, Beijing, China

Abstract

Existing deepfake detection research has primarily focused on scenarios where the manipulated subject is actively speaking, i.e., generating fabricated content by altering the speaker's appearance or voice. However, in realistic interaction settings, attackers often alternate between falsifying speaking and listening states to mislead their targets, thereby enhancing the realism and persuasiveness of the scenario. Although the detection of 'listening deepfakes' remains largely unexplored and is hindered by a scarcity of both datasets and methodologies, the relatively limited quality of synthesized listening reactions presents an excellent breakthrough opportunity for current deepfake detection efforts. In this paper, we present the task of Listening Deepfake Detection (LDD). We introduce ListenForge, the first dataset specifically designed for this task, constructed using five Listening Head Generation (LHG) methods. To address the distinctive characteristics of listening forgeries, we propose MANet, a Motion-aware and Audio-guided Network that captures subtle motion inconsistencies in listener videos while leveraging speaker's audio semantics to guide cross-modal fusion. Extensive experiments demonstrate that existing Speaking Deepfake Detection (SDD) models perform poorly in listening scenarios. In contrast, MANet achieves significantly superior performance on ListenForge. Our work highlights the necessity of rethinking deepfake detection beyond the traditional speaking-centric paradigm and opens new directions for multimodal forgery analysis in interactive communication settings. The dataset and code are available at https://anonymous.4open.science/r/LDD-B4CB.

Figures & tables

Figure 1. A visualization comparison between common speaking deepfakes and our studied listening deepfakes. The former relies on the synchronization between the lip region and the given audio during speaking, while the latter manifests forged facial movements such as smiling, nodding, or shaking the head during listening.

Figure 2. ListenForge dataset construction. Utilizing five methods, we generated realistic listening videos accompanied by the speaker’s audio. The figure on the right illustrates a subset of real and fake samples from ListenForge dataset.

Panel of Figure 3 — Figure 3. Distribution of the ListenForge dataset. (a) The ratio of real and fake samples. (b) The ratio of different methods used in forgery listening videos.

Figure 4. Illustration of the proposed MANet. (a) illustrates the overall framework. (b) illustrates the motion-aware module. (c) illustrates the audio-guided module.

Figure 5. Illustration of the spatial and channel attention.

Method	SDD		LDD
Method	AUC	ACC	AUC	ACC
Baseline	69.09	77.57	92.03	81.88

Table 1. The AUC(%) and ACC(%) on of the baseline on the speaker-targeted and listener-targeted deepfake testsets.

Method	Modality	Valset		Testset
Method	Modality	AUC	ACC	AUC	ACC
Xception	Visual	60.45	58.58	62.02	57.23
MesoNet	Visual	44.68	50.39	43.28	46.99
CViT	Visual	61.92	45.07	56.64	52.15
AVTFD	Audio-Visual	58.76	55.24	54.30	44.19
MRDF	Audio-Visual	55.81	62.47	45.98	50.47

Table 2. The AUC(%) and ACC(%) of the models on ListenForge without retraining.

Method	Modality	Valset		Testset
Method	Modality	AUC	ACC	AUC	ACC
Xception	Visual	97.92	98.05	88.24	87.23
MesoNet	Visual	99.86	96.99	89.31	88.80
CViT	Visual	97.97	97.59	90.16	87.96
AVTFD	Audio-Visual	99.39	98.43	93.19	89.28
MRDF	Audio-Visual	91.51	88.58	90.32	90.58

Table 3. The AUC(%) and ACC(%) of the models on ListenForge with retraining.

Method	Valset		Testset
Method	AUC	ACC	AUC	ACC
Baseline	98.71	93.51	92.03	81.88
+ CA	99.32	91.82	95.36	79.58
+ SA	98.56	91.20	95.36	84.40
+ SCA	9 9.18	94.34	95.43	8 4.08
+ CSA	98.01	91.20	94.42	78.53

Table 4. The AUC(%) and ACC(%) on ListenForge for models using different motion-aware modules. Underlined values indicate sub-optimal results.

Method	Valset		Testset
Method	AUC	ACC	AUC	ACC
Baseline + MAM	99.17	94.34	95.43	84.08
+ Spk_Aud	99.05	94.03	96.51	82.93
+ Spk_Aud + AGM	99.87	98.53	97.24	89.74
+ Spk_Vid + AGM	99.79	97.48	95.88	85.55
+ Spk_AV + AGM	99.88	98.43	95.80	83.77

Table 5. The AUC(%) and ACC(%) on ListenForge for models using different audio-guided module.

Figure 6. Visualization of the attention maps. The redder the color, the higher the attention allocated to that region by the model during the inference process.

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 241 papers

May 27, 2026

From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.

Ke Liu, Jiwei Wei, Wenyu Zhang +5

Center for Future Media, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China

Audio Deepfake Detection Domain Generalization

Ask PDF

Aug 10, 2026

MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection

Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.

Yanqiu Li, Yang Xiao, Jisheng Bai +3

The University of Melbourne, Melbourne, Australia · Xi’an University of Posts and Telecommunications, Xi’an, China · The University of Auckland, Auckland, New Zealand

Audio Deepfake Detection Deepfake Detection Audio Deepfake Detection Robustness

Ask PDF

Oct 5, 2026

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

Reliable and practical audio-visual DeepFake detection requires benchmarks that reflect diverse linguistic contexts and modern data synthesis pipelines for visual as well as audio manipulations. However, existing datasets predominantly contain footage of English-speakers, often include outdated manipulation types, or overlook the audio modality. Further, many datasets feature individuals who did not consent to be used in DeepFake creation. We introduce BabelFake, a multilingual audio-visual DeepFake benchmark recorded with consenting participants. BabelFake contains 399k clips (1,323 hours) from 496 individuals spanning five languages (English, German, Italian, French, Spanish). Our modular data generation pipeline pairs 11 modern video manipulation methods with 4 voice cloning engines, distinguishing visual-only (face swapping) and joint audio-visual manipulations (lip synchronization and portrait animation). By benchmarking state-of-the-art detectors, we show that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved. Cross-language/demographic evaluation reveals sensitivity varying across detector architectures and training data, while human evaluation reveals that perceived realism and machine-detection difficulty do not necessarily align.

Carlotta Segna, Joel Tschesche, Anna Rohrbach

TU Darmstadt & Hessian.AI, Germany

Audio Deepfake Detection Deepfake Detection Multimodal Deepfake Detection

Ask PDF

Abstract

Deepfake Detection

Explore similar work

From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark