Organizations: Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250014, China · Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University · Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, 518129 · Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University · School of Computing and Information Technology, Great Bay University, Dongguan, 523000, China · Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250014, China · Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology, Tianjin, 300384, China
Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods due to the limited exploration in visual modalities alone. They tend to assess the attribution or detection performance of models on unseen advanced generators, coarsely, and fail to consider the synergy of the two tasks. To this end, we propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD). Specifically, we conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models. Additionally, we introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks. Built upon the novel observation that there are significant distribution differences between pristine and forged gaze vectors, and the preservation of the target gaze in facial images generated by GAN and diffusion varies significantly, we design a visual perception encoder to employ the inherent gaze differences to mine global forgery embeddings across appearance and gaze domains. We propose a gaze-aware image encoder (GIE) that fuses forgery gaze prompts extracted via a gaze encoder with common forged image embeddings to capture general attribution patterns, allowing features to be transformed into a more stable and common DFAD feature space. We build a language refinement encoder (LRE) to generate dynamically enhanced language embeddings via an adaptive-enhanced word selector for precise vision-language matching. Extensive experiments on our benchmark show that our model outperforms the state-of-the-art by 6.56% ACC and 5.32% AUC in average performance under the attribution and detection settings, respectively. Codes will be available on GitHub.
Figures & tables
Fig. 1: Illustration of the gaze-sensitive prior trait. Each column shows a face yielded by various generators. The first to second rows display the RGB image and the gaze image derived from the pre-trained gaze estimator, respectively. We randomly select 10k real gaze prior vectors and 10k fake ones for each generator to calculate the FID score. The higher the FID score, the greater the difference between the real and fake gaze vector distributions. EFS denotes entire face synthesis, FS is face swap, and AM means attribute manipulation.
Fig. 2: Existing DFA (a) or DFD (b) tasks conduct the coarse-grained evaluation, with generators that are mixed and lack flow models, and fail to achieve collaboration between DFA and DFD. (c) Our DFAD task features the fine-grained evaluation of the model generalization on advanced generators, such as diffusion and flow models, and realize the synergy of DFA and DFD. (d) Existing CLIP-based models hardly introduce the forgery prior and perform dynamically enhanced language modelling. (e) Our GazeCLIP method employs the novel gaze prior and explores the adaptive-enhanced fine-grained language embeddings to achieve powerful generalization.
Fig. 3: The workflow of our proposed GazeCLIP. After obtaining multiple patches and fine-grained texts of the input face image, VPE is employed to generate gaze features via the gaze encoder and global gaze-aware appearance forgery patterns by AGPM. Then the gaze features and multiple patches are passed into GIE to generate general gaze-guided image embeddings. They are then fused with gaze-appearance forgery traces to derive visual counterfeit features. LRE generates adaptively enhanced language representations via AWS for vision-language matching. At last, the DFAD module takes visual counterfeit embeddings as input to make predictions. During testing, the trained VPE and GIE module are applied to achieve DFAD.
Fig. 4: The pipeline of the gaze injector. Image forgery embeddings are decomposed into the class token and patch tokens, and only the class token is used as a query to interact with gaze features to extract gaze-aware common forgery patterns.
Fig. 5: The architecture of the adaptive-enhanced word selector. The brighter (yellow) the color, the more important the word features.
Fig. 6: Resilience to unseen image distortions.
Seen Sets
Unseen Sets
Type
Sub-Type
Generator
Train #
Test #
DFD Label
DFA Label
GenFace
-
EFS
GAN
StyleGAN2 [ 29 ]
10,000
1,250
1
9
GenFace
-
EFS
GAN
StyleGAN3 [ 30 ]
10,000
1,250
1
10
GenFace
-
EFS
Diffusion
LatDiff [ 31 ]
10,000
1,250
1
3
GenFace
-
EFS
Diffusion
CollDiff [ 3 ]
10,000
1,250
1
4
GenFace
-
EFS
Diffusion
DDPM [ 32 ]
10,000
1,250
1
5
GenFace
-
FS
GAN
FSLSD [ 33 ]
10,000
1,250
1
6
TABLE I: The outline of the dataset construction and protocol. # denotes the number of the dataset. 0 means the real image, and 1 is the fake image. -1 denotes the unseen generator.
Model
Attribution
Detection
FFHQ
SimSwap
InSwapper
UniFace
e4s
VQGAN
SD-2.1
DiT
SiT
REFace
Average
Average
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
AUC
ResNet [ 59 ]
38.24
37.44
48.80
48.00
65.36
46.24
59.60
60.56
61.44
46.48
51.22
83.10
63.22
Xception [ 57 ]
24.64
58.08
70.64
65.28
76.16
49.04
74.32
84.32
83.52
48.64
64.02
78.58
50.78
ViT [ 58 ]
48.00
65.52
62.00
64.24
58.48
53.12
63.92
58.72
56.80
58.32
58.91
84.42
62.31
CViT [ 20 ]
1.28
34.24
36.72
31.44
49.76
13.28
49.84
66.64
65.04
36.56
38.48
86.29
64.73
TABLE II: Comparison of attribution and detection performance across different methods. For attribution, ACC scores (%) on unseen generators from DF40 and FFHQ, after training using seen generators and CelebA-HQ. For detection, ACC and AUC scores (%) on unseen generators are averaged. Bold and underline denote the best and the second-best results.
Model
Attribution
Detection
LivePortrait
LIA
FSRT
DiffusionAct
EDTalk
AniTalker
Real3DPortrait
FLOAT
Average
Average
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
AUC
ResNet [ 59 ]
42.56
40.80
46.24
62.40
51.76
46.64
44.88
34.08
46.17
88.74
77.95
Xception [ 57 ]
65.28
68.16
67.28
84.08
40.00
34.80
97.04
74.80
66.43
85.27
35.95
ViT [ 58 ]
28.16
16.88
17.84
11.20
12.80
5.92
70.48
80.56
30.48
88.87
68.72
CViT [ 20 ]
27.84
16.32
17.60
11.44
12.24
5.60
70.16
80.64
30.23
88.88
68.78
TABLE III: Comparison of attribution and detection performance across different methods. For attribution, ACC scores (%) on unseen generators from Celeb-DF++, after training using seen generators and CelebA-HQ. For detection, ACC and AUC scores (%) on unseen generators are averaged. Bold and underline denote the best and the second-best results.
Methods
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
GazeCLIP w/AWS†
80.66
89.80
77.35
GazeCLIP w/AWS*
71.66
89.68
77.53
GazeCLIP w/AWS‡
79.34
89.25
70.24
TABLE IV: Ablation study of the placement of AWS. † denotes the AWS before MHA and FF. * means the AWS between MHA and FF. ‡ is the AWS after MHA and FF.
Gaze Estimator
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
Park et al. [ 60 ]
68.20
76.13
70.57
PureGaze [ 61 ]
72.13
80.44
75.89
ETH-Xgaze [ 16 ]
80.66
89.80
77.35
TABLE V: Effects of various gaze estimators.
Model
Attribution
Detection
Unseen
Unseen
AGPM
GE
GIE
LRE
ACC
ACC
AUC
✓
51.96
81.79
63.72
✓
✓
62.03
83.82
65.41
✓
✓
✓
64.65
84.77
65.53
✓
✓
✓
67.99
86.78
68.26
TABLE VI: Model ablation.
Method
Attribution
Detection
Params (M)
FLOPs (G)
Inference Time (s)
Celeb-DF
DFDC
Wild Deepfake
Average
Average
ACC
ACC
ACC
ACC
ACC
AUC
ResNet [ 59 ]
53.36
55.44
72.56
60.45
74.80
50.07
11.19
1.82
0.004
Xception [ 57 ]
52.72
80.40
81.92
71.68
75.06
22.38
20.83
4.64
0.006
ViT [ 58 ]
62.88
70.88
63.76
65.84
77.12
74.86
85.63
16.88
0.007
CViT [ 20 ]
52.16
60.08
73.92
62.05
73.82
38.97
91.11
6.69
0.007
TABLE VII: Comparison of performance and computational efficiency on unseen datasets, including Celeb-DF [ 62 ] , DFDC [ 63 ] , and WildDeepfake [ 64 ] , after training using seen generators and CelebA-HQ.
Model
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
Params (M)
FLOPs (G)
CLIP w/o AWS
74.10
65.55
69.91
84.26
15.91
CLIP w/ AWS
78.36
73.29
70.26
84.26
15.91
Ours w/o AWS
60.42
87.68
68.31
204.40
17.65
Ours w/ AWS
80.66
89.80
77.35
204.40
17.65
TABLE IX: Ablations of the plug-and-play AWS.
Fig. 7: Visualization of gaze vectors produced by various gaze estimators, including ETH-XGaze and PureGaze, for fake images synthesized by various generators. We randomly select 10k face image pairs, each consisting of a real target face image and a corresponding forged face image, to extract gaze vectors using the pre-trained gaze estimator, and calculate the cosine similarity for each pair of gaze vectors. The final score is obtained by averaging these similarities.
Fig. 8: The heatmap visualizations of our GazeCLIP model (w/o or w/ gaze) on different generator samples. The hotter (red color) a position is, the more forgery features are captured by the network.
Fig. 9: Left: The attribution performance of various DFAD models (only w/ AE or w/ AE+DE). Right: The detection performance of various DFAD models (only w/ DE or w/ AE+DE). AE and DE are attribution expert and detection expert, respectively.
Model
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
Params (M)
FLOPs (G)
Query w/patch
78.96
88.94
76.73
204.40
18.54
Query w/all
81.23
88.88
72.48
204.40
18.56
Query w/cls
80.66
89.80
77.35
204.40
17.50
TABLE X: Ablations of various queries. Results are averaged over unseen generators for attribution and detection.
Model
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
L1
72.76
84.07
61.73
L1+L2
78.03
85.38
63.59
L1+L2+L3
67.80
86.26
65.25
L1+L2+L3+L4
80.66
89.80
77.35
TABLE XI: Ablations of various language prompts. LP is the learnable language prompts. Results are averaged over unseen generators for attribution and detection.
LoRA Number
Attribution
Detection
Unseen
Unseen
GIE
LRE
ACC
ACC
AUC
1
1
78.86
85.27
46.48
2
1
80.66
89.80
77.35
3
1
80.07
89.70
66.45
4
1
71.13
90.00
76.15
TABLE XII: Ablation study of different numbers of LoRAs. Results are averaged over unseen generators for attribution and detection.
Fig. 10: The t-SNE visualization of various generator distributions generated by GazeCLIP without or with language prompts. We randomly select 500 real samples, and for each generator, we pick 500 fake samples.
Model
Attribution
Detection
Unseen
Unseen
ACC
ACC
AUC
Ours w/o GI
74.89
85.90
70.63
Ours w/o Gaze
65.87
80.97
68.22
Ours w/ finetuned GE
78.48
86.39
73.71
GazeCLIP (Ours)
80.66
89.80
77.35
TABLE XIII: Ablation study. Results are averaged over unseen generators for attribution and detection.
Fig. 11: Visualization of language feature matrix created by CLIP or Gaze CLIP (w/o or w/ AWS) for generator samples. The brighter the colour, the more important the word embedding.
The rapid advancement of deepfake generation techniques poses significant threats to public safety and causes societal harm through the creation of highly realistic synthetic facial media. While existing detection methods demonstrate limitations in generalizing to emerging forgery patterns, this paper presents Deepfake Forensics Adapter (DFA), a novel dual-stream framework that synergizes vision-language foundation models with targeted forensics analysis. Our approach integrates a pre-trained CLIP model with three core components to achieve specialized deepfake detection by leveraging the powerful general capabilities of CLIP without changing CLIP parameters: 1) A Global Feature Adapter is used to identify global inconsistencies in image content that may indicate forgery, 2) A Local Anomaly Stream enhances the model's ability to perceive local facial forgery cues by explicitly leveraging facial structure priors, and 3) An Interactive Fusion Classifier promotes deep interaction and fusion between global and local features using a transformer encoder. Extensive evaluations of frame-level and video-level benchmarks demonstrate the superior generalization capabilities of DFA, particularly achieving state-of-the-art performance in the challenging DFDC dataset with frame-level AUC/EER of 0.816/0.256 and video-level AUC/EER of 0.836/0.251, representing a 4.8% video AUC improvement over previous methods. Our framework not only demonstrates state-of-the-art performance, but also points out a feasible and effective direction for developing a robust deepfake detection system with enhanced generalization capabilities against the evolving deepfake threats. Our code is available at https://github.com/Liao330/DFA.git
Jianfeng Liao, Yichen Wei, Raymond Chan Ching Bon +3
Shenzhen Technology University, Guangdong, China · Singapore Institute of Technology, Singapore, Singapore · The University of Hong Kong, Hong Kong, China +1
Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. While multimodal detection has shown promise, most approaches are binary classification tasks that often latch onto dataset-specific artifacts rather than genuine generative traces. We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal. Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space, forcing the model to encode generator-specific forensic content rather than shortcuts. We propose the Attribution-Guided Multimodal Deepfake Detection (AMDD) framework, which jointly learns to detect and attribute manipulation. AMDD treats generator attribution as a structured regularization that constrains representation geometry toward forensically meaningful features. We introduce a Cross-Modal Forensic Fingerprint Consistency (CMFFC) loss to enforce alignment between generator-induced artifacts in visual and audio streams. This exploits the fact that coherent manipulation leaves correlated traces across modalities, grounded in the physical coupling between speech and facial articulation that synthetic pipelines routinely disrupt. Architecturally, we pair a ResNet50 with temporal attention for visual encoding against a pretrained ResNet18 for mel spectrograms, closing the encoder capacity gap found in prior models. On FakeAVCeleb, AMDD achieves 99.7% balanced accuracy and 99.8% AUC with 95.9% attribution accuracy. Cross-dataset evaluation on DeepfakeTIMIT, DFDM, and LAV-DF confirms that real video detection generalizes robustly, while fake detection on unseen generators remains an open challenge that we analyze in depth.
Wasim Ahmad, Wei Zhang, Xuerui Mao
School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
The challenge of fair deepfake detection (FDD) has attracted increasing attention. Existing fairness-enhanced detectors often suffer from suboptimal generalization to unseen manipulations and fairness across demographic groups. They are typically developed and evaluated on demographically imbalanced distributions, resulting in biased predictions toward minority groups. In this paper, we construct a novel demographically balanced FDD benchmark to train and evaluate the fairness of detectors under both balanced and imbalanced population scenarios. Additionally, we introduce a novel expression and demographic perceptual vision-language model, termed FairForensics, for generalizable fair deepfake detection. FairForensics conducts face forgery generalization enhancement and demographic-aware fairness regularization. During face forgery generalization enhancement, built upon the novel observation of significant distribution differences between pristine and forged expression vectors, we design an expression encoder to capture high-level expression-guided forgery patterns, and an expression-perceptual visual encoder that integrates global appearance and expression forgery features while mitigating identity bias using an identity-aware patch perturbation module. Under demographic-aware fairness regularization, we propose a demographic-guided language encoder to extract population-aware global language embeddings, which boosts the decoupling of forgery features from demographic information via vision-language alignment. We devise a population-aware prototype fairness objective to enforce both inter-class separability and intra-class alignment across demographic subgroups. Extensive experiments on our balanced demographic benchmark show that our method achieves the state-of-the-art in terms of generalization and fairness.
Yaning Zhang, Jiao Wu, Zan Gao +1
Computer Vision Institute, School of Artificial Intelligence, Shenzhen University, Shenzhen, 518060, China · Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250014, China · Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology, Tianjin, 300384, China +2