Beyond Shortcut Learning: Hierarchical Spatial Prototype Regularization for Face Forgery Detection
Authors: Chao Shuai, Shaojing Fan, Kun Pan, Jinwu Hu, Weichen Lian, Ziqi Yang, Zhongjie Ba, Zhenguang Liu
Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · National University of Singapore, Singapore · South China University of Technology, Guangzhou, China
Recent advances in generative models have enabled highly realistic face forgeries, yet generalizing deepfake detectors to unseen manipulation pipelines remains a major challenge. One critical yet underexplored issue is that the learned forensic representation itself may fail to remain stable across different manipulation sources. In this paper, we provide new evidence that naively fine-tuned vision models suffer from severe forensic representation collapse: their extracted cues are often entangled with unstable, dataset-specific artifacts, resulting in structurally fragile representations and poor cross-domain generalization. Leveraging this insight, we propose Hierarchical Spatial Prototype (HSP) regularization, which explicitly models the forgery manifold by learning a structured prototype subspace over spatial features. We further introduce prototype orthogonality and second-order assignment balance, which jointly preserve the geometric independence of latent forgery factors and promote high-effective-rank utilization of the forensic representation, thereby reducing reliance on dataset-specific shortcuts. Empirical results demonstrate that our method substantially improves generalizability, yielding average frame-level AUC gains of 5.6% and 5.3% in the cross-dataset and cross-manipulation settings, respectively. Moreover, when used as a plug-in module, HSP boosts the average video-level AUC of existing state-of-the-art detectors to 95.3% (+1.8%). These results demonstrate the practical value of HSP as an easy-to-adopt regularization module for enhancing deepfake detectors.
Figures & tables
Figure 1: Structured-manifold modeling of forgery representations. (a) Existing methods separate real and fake samples within an entangled feature space. (b) Our approach instead organizes forgery traces into a structured manifold supported by multiple latent forgery factors. (c) The t-SNE visualization provides empirical support for this structured representation.
Methods
Hˉ
I
P
ACC
FaceForensics++
CLIP
0.553±0.009
0.005±0.002
0.968±0.027
0.854
CLIP-FT
0.997±0.001
0.518±0.003
0.815±0.146
0.931
HSP-CLIP-FT
0.998±0.001
0.521±0.001
0.917±0.115
0.954
Celeb-DF-v2
CLIP
0.612±0.012
0.016±0.003
0.126±0.121
0.665
Table 1: Quantitative analysis of feature representations across three models: pretrained and naively finetuned variants of CLIP ViT-L/14, alongside HSP-CLIP.
Figure 2: PCA-based analysis of forgery representations. HSP-CLIP requires more principal components to explain the same variance and produces broader cumulative spatial activations, suggesting that it captures more latent forensic factors within a structured subspace.
Figure 3: Overview of the proposed H ierarchical S patial P rototype ( HSP ) module for structured-manifold modeling. HSP introduces K learnable spatial prototypes to model latent forgery factors. A soft assignment matrix A decomposes each token into a combination of multiple prototypes and enables heterogeneous forgery traces to be modeled within a prototype-supported manifold. Prototype orthogonality keeps these forgery factors geometrically decoupled, while second-order assignment balance encourages diverse utilization of the prototype space and prevents the representation from collapsing into a few dominant factors.
Methods
Video-level AUC
Methods
Frame-level AUC
CDF-v2
CDF-v1
DFDC
DFDCP
DFD
Avg.
CDF-v2
CDF-v1
DFDC
DFDCP
DFD
Avg.
F3Net (ECCV’20)
78.9
–
71.8
74.9
84.4
77.5
Efficient-b4 † (ICML’19)
74.9
70.5
69.6
72.8
81.5
73.9
LaV (MM’23)
92.2
84.7
82.0
89.7
96.8
89.1
Face X-Ray (CVPR’20)
67.9
70.9
63.3
69.4
76.7
70.0
IID (CVPR’23)
83.8
–
70.0
68.9
93.9
79.2
RECCE (CVPR’22)
73.2
–
71.3
74.2
74.2
73.2
CFM (TIFS’23)
89.7
–
70.6
80.2
95.2
83.9
UCF † (ICCV’23)
82.4
77.9
80.5
75.9
87.8
80.9
SBIs (CVPR’22)
90.6
–
75.2
87.7
88.2
85.4
ICT (CVPR’23)
85.7
81.4
–
–
84.1
83.7
Table 2: Comparison of video- and frame-level AUC (%) on cross-dataset benchmarks. † denotes results from official models or our reproduction; other results are from ( Yan et al. 2024b ; Chen et al. 2024 ) and original papers.
Methods
Video-level AUC
Methods
Frame-level AUC
UniFace
e4s
FaceDancer
FSGAN
InSwap
SimSwap
Avg.
UniFace
e4s
FaceDancer
FSGAN
InSwap
SimSwap
Avg.
RECCE (CVPR’22)
89.8
68.3
84.8
94.9
84.8
76.8
83.2
RECCE (CVPR’22)
84.2
65.2
78.3
88.4
79.5
73.0
78.1
SBIs (CVPR’22)
72.4
75.0
59.4
80.3
71.2
70.1
71.4
IID (CVPR’23)
79.5
71.0
79.0
86.4
74.4
64.0
75.7
UCF (ICCV’23)
83.1
73.1
86.2
93.7
80.9
64.7
80.3
SBIs (CVPR’22)
64.4
69.0
44.7
87.9
63.3
56.8
64.4
IID (CVPR’23)
83.9
76.6
84.4
92.7
78.9
64.4
80.2
UCF (ICCV’23)
78.7
69.2
80.0
88.1
76.8
64.9
77.5
LSDA (CVPR’24)
87.2
69.4
72.1
93.9
85.5
69.3
79.6
LSDA (CVPR’24)
85.4
68.4
75.9
83.2
81.0
72.7
77.8
Table 3: Video-level and frame-level AUC (%) comparison on unseen manipulation methods. Results are reported on six representative face-swapping methods selected from DF40, where all samples are generated based on the FF++ domain.
Lrec
Lorth
Lbal
CDF-v2
DFDCP
DFDC
InSwap
✗
✗
✗
78.6
77.9
72.8
79.8
✓
✗
✗
81.3
80.3
76.2
81.5
✓
✓
✗
82.1
81.0
77.1
82.1
✓
✓
✓
82.8
81.6
77.5
82.9
Table 4: Ablation studies (ACC, %) regarding the proposed HSP-CLIP with loss constraints.
Figure 4: Performance gains of HSP over Effort and ablation study of HSP insertion positions. Figure (a) and (b) report video-level AUC (%) and frame-level ACC (%).
Method
CDF-v2
CDF-v1
FSGAN
ACC
AUC
ACC
AUC
ACC
AUC
ViT-Base
70.3
79.3
68.8
76.4
76.9
84.9
+Ours
77.4 +7.1
82.7 +3.4
75.6 +6.8
83.5 +7.1
82.7 +5.8
90.4 +5.5
SigLIP
69.9
82.8
72.8
82.4
85.3
90.7
+Ours
74.9 +5.0
85.9 +3.1
77.2 +4.4
86.1 +3.7
87.9 +2.6
94.4 +3.7
CLIP
78.6
84.5
79.2
87.4
88.7
93.9
Table 5: Ablation studies (%) of HSP integrated into different vision foundation models (VFMs).
Figure 5: Impact of different numbers of prototypes ( K ).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Robustness evaluation under various unseen perturbations. The grouped bar charts illustrate the video-level AUC (%) performance of our method versus competing approaches across five different perturbation types: Color Saturation, JPEG Compression, Color Contrast, Block Wise, and Gaussian Blur. Performance is plotted against increasing severity levels from 0 to 5. Overall, our method demonstrates the strongest comprehensive robustness, achieving highly competitive performance across different unseen perturbation types.
Method
Hˉ
P
M=10
M=20
M=30
M=10
M=20
M=30
Celeb-DF-v2
CLIP-FT
0.996
0.973
0.907
0.483
0.853
0.914
HSP-CLIP-FT
0.998
0.985
0.936
0.832
0.960
0.934
DFDCP
CLIP-FT
0.965
0.950
0.861
0.544
0.914
0.942
Appendix
Table 6: Quantitative comparison of the feature representations learned by CLIP-FT and HSP-CLIP under different values of M , measured by cluster-level fake density Hˉ and cross-dataset fake density correlation P .
K
Metric
CD2
CD1
DFDCP
DFD
DFDC
SimSwap
UniFace
e4s
FaceDancer
FSGAN
InSwap
8
ACC
79.1
78.9
78.4
91.2
76.8
83.0
87.2
90.3
83.1
89.1
83.2
AUC
89.5
90.6
87.8
94.1
87.2
89.2
93.9
96.8
90.4
95.5
90.3
16
ACC
81.6
81.3
79.5
90.9
76.3
85.0
87.7
90.1
84.6
90.2
84.2
AUC
89.8
91.3
87.0
94.1
86.2
91.9
95.1
97.0
92.3
95.9
91.6
32
ACC
82.0
82.9
80.3
90.9
76.7
83.6
89.1
93.1
85.1
90.4
82.4
AUC
90.3
89.7
87.9
94.2
85.6
90.2
94.7
97.7
92.4
96.3
90.4
Appendix
Table 7: Performance comparison across different numbers of prototypes. Results are reported in frame-level ACC and AUC (%). The best result for each dataset and metric is shown in bold . Blue cells highlight the selected high-performance region.
Weighting Strategy
Metric
CD2
CD1
DFDCP
DFD
DFDC
SimSwap
UniFace
e4s
FaceDancer
FSGAN
InSwap
Uniform
ACC
82.2
83.8
81.8
92.2
76.8
80.7
85.1
89.8
83.5
90.6
82.1
AUC
90.6
91.8
88.9
93.7
85.9
87.9
92.7
97.0
91.7
95.5
90.2
Hierarchical
ACC
82.8
83.5
81.6
90.1
77.5
81.6
86.0
91.1
83.5
90.7
82.9
AUC
90.9
91.9
90.0
94.3
86.7
88.7
93.4
97.2
91.8
95.8
91.4
Appendix
Table 8: Ablation study on layer-wise loss weighting. Uniform Weighting assigns identical loss weights to both HSP layers, whereas Hierarchical Weighting reduces the loss weights of the second HSP layer by half. Results are reported in frame-level ACC and AUC (%), with the best result for each dataset and metric shown in bold .
Figure 7: Cosine similarity analysis of the learned spatial prototypes in the two HSP modules ( K=128 ). (a,b) Prototype cosine similarity matrices, where the off-diagonal entries are concentrated near zero. (c) Distributions of off-diagonal cosine similarities, with mean absolute values of 1.90×10−4 and 4.66×10−4 for Modules 1 and 2, respectively. These near-zero similarities demonstrate that HSP learns geometrically decoupled prototypes with minimal redundancy.
Figure 8: Prototype assignment balance across the two HSP modules ( K=128 ) on CDF-v2. The left panel shows the prototype-usage frequencies, with the red dashed line denoting the uniform baseline ( 1/K ); both modules achieve Perplexity scores close to the theoretical maximum of 128. The right panel shows the normalized eigenvalue spectra of the centered assignment covariance, where the effective ranks reach 95.09 and 103.87 and approach the maximum admissible value of 127, indicating broad utilization of the admissible assignment subspace.
Figure 9: Visualization of spatial prototype activation maps. We visualize the bilinearly upsampled token-to-prototype soft assignments for 8 selected prototypes (columns) across various input faces (rows). Notably, each prototype isolates distinct and consistent localized regions ( e.g. , eyes, mouth, or blending boundaries) regardless of the subject’s identity. This demonstrates that the HSP framework successfully disentangles forgery artifacts into diverse, independent structural patterns, mitigating over-reliance on localized shortcuts.
Deepfake detection suffers from poor generalization across forgery methods, as existing models tend to rely on spurious method-specific shortcuts that fail to transfer to unseen manipulations. While recent approaches attempt to improve generalization, they lack an explicit mechanism to identify and suppress such shortcuts in learned representations. In this work, we propose Shortcut Subspace Suppression (S^3) framework that explicitly characterizes and suppresses method-specific shortcuts via subspace modeling. Our key insight is that variations distinguishing different forgery methods capture method-specific artifacts and thus serve as an effective proxy for method-specific shortcuts. To this end, we train a lightweight linear probe for forgery method classification and perform Singular Value Decomposition (SVD) to extract the dominant shortcut subspace. Building on this formulation, we develop two complementary strategies to reduce shortcut reliance. During training, we softly suppress the shortcut subspace in feature representations, encouraging the model to rely on more generalizable cues for real/fake discrimination. At inference time, we introduce a training-free counterpart that attenuates neurons aligned with the identified shortcut directions, enabling plug-and-play generalization enhancement with improved interpretability. Extensive experiments on multiple benchmarks demonstrate that our method significantly improves cross-method generalization while maintaining strong in-domain performance. The code will be released upon acceptance of the submission.
Yihui Wang, Yonghui Yang, Jilong Liu +3
Hefei University of Technology China · National University of Singapore Singapore
The rapid evolution of generative models has enabled the creation of hyper-realistic facial deepfakes, exposing a critical vulnerability in modern digital forensics: the inability of detectors to generalize to unseen manipulation techniques. Traditional networks suffer from representation collapse, overfitting to localized artifact fingerprints of specific training generators. This work investigates whether modern Vision Foundation Models can serve as generalizable, out-of-the-box feature extractors capable of tracking forensic anomalies across entirely unseen generative manifolds. We conduct a systematic cross-domain evaluation comparing three foundational learning paradigms: fully supervised macro-semantic features (RoPE-ViT), pure self-supervised geometric features (DINOv3), and multi-teacher agglomerative representations (NVIDIA C-RADIOv4-H). By deploying frozen backbones subjected to downstream linear probing, we map the performance limitations of these architectures on the challenging DF40 benchmark. Our empirical findings expose the intrinsic trade-offs between pre-training paradigms and parameter scale, proving that while foundation models retain high discriminative capabilities for entire face synthesis, localized face editing techniques expose fundamental boundaries in linear probe evaluation structures. Source code and model weights are available in http://github.com/mribrahim/deepfake
Ibrahim Delibasoglu
Department of Software Engineering, Faculty of Computer and Information Sciences, Sakarya University, Esentepe, Sakarya, 54050 Türkiye.
Face forgery detectors often achieve strong results on controlled benchmarks, but their reliability under realistic image degradations remains limited. This paper presents a standardized benchmark for face forgery detection using the Multi-Dimensional Face Forgery Image (MFFI) dataset and evaluates performance on both clean and degraded test partitions. We compare six model families, including convolutional networks, transformer-based models, and a frozen self-supervised DINOv3 backbone, across spatial, spectral, and hybrid input representations. The results show that clean-set performance is not a reliable indicator of robustness under compression, resizing, and blurring. Xception with RGB obtains the best clean performance, reaching 0.884 mean ROC-AUC, but degrades substantially on the harder partition. In contrast, frozen DINOv3 achieves the strongest degraded-set result, with 0.726 mean ROC-AUC, while training only a linear classification head. The representation analysis indicates that Fourier-domain cues are most useful when combined with RGB information, whereas purely spectral inputs consistently underperform spatial representations. Qualitative attribution maps further suggest that convolutional detectors focus on localized artifacts, while DINOv3 relies on broader facial structure. These findings reinforce the need for degraded evaluation protocols and highlight self-supervised visual representations as a promising direction for robust face forgery detection. Our source code is publicly available at https://github.com/lucasdocunha/FaceForgery-Benchmark/.
Lucas Cunha, Lucas Sotomaior, Lucas Gasperin +3
Pontifical Catholic University of Paraná, Curitiba, Brazil