Beyond Shortcut Learning: Hierarchical Spatial Prototype Regularization for Face Forgery Detection

Chao Shuai; Shaojing Fan; Kun Pan; Jinwu Hu; Weichen Lian; Ziqi Yang; Zhongjie Ba; Zhenguang Liu

cs.CVOct 21, 2026

Beyond Shortcut Learning: Hierarchical Spatial Prototype Regularization for Face Forgery Detection

Authors: Chao Shuai, Shaojing Fan, Kun Pan, Jinwu Hu, Weichen Lian, Ziqi Yang, Zhongjie Ba, Zhenguang Liu

Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · National University of Singapore, Singapore · South China University of Technology, Guangzhou, China

Abstract

Recent advances in generative models have enabled highly realistic face forgeries, yet generalizing deepfake detectors to unseen manipulation pipelines remains a major challenge. One critical yet underexplored issue is that the learned forensic representation itself may fail to remain stable across different manipulation sources. In this paper, we provide new evidence that naively fine-tuned vision models suffer from severe forensic representation collapse: their extracted cues are often entangled with unstable, dataset-specific artifacts, resulting in structurally fragile representations and poor cross-domain generalization. Leveraging this insight, we propose Hierarchical Spatial Prototype (HSP) regularization, which explicitly models the forgery manifold by learning a structured prototype subspace over spatial features. We further introduce prototype orthogonality and second-order assignment balance, which jointly preserve the geometric independence of latent forgery factors and promote high-effective-rank utilization of the forensic representation, thereby reducing reliance on dataset-specific shortcuts. Empirical results demonstrate that our method substantially improves generalizability, yielding average frame-level AUC gains of 5.6% and 5.3% in the cross-dataset and cross-manipulation settings, respectively. Moreover, when used as a plug-in module, HSP boosts the average video-level AUC of existing state-of-the-art detectors to 95.3% (+1.8%). These results demonstrate the practical value of HSP as an easy-to-adopt regularization module for enhancing deepfake detectors.

Figures & tables

Figure 1: Structured-manifold modeling of forgery representations. (a) Existing methods separate real and fake samples within an entangled feature space. (b) Our approach instead organizes forgery traces into a structured manifold supported by multiple latent forgery factors. (c) The t-SNE visualization provides empirical support for this structured representation.

Methods	$\mathcal{\bar{H}}$	$\mathcal{I}$	$\mathcal{P}$	ACC
FaceForensics++
CLIP	$0.553^{\pm 0.009}$	$0.005^{\pm 0.002}$	$\mathbf{0.968^{\pm 0.027}}$	0.854
CLIP-FT	$0.997^{\pm 0.001}$	$0.518^{\pm 0.003}$	$0.815^{\pm 0.146}$	0.931
HSP-CLIP-FT	$\mathbf{0.998^{\pm 0.001}}$	$\mathbf{0.521^{\pm 0.001}}$	$0.917^{\pm 0.115}$	$\mathbf{0.954}$
Celeb-DF-v2
CLIP	$0.612^{\pm 0.012}$	$0.016^{\pm 0.003}$	$0.126^{\pm 0.121}$	0.665

Table 1: Quantitative analysis of feature representations across three models: pretrained and naively finetuned variants of CLIP ViT-L/14, alongside HSP-CLIP.

Figure 2: PCA-based analysis of forgery representations. HSP-CLIP requires more principal components to explain the same variance and produces broader cumulative spatial activations, suggesting that it captures more latent forensic factors within a structured subspace.

Figure 3: Overview of the proposed H ierarchical S patial P rototype ( HSP ) module for structured-manifold modeling. HSP introduces $K$ learnable spatial prototypes to model latent forgery factors. A soft assignment matrix $A$ decomposes each token into a combination of multiple prototypes and enables heterogeneous forgery traces to be modeled within a prototype-supported manifold. Prototype orthogonality keeps these forgery factors geometrically decoupled, while second-order assignment balance encourages diverse utilization of the prototype space and prevents the representation from collapsing into a few dominant factors.

Methods	Video-level AUC						Methods	Frame-level AUC
Methods	CDF-v2	CDF-v1	DFDC	DFDCP	DFD	Avg.	Methods	CDF-v2	CDF-v1	DFDC	DFDCP	DFD	Avg.
F3Net (ECCV’20)	78.9	–	71.8	74.9	84.4	77.5	Efficient-b4 $\dagger$ (ICML’19)	74.9	70.5	69.6	72.8	81.5	73.9
LaV (MM’23)	92.2	84.7	82.0	89.7	96.8	89.1	Face X-Ray (CVPR’20)	67.9	70.9	63.3	69.4	76.7	70.0
IID (CVPR’23)	83.8	–	70.0	68.9	93.9	79.2	RECCE (CVPR’22)	73.2	–	71.3	74.2	74.2	73.2
CFM (TIFS’23)	89.7	–	70.6	80.2	95.2	83.9	UCF $\dagger$ (ICCV’23)	82.4	77.9	80.5	75.9	87.8	80.9
SBIs (CVPR’22)	90.6	–	75.2	87.7	88.2	85.4	ICT (CVPR’23)	85.7	81.4	–	–	84.1	83.7

Table 2: Comparison of video- and frame-level AUC (%) on cross-dataset benchmarks.

\dagger

denotes results from official models or our reproduction; other results are from ( Yan et al. 2024b ; Chen et al. 2024 ) and original papers.

Methods	Video-level AUC							Methods	Frame-level AUC
Methods	UniFace	e4s	FaceDancer	FSGAN	InSwap	SimSwap	Avg.	Methods	UniFace	e4s	FaceDancer	FSGAN	InSwap	SimSwap	Avg.
RECCE (CVPR’22)	89.8	68.3	84.8	94.9	84.8	76.8	83.2	RECCE (CVPR’22)	84.2	65.2	78.3	88.4	79.5	73.0	78.1
SBIs (CVPR’22)	72.4	75.0	59.4	80.3	71.2	70.1	71.4	IID (CVPR’23)	79.5	71.0	79.0	86.4	74.4	64.0	75.7
UCF (ICCV’23)	83.1	73.1	86.2	93.7	80.9	64.7	80.3	SBIs (CVPR’22)	64.4	69.0	44.7	87.9	63.3	56.8	64.4
IID (CVPR’23)	83.9	76.6	84.4	92.7	78.9	64.4	80.2	UCF (ICCV’23)	78.7	69.2	80.0	88.1	76.8	64.9	77.5
LSDA (CVPR’24)	87.2	69.4	72.1	93.9	85.5	69.3	79.6	LSDA (CVPR’24)	85.4	68.4	75.9	83.2	81.0	72.7	77.8

Table 3: Video-level and frame-level AUC (%) comparison on unseen manipulation methods. Results are reported on six representative face-swapping methods selected from DF40, where all samples are generated based on the FF++ domain.

$\mathcal{L}_{rec}$	$\mathcal{L}_{orth}$	$\mathcal{L}_{bal}$	CDF-v2	DFDCP	DFDC	InSwap
✗	✗	✗	78.6	77.9	72.8	79.8
✓	✗	✗	81.3	80.3	76.2	81.5
✓	✓	✗	82.1	81.0	77.1	82.1
✓	✓	✓	82.8	81.6	77.5	82.9

Table 4: Ablation studies (ACC, %) regarding the proposed HSP-CLIP with loss constraints.

Figure 4: Performance gains of HSP over Effort and ablation study of HSP insertion positions. Figure (a) and (b) report video-level AUC (%) and frame-level ACC (%).

Method	CDF-v2		CDF-v1		FSGAN
Method	ACC	AUC	ACC	AUC	ACC	AUC
ViT-Base	70.3	79.3	68.8	76.4	76.9	84.9
+Ours	77.4 +7.1	82.7 +3.4	75.6 +6.8	83.5 +7.1	82.7 +5.8	90.4 +5.5
SigLIP	69.9	82.8	72.8	82.4	85.3	90.7
+Ours	74.9 +5.0	85.9 +3.1	77.2 +4.4	86.1 +3.7	87.9 +2.6	94.4 +3.7
CLIP	78.6	84.5	79.2	87.4	88.7	93.9

Table 5: Ablation studies (%) of HSP integrated into different vision foundation models (VFMs).

Figure 5: Impact of different numbers of prototypes (

K

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Robustness evaluation under various unseen perturbations. The grouped bar charts illustrate the video-level AUC (%) performance of our method versus competing approaches across five different perturbation types: Color Saturation, JPEG Compression, Color Contrast, Block Wise, and Gaussian Blur. Performance is plotted against increasing severity levels from 0 to 5. Overall, our method demonstrates the strongest comprehensive robustness, achieving highly competitive performance across different unseen perturbation types.

Method	$\bar{\mathcal{H}}$			$\mathcal{P}$
Method	$M=10$	$M=20$	$M=30$	$M=10$	$M=20$	$M=30$
Celeb-DF-v2
CLIP-FT	0.996	0.973	0.907	0.483	0.853	0.914
HSP-CLIP-FT	0.998	0.985	0.936	0.832	0.960	0.934
DFDCP
CLIP-FT	0.965	0.950	0.861	0.544	0.914	0.942

Appendix

Table 6: Quantitative comparison of the feature representations learned by CLIP-FT and HSP-CLIP under different values of

M

, measured by cluster-level fake density

\bar{\mathcal{H}}

and cross-dataset fake density correlation

\mathcal{P}

$K$	Metric	CD2	CD1	DFDCP	DFD	DFDC	SimSwap	UniFace	e4s	FaceDancer	FSGAN	InSwap
8	ACC	79.1	78.9	78.4	91.2	76.8	83.0	87.2	90.3	83.1	89.1	83.2
8	AUC	89.5	90.6	87.8	94.1	87.2	89.2	93.9	96.8	90.4	95.5	90.3
16	ACC	81.6	81.3	79.5	90.9	76.3	85.0	87.7	90.1	84.6	90.2	84.2
16	AUC	89.8	91.3	87.0	94.1	86.2	91.9	95.1	97.0	92.3	95.9	91.6
32	ACC	82.0	82.9	80.3	90.9	76.7	83.6	89.1	93.1	85.1	90.4	82.4
32	AUC	90.3	89.7	87.9	94.2	85.6	90.2	94.7	97.7	92.4	96.3	90.4

Appendix

Table 7: Performance comparison across different numbers of prototypes. Results are reported in frame-level ACC and AUC (%). The best result for each dataset and metric is shown in bold . Blue cells highlight the selected high-performance region.

Weighting Strategy	Metric	CD2	CD1	DFDCP	DFD	DFDC	SimSwap	UniFace	e4s	FaceDancer	FSGAN	InSwap
Uniform	ACC	82.2	83.8	81.8	92.2	76.8	80.7	85.1	89.8	83.5	90.6	82.1
Uniform	AUC	90.6	91.8	88.9	93.7	85.9	87.9	92.7	97.0	91.7	95.5	90.2
Hierarchical	ACC	82.8	83.5	81.6	90.1	77.5	81.6	86.0	91.1	83.5	90.7	82.9
Hierarchical	AUC	90.9	91.9	90.0	94.3	86.7	88.7	93.4	97.2	91.8	95.8	91.4

Appendix

Table 8: Ablation study on layer-wise loss weighting. Uniform Weighting assigns identical loss weights to both HSP layers, whereas Hierarchical Weighting reduces the loss weights of the second HSP layer by half. Results are reported in frame-level ACC and AUC (%), with the best result for each dataset and metric shown in bold .

Figure 7: Cosine similarity analysis of the learned spatial prototypes in the two HSP modules ( $K=128$ ). (a,b) Prototype cosine similarity matrices, where the off-diagonal entries are concentrated near zero. (c) Distributions of off-diagonal cosine similarities, with mean absolute values of $1.90\times 10^{-4}$ and $4.66\times 10^{-4}$ for Modules 1 and 2, respectively. These near-zero similarities demonstrate that HSP learns geometrically decoupled prototypes with minimal redundancy.

Figure 8: Prototype assignment balance across the two HSP modules ( $K=128$ ) on CDF-v2. The left panel shows the prototype-usage frequencies, with the red dashed line denoting the uniform baseline ( $1/K$ ); both modules achieve Perplexity scores close to the theoretical maximum of 128. The right panel shows the normalized eigenvalue spectra of the centered assignment covariance, where the effective ranks reach 95.09 and 103.87 and approach the maximum admissible value of 127, indicating broad utilization of the admissible assignment subspace.

Figure 9: Visualization of spatial prototype activation maps. We visualize the bilinearly upsampled token-to-prototype soft assignments for 8 selected prototypes (columns) across various input faces (rows). Notably, each prototype isolates distinct and consistent localized regions ( e.g. , eyes, mouth, or blending boundaries) regardless of the subject’s identity. This demonstrates that the HSP framework successfully disentangles forgery artifacts into diverse, independent structural patterns, mitigating over-reliance on localized shortcuts.

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 240 papers

Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection
Jun 1, 2026Yihui Wang, Yonghui Yang, Jilong Liu +3OOD Generalization Deepfake Detection
Hefei University of Technology China · National University of Singapore Singapore
Cross-Domain Generalization Limits of Vision Foundation Models in Facial Deepfake Detection
May 24, 2026Ibrahim DelibasogluVision Foundation Models Cross-Domain Generalization
Department of Software Engineering, Faculty of Computer and Information Sciences, Sakarya University, Esentepe, Sakarya, 54050 Türkiye.
Benchmarking Spatial, Spectral, and Self-Supervised Cues for Face Forgery Detection under Realistic Degradation
Sep 1, 2026Lucas Cunha, Lucas Sotomaior, Lucas Gasperin +3Image Corruption Robustness Image Forgery Detection
Pontifical Catholic University of Paraná, Curitiba, Brazil

Jun 1, 2026J/K move · Enter open · S save

Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection

Yihui Wang, Yonghui Yang, Jilong Liu, Fengbin Zhu+2

Hefei University of Technology China · National University of Singapore Singapore

OOD Generalization Deepfake Detection Shortcut Learning

Open paper AskPDF arXiv

Deepfake detection suffers from poor generalization across forgery methods, as existing models tend to rely on spurious method-specific shortcuts that fail to transfer to unseen manipulations. While recent approaches attempt to improve generalization, they lack an explicit mechanism to identify and suppress such shortcuts in learned representations. In this work, we propose Shortcut Subspace Suppression (S^3) framework that explicitly characterizes and suppresses method-specific shortcuts via subspace modeling. Our key insight is that variations distinguishing different forgery methods capture method-specific artifacts and thus serve as an effective proxy for method-specific shortcuts. To this end, we train a lightweight linear probe for forgery method classification and perform Singular Value Decomposition (SVD) to extract the dominant shortcut subspace. Building on this formulation, we develop two complementary strategies to reduce shortcut reliance. During training, we softly suppress the shortcut subspace in feature representations, encouraging the model to rely on more generalizable cues for real/fake discrimination. At inference time, we introduce a training-free counterpart that attenuates neurons aligned with the identified shortcut directions, enabling plug-and-play generalization enhancement with improved interpretability. Extensive experiments on multiple benchmarks demonstrate that our method significantly improves cross-method generalization while maintaining strong in-domain performance. The code will be released upon acceptance of the submission.

Abstract

Deepfake Detection

Explore similar work

Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection

Cross-Domain Generalization Limits of Vision Foundation Models in Facial Deepfake Detection

Benchmarking Spatial, Spectral, and Self-Supervised Cues for Face Forgery Detection under Realistic Degradation