Beyond Semantic Priors: Mitigating Optimization Collapse for Generalizable Visual Forensics

Jipeng Liu; Haichao Shi; Siyu Xing; Rong Yin; Xiao-Yu Zhang

cs.CVMar 25, 2026

Beyond Semantic Priors: Mitigating Optimization Collapse for Generalizable Visual Forensics

Authors: Jipeng Liu, Haichao Shi, Siyu Xing, Rong Yin, Xiao-Yu Zhang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100193, China · School of Cyber Science and Technology, Beihang University, Beijing 100191, China

Abstract

While Vision-Language Models (VLMs) like CLIP have emerged as a dominant paradigm for generalizable deepfake detection, a representational disconnect remains: their semantic-centric pre-training is ill-suited for capturing non-semantic artifacts inherent to hyper-realistic synthesis. In this work, we identify a failure mode termed Optimization Collapse, where detectors trained with Sharpness-Aware Minimization (SAM) degenerate to random guessing on non-semantic forgeries once the perturbation radius exceeds a narrow threshold. To theoretically formalize this collapse, we propose the Critical Optimization Radius (COR) to quantify the geometric stability of the optimization landscape, and leverage the Gradient Signal-to-Noise Ratio (GSNR) to measure generalization potential. We establish a theorem proving that COR increases monotonically with GSNR, thereby revealing that the geometric instability of SAM optimization originates from degraded intrinsic generalization potential. This result identifies the layer-wise attenuation of GSNR as the root cause of Optimization Collapse in detecting non-semantic forgeries. Although naively reducing perturbation radius yields stable convergence under SAM, it merely treats the symptom without mitigating the intrinsic generalization degradation, necessitating enhanced gradient fidelity. Building on this insight, we propose the Contrastive Regional Injection Transformer (CoRIT), which integrates a computationally efficient Contrastive Gradient Proxy (CGP) with three training-free strategies: Region Refinement Mask to suppress CGP variance, Regional Signal Injection to preserve CGP magnitude, and Hierarchical Representation Integration to attain more generalizable representations. Extensive experiments demonstrate that CoRIT mitigates optimization collapse and achieves state-of-the-art generalization across cross-domain and universal forgery benchmarks.

Figures & tables

Fig. 1: Empirical observation of Optimization Collapse. (a) Detection performance at the standard SAM radius ( $\rho=0.05$ ). (b) Layer-wise evolution of the Critical Optimization Radius (COR). The red region denotes the ”Collapse Zone” (COR $<0.05$ ). (c) Representative samples from the evaluated datasets.

Fig. 2: AUC curves of models trained with Sharpness-Aware Minimization (SAM) across varying perturbation radii

\rho

. (a) Empirical analysis of Optimization Collapse. The Critical Optimization Radius (COR) (diamond markers) across various datasets. The red region highlights optimization failure at the standard radius (

\rho=0.05

). (b) AUC trends across the entire span of

\rho

. The gray dashed box indicates the localized region magnified on the right, highlighting AUC variations around the recommended radius (

\rho=0.05

Fig. 3: GSNR trajectories during optimization. (a) GSNR values across four datasets at the 24-th layer. (b) Layer-wise GSNR values on NeuralTextures. Diamond markers denote the bottleneck step

t^{*}

Fig. 4: Overview of the proposed Co ntrastive R egional I njection T ransformer (CoRIT) pipeline. Given an original image and its Self-Blended Image (SBI) counterpart, both are fed through the frozen CLIP image encoder in parallel. The discrepancy of visual tokens serves as the Contrastive Gradient Proxy (CGP). At each transformer layer, CoRIT applies three training-free components: (i) The Region Refinement Mask (RRM) clusters visual tokens around region anchors derived from the CGP, filtering out outlier tokens. (ii) The Regional Signal Injection (RSI) aggregates the refined tokens via average pooling and injects them into additional Region Tokens through intra-layer residual connections. (iii) The Hierarchical Representation Integration (HRI) concatenates the class token and region tokens from an intermediate layer $l_{\text{mid}}$ and the final layer $L$ for binary classification.

Method	Venues	CDFv1	CDFv2	DFDCP	DFDC	DFD	Avg.
Xception [ 34 ]	ICCV’19	77.9	73.7	73.7	70.8	81.6	75.5
EfficientB4 [ 18 ]	ICML’19	79.1	74.9	72.8	69.6	81.5	75.6
FWA [ 50 ]	CVPRW’19	79.0	66.8	63.8	-	74.0	-
F3Net [ 16 ]	AAAI’20	77.7	73.5	73.5	70.2	79.8	74.9
Face X-ray [ 7 ]	CVPR’20	70.9	67.9	69.4	63.3	76.6	69.6
FFD [ 51 ]	CVPR’20	78.4	74.4	74.3	70.3	80.2	75.5

TABLE I: Cross-domain performance comparison on classical forensic benchmarks (Frame-Level AUC (%)).

\dagger

denotes methods using a frozen backbone with a lightweight learnable classifier. Methods below the midrule share the same CLIP-ViT-L/14 backbone and SAM optimizer. Best results are in bold .

Fig. 5: Training loss curves of compared methods illustrating generalization stability. Dashed lines indicate intra-dataset evaluation; solid lines indicate cross-dataset evaluation.

Method	Venues	CDFv2	DFDC	DFDCP	Avg.
LipForensics [ 58 ]	CVPR’21	82.4	73.5	-	-
FTCN [ 59 ]	ICCV’21	86.9	-	74.0	-
PCL + I2G [ 60 ]	ICCV’21	90.0	67.5	74.3	77.3
HCIL [ 61 ]	ECCV’22	79.0	69.2	-	-
SBI [ 9 ]	CVPR’22	92.8	71.9	85.5	83.4
3DDet. [ 32 ]	TPAMI’23	72.0	72.4	-	-

TABLE II: Cross-domain performance on classical forensic benchmarks (Video-Level AUC (%)). Best results are in bold .

Method	Venues	uniface	e4s	facedancer	fsgan	inswap	Avg.
Recce [ 54 ]	CVPR’22	84.2	65.2	78.3	88.4	79.5	79.1
SBI [ 9 ]	CVPR’22	64.4	69.0	44.7	87.9	63.3	65.9
CORE [ 64 ]	CVPRW’22	81.7	63.4	71.7	91.1	79.4	77.5
IID [ 65 ]	CVPR’23	79.5	71.0	79.0	86.4	74.4	78.1
UCF [ 15 ]	ICCV’23	78.7	69.2	80.0	88.1	76.8	78.6
LSDA [ 19 ]	CVPR’24	85.4	68.4	75.9	83.2	81.0	78.8

TABLE III: Cross-domain performance on five representative face forgery types in DF40 (Frame-Level AUC (%)). Best results are in bold .

Method	GAN						Deep fakes	Low level		Perceptual loss		Guided	LDM			Glide			Dalle	mAP
	Pro-	Cycle-	Big-	Style-	Gau-	Star-		SITD	SAN	CRN	IMLE		200	200	100	100	50	100
	GAN	GAN	GAN	GAN	GAN	GAN		SITD	SAN	CRN	IMLE		steps	w/cfg	steps	27	27	10
CNN-Spot [ 68 ]	100.0	93.47	84.50	99.54	89.49	98.15	89.02	73.75	59.47	98.24	98.40	73.72	70.62	71.00	70.54	80.65	84.91	82.07	70.59	83.58
PatchForensics [ 69 ]	80.88	72.84	71.66	85.75	65.99	69.25	76.55	76.19	76.34	74.52	68.52	75.03	87.10	86.72	86.40	85.37	83.73	78.38	75.67	77.73
Co-occurrence [ 70 ]	99.74	80.95	50.61	98.63	53.11	67.99	59.14	68.98	60.42	73.06	87.21	70.20	91.21	89.02	92.39	89.32	88.35	82.79	80.96	78.11
Freq-spec [ 71 ]	55.39	100.0	75.08	55.11	66.08	100.0	45.18	47.46	57.12	53.61	50.98	57.72	77.72	77.25	76.47	68.58	64.58	61.92	67.77	66.21

TABLE IV: Generalization performance (mAP (%)) on next-generation generative benchmarks. Best and second-best results are in bold and underline .

Method	GAN						Deep fakes	Low level		Perceptual loss		Guided	LDM			Glide			Dalle	mAcc
	Pro-	Cycle-	Big-	Style-	Gau-	Star-		SITD	SAN	CRN	IMLE		200	200	100	100	50	100
	GAN	GAN	GAN	GAN	GAN	GAN		SITD	SAN	CRN	IMLE		steps	w/cfg	steps	27	27	10
CNN-Spot [ 68 ]	99.99	85.20	70.20	85.70	78.95	91.70	53.47	66.67	48.69	86.31	86.26	60.07	54.03	54.96	54.14	60.78	63.80	65.66	55.58	69.58
PatchForensics [ 69 ]	75.03	68.97	68.47	79.16	64.23	63.94	75.54	75.14	75.28	72.33	55.30	67.41	76.50	76.10	75.77	74.81	73.28	68.52	67.91	71.24
Co-occurrence [ 70 ]	97.70	63.15	53.75	92.50	51.10	54.70	57.10	63.06	55.85	65.65	65.80	60.50	70.70	70.55	71.00	70.25	69.60	69.90	67.55	66.86
Freq-spec [ 71 ]	49.90	99.90	50.50	49.90	50.30	99.70	50.10	50.00	48.00	50.60	50.10	50.90	50.40	50.40	50.30	51.70	51.40	50.40	50.00	55.45

TABLE V: Generalization performance (mAcc (%)) on next-generation generative benchmarks. Best and second-best results are in bold and underline .

Method	Train on remaining three				Avg
Method	DF	F2F	FS	NT	Avg
UniFD [ 25 ]	95.8	75.0	90.1	59.4	80.1
ForAda [ 26 ]	97.0	83.3	92.0	64.4	84.2
Ours	98.3	87.5	93.6	67.6	86.8

TABLE VI: Cross-manipulation performance in FF++ dataset (Frame-Level AUC (%)). Best results are in bold .

Method	L5	L10	L15	L20	L24
UniFD [ 25 ]	0.57	0.49	0.04	0.03	0.01
Ours	0.60	2.67	2.60	0.40	0.38

TABLE VII: Layer-wise COR comparison between UniFD and our CoRIT on the NT dataset. Best results are in bold .

Fig. 6: Comparison of layer-wise GSNR curves during training between UniFD and CoRIT on the NT dataset.

Fig. 7: Comparison of loss landscapes between UniFD and the proposed CoRIT on the NT dataset.

RSI	RRM	HRI	CDFv2	DFDC	facedancer	fsgan	Avg.
-	-	-	76.0	74.8	77.9	85.6	78.6
✓	-	-	86.9	81.6	79.2	87.3	83.8
✓	✓	-	88.5	82.7	84.4	91.2	86.7
-	-	✓	77.4	79.4	82.6	89.9	82.3
✓	✓	✓	89.1	84.5	87.8	93.8	88.8

TABLE VIII: Ablation study on proposed components (Frame-Level AUC (%)). Best results are in bold .

Region	$K$	Pooling	CDFv1	CDFv2	DFDCP	DFDC	DFD	Avg.
Random	3	Avg.	76.1	79.7	80.9	78.6	88.0	80.7
F	1	Avg.	88.6	86.6	84.9	82.5	91.2	86.8
Bd	1	Avg.	76.4	77.3	78.1	78.2	87.4	79.5
Bg	1	Avg.	67.1	61.7	70.5	70.0	77.0	69.3
F+Bd	2	Avg.	89.3	89.1	88.5	83.7	91.3	88.4
F+Bg	2	Avg.	89.9	89.1	87.3	83.2	91.3	88.2

TABLE IX: Ablation study on spatial regions and pooling strategies (Frame-Level AUC (%)). F, Bd, and Bg denote Foreground, Boundary, and Background, respectively.

K

is the number of regions. Best results are in bold .

Backbone	Method	CDFv1	CDFv2	DFDCP	DFDC	Avg.
IMG-L	UniFD [ 25 ]	53.0	54.0	63.2	61.4	57.9
IMG-L	Ours	66.3	65.0	75.0	67.6	68.5
CLIP-B	UniFD [ 25 ]	75.0	73.3	72.4	70.0	72.7
CLIP-B	Ours	81.0	77.2	80.8	76.5	78.9
CLIP-L	UniFD [ 25 ]	71.2	76.0	73.6	74.8	73.9
CLIP-L	Ours	90.9	89.1	88.8	84.5	88.3

TABLE X: Ablation study across different backbones: IMG-L (ImageNet-ViT-Large), CLIP-B (CLIP-ViT-Base), and CLIP-L (CLIP-ViT-Large). Best results are in bold .

$\alpha$	CDFv1	CDFv2	DFDCP	DFDC	DFD	Avg.
0	88.6	88.5	89.1	84.1	91.2	88.3
0.1	90.9	89.1	88.8	84.5	91.3	88.9
0.5	89.1	88.7	89.3	84.3	91.3	88.5
0.9	88.9	88.7	89.1	84.2	91.1	88.4
1.3	87.9	86.4	85.3	82.3	90.1	86.4

TABLE XI: Sensitivity analysis of the coherence coefficient

\alpha

on classical forensic benchmarks (Frame-Level AUC (%)). Best results are in bold .

Layer	CDFv1	CDFv2	DFDCP	DFDC	DFD	Avg.
10	86.6	86.8	85.3	84.1	91.5	86.9
16	88.4	88.2	87.8	84.4	91.9	88.1
18	90.9	89.1	88.4	84.5	91.5	88.9
20	90.3	88.9	85.5	85.0	91.2	88.2

TABLE XII: Ablation study on the selection of intermediate layers for feature fusion. The index indicates the encoder layer used for extracting shallow features. Best results are in bold .

Type	CDFv1	CDFv2	DFDCP	DFDC	DFD	Avg.
CrossAttn	91.5	88.4	85.6	83.9	90.4	88.0
Concat	90.9	89.1	88.4	84.5	91.5	88.9

TABLE XIII: Ablation study on fusion architecture. We compare feature concatenation (Concat) with a cross-attention mechanism (CrossAttn). Best results are in bold .

Fig. 8: Robustness evaluation under different perturbation levels: Gaussian Blur, Block-wise Distortion, Contrast Change, and JPEG Compression. Relative AUC Variation curves show AUC degradation as perturbation severity increases.

Optimization & Loss
Symbol	Definition
$\mathbf{w}_{t}$	Model parameters at optimization step $t$
$\mathcal{L}_{t}$	Expected NLL loss $\mathbb{E}_{\mathbf{x}\sim q}[-\log p_{\mathbf{w}_{t}}(\mathbf{x})]$
$\mathbf{g}(\mathbf{x},\mathbf{w}_{t})$	Stochastic gradient for a single sample $\mathbf{x}$
$\nabla\mathcal{L}_{t}$	Expected (population) gradient
$\mathbf{H}_{t}$	Expected Hessian matrix $\nabla^{2}\mathcal{L}(\mathbf{w}_{t})$

TABLE XIV: Summary of principal notation.

Fig. 9: Layer-wise visualization of the Region Refinement Mask (RRM) for sample 1. Each row displays the predicted mask components (Region 0, Region 1, and Region 2) at a specific network layer, accompanied by the input image and the SBI baseline for reference. The progression is visualized from the initialization stage ( Init ) through all 24 transformer layers ( Layers 1 to 24 ). Within each map, higher intensity areas indicate tokens retained after refinement, whereas darker regions correspond to suppressed outlier tokens.

Fig. 10: Layer-wise visualization of RRM for sample 2. The layout and color coding follow Fig. 9 .

Fig. 11: Layer-wise visualization of RRM for sample 2. The layout and color coding follow Fig. 9 .

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 240 papers

Aug 18, 2026

Environment-Invariant Subspace Learning for Generalizable Deepfake Detection

Cross-distribution generalization remains a critical bottleneck in deepfake detection. While recent efforts leverage the semantic priors of large-scale visual foundation models (VFMs), a noteworthy yet underexplored challenge remains: the susceptibility of these semantic priors to environmental interference from factors such as lighting and style. Crucially, this interference establishes spurious correlations between forgery cues and environmental patterns that severely limit generalization. To address this fundamental challenge, we propose an innovative Environment-Invariant Subspace Learning (EISL) framework. The core contribution of EISL is that it aims to disentangle features into orthogonal forgery-relevant invariant factors and environment-related residual factors via a learnable low-rank projection. To facilitate robust feature disentanglement, we also design an Environmental Intervention module that generates diverse and challenging intervention pairs, simulating out-of-distribution environmental shifts to guide the model toward discovering truly invariant forgery representations. Experiments across cross-dataset, cross-generator, whole-face synthesis, and corruption settings show consistent gains and competitive or leading performance against strong detectors, demonstrating improved robustness to unseen forgery types and environmental variations. This work provides a new perspective and a valuable exploration for understanding and tackling the generalization barriers of VFMs in deepfake detection.

Shenghao Chen, Hao Jia, Chen Li +3

Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology, Tianjin 300384, China · Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China

Deepfake Detection OOD Generalization Invariant Representation Learning

Ask PDF

Mar 25, 2026

Unleashing Vision-Language Semantics for Deepfake Video Detection

Recent Deepfake Video Detection (DFD) studies have demonstrated that pre-trained Vision-Language Models (VLMs) such as CLIP exhibit strong generalization capabilities in detecting artifacts across different identities. However, existing approaches focus on leveraging visual features only, overlooking their most distinctive strength -- the rich vision-language semantics embedded in the latent space. We propose VLAForge, a novel DFD framework that unleashes the potential of such cross-modal semantics to enhance model's discriminability in deepfake detection. This work i) enhances the visual perception of VLM through a ForgePerceiver, which acts as an independent learner to capture diverse, subtle forgery cues both granularly and holistically, while preserving the pretrained Vision-Language Alignment (VLA) knowledge, and ii) provides a complementary discriminative cue -- Identity-Aware VLA score, derived by coupling cross-modal semantics with the forgery cues learned by ForgePerceiver. Notably, the VLA score is augmented by an identity prior-informed text prompting to capture authenticity cues tailored to each identity, thereby enabling more discriminative cross-modal semantics. Comprehensive experiments on video DFD benchmarks, including classical face-swapping forgeries and recent full-face generation forgeries, demonstrate that our VLAForge substantially outperforms state-of-the-art methods at both frame and video levels. Code is available at https://github.com/mala-lab/VLAForge.

Jiawen Zhu, Yunqi Miao, Xueyi Zhang +2

Singapore Management University, Singapore · The University of Warwick, UK · Nanyang Technological University, Singapore +1

Deepfake Detection Vision-Language Alignment

Ask PDF

Mar 10, 2026

When Detectors Forget Forensics: Blocking Semantic Shortcuts for Generalizable AI-Generated Image Detection

The growing realism of generative models has blurred the boundary between real and synthetic content, posing significant challenges to reliable AI-generated image detection. Although large-scale pre-trained Vision Foundation Models have advanced detection capability, their generalization to images from unseen generation pipelines remains inadequate. In this paper, we identify, for the first time, a key failure mechanism, termed semantic fallback, wherein forensic fine-tuning fails to fully reshape the representation space. Consequently, the resulting representations remain organized along high-level semantic structures rather than manipulation-specific forensic cues. Building on this insight, we propose a Geometric Semantic Decoupling (GSD) framework, which explicitly suppresses semantically dominant directions, thereby promoting invariant forensic representations. Specifically, GSD leverages a frozen CLIP encoder to estimate the dominant semantic subspace via Singular Value Decomposition (SVD). It then suppresses the semantic components through a geometry-constrained formulation with the suppression strength adaptively modulated across samples and layers. We further introduce a mini-batch SVD approximation strategy that amortizes subspace estimation, achieving over a

15 \times

reduction in computational overhead while preserving effectiveness. Finally, considering practical scenarios spanning both large-scale and online evaluation, we develop three inference protocols, batch, per-sample, and reference-based inference, and demonstrate that they induce consistent semantic decoupling, yielding a stable forgery-oriented feature manifold.

Chao Shuai, Shaojing Fan, Chenlin Zou +6

State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · National University of Singapore, Singapore · Chongqing University of Posts and Telecommunications, Chongqing, China

AI-Generated Image Detection Shortcut Learning Representation Disentanglement

Ask PDF

Abstract

Deepfake Detection

Explore similar work

Environment-Invariant Subspace Learning for Generalizable Deepfake Detection

Unleashing Vision-Language Semantics for Deepfake Video Detection

When Detectors Forget Forensics: Blocking Semantic Shortcuts for Generalizable AI-Generated Image Detection