PairCLIP-SWA: Similarity-Guided Balancing and Stochastic Weight Averaging for Generalized Deepfake Image Detection with OpenCLIP

Md. Nakib; Sudipto Sarkar; A. J. A. Jubayer Talukder; Mohammad Ariful Haque

cs.CVOct 27, 2026

PairCLIP-SWA: Similarity-Guided Balancing and Stochastic Weight Averaging for Generalized Deepfake Image Detection with OpenCLIP

Authors: Md. Nakib, Sudipto Sarkar, A. J. A. Jubayer Talukder, Mohammad Ariful Haque

Organizations: Department of Electrical and Electronic Engineering, BUET, Dhaka 1205, Bangladesh

Abstract

Deepfake detectors are almost always deployed as the single checkpoint that scored best on a validation split, yet the reliability of that choice is rarely measured. We measure it with PairCLIP-SWA, a single-backbone detector that adapts an OpenCLIP ViT-L/14 visual tower by progressive block unfreezing on a class-balanced corpus built by CLIP-similarity-guided one-to-one subsampling, and averages the fine-tuning trajectory in weight space. Trained in-domain on the IEEE SP Cup 2025 (DF-Wild-Cup) benchmark under the same protocol as the strongest published three-backbone ensemble, the weight-averaged detector reaches 97.04% accuracy, 98.64% AU-ROC and a 4.07% equal error rate, against 94.46%, 97.60% and 5.00%. Its training log shows why averaging matters: over the averaged epochs the non-averaged checkpoint swings across 5.34 points of validation accuracy while its validation loss more than doubles, whereas the average is never worse and is up to 5.99 points better. On an OpenForensics-derived benchmark the same instability becomes a misranking: the checkpoint that validation accuracy selects (98.43%) scores 88.85% on a disjoint 10,905-image test partition, rejecting 21.1% of genuine photographs, while the terminal average, ranked last on validation, scores 94.76%. A study of twenty-two in-house configurations on the identical DF-Wild-Cup split, including preprocessing ablations, shows that CLIP pre-training rather than ensembling drives accuracy, that secondary face cropping costs about 34 points, and that frequency-domain inputs transfer poorly. We recommend per-class selection criteria measured off the early-stopping split.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of PairCLIP-SWA and of the analyses in this paper. Labels P1–P8 refer to the preprocessing steps of Table I . Stage 1 builds a balanced training corpus from the DF-Wild-Cup training partition by selecting, for each real image, its most similar unused fake in CLIP space. Stage 2 applies trace-preserving preprocessing, adapts an OpenCLIP ViT-L/14 tower with a widening unfreezing schedule, and averages the trajectory. Stage 3 evaluates the resulting checkpoints and analyses selection and representation geometry.

	Step	Setting	Rationale	Evidence
P1	Input region	Image as released; no secondary face detection or crop	Blending boundaries and face–context inconsistencies are among the most generator-agnostic cues [ 25 , 26 ] ; a tight crop removes them. DF-Wild-Cup images are already aligned by DeepfakeBench [ 8 ] , so a second detector pass compounds rather than removes variation, and fails silently on profile, occluded or low-resolution faces.	dlib crop: $56.05\%$ val., $\approx 34$ points below full frame (Table XI )
P2	Decoding	3-channel RGB (greyscale and alpha converted)	Matches the input contract of CLIP pre-training; greyscale captures occur in the wild (Fig. 6 ).	design
P3	Resampling	Bilinear resize to $224\times 224$ , no crop	Reproduces the pre-training token grid ( $16\times 16$ patches of $14$ pixels) so positional embeddings need no interpolation; resizing rather than cropping keeps peripheral boundary evidence; the aspect distortion is identical for both classes and cannot act as a label cue.	design
P4	Normalization	Eq. ( 1 ), CLIP $\boldsymbol{\mu},\boldsymbol{\sigma}$	The frozen patch embedding and at least $41.98\%$ of the tower cannot adapt to a different input scale; ImageNet statistics would shift the distribution the frozen prefix sees.	Section IV-B : a double rescale compresses inputs $255\times$
P5	Augmentation	Horizontal flip, $p=0.5$ , training only	Flipping is label-preserving and leaves pixel statistics untouched; colour jitter, blur, JPEG re-compression and random crops alter exactly the high-frequency and colour evidence of manipulation, or remove peripheral boundaries (P1, P3).	ViT+LoRA: augmentation raises the early peak but the gain is not retained (Table XI )
P6	Class balancing	One-to-one CLIP-similarity-guided subsample of the fake pool, without replacement (Section III-C )	One balanced corpus in one training stage, instead of oversampling (repeated reals), reweighting (majority domination), SMOTE (synthetic “reals” are invalid for authenticity) or five sequential stages [ 9 ] ; intended to reduce class-conditional content differences [ 28 ] .	Random undersampling, same backbone, different schedule: $93.36\%$ (Table XI )

TABLE I: Preprocessing and data-preparation pipeline. “Evidence” points to the ablation or measurement that bears on each choice; “design” marks choices motivated by argument only.

Component	Parameters	Share
Patch embedding ( $3{\times}14{\times}14{\to}1024$ )	$602{,}112$	$0.20\%$
Class token and positional embedding	$264{,}192$	$0.09\%$
Pre-/post-layer normalization	$4{,}096$	$<\!0.01\%$
Transformer blocks ( $24\times 12.60$ M)	$302{,}309{,}376$	$99.45\%$
Output projection ( $1024{\to}768$ )	$786{,}432$	$0.26\%$
Linear head ( $768{\to}2$ )	$1{,}538$	$<\!0.01\%$

TABLE II: Parameter budget and trainable fraction. The design cap

K_{\max}=14

was not reached within the

20

-epoch budget of the reported run, which ended at

k=11

blocks.

Corpus	Split	Images	Real/Fake	Role
DF-Wild- Cup	train ∗	$85{,}380$	$1{:}1$	training
DF-Wild- Cup	val.	$3{,}072$	$1548/1524$	selection, eval.
Open- Forensics	train	—	—	training
	val.	$39{,}428$	$19787/19641$	selection
	test	$10{,}905$	$5413/5492$	held-out test
∗ Stage-1 subsample of $262{,}160$ training images.

TABLE III: Splits and their roles. Only the OpenForensics test partition is untouched by both training and selection.

Setting	Value
Backbone	OpenCLIP ViT-L/14, OpenAI weights
Initialization	pre-trained tower, random linear head
Preprocessing	Table I (P1–P8)
Training corpus	$85{,}380$ -image $1{:}1$ subsample
Batch size	$32$ ( $16{+}16$ ), $2\times$ T4, DataParallel
Optimizer	AdamW, $\eta_{0}=10^{-4}$ , $\lambda=0$

TABLE IV: Configuration of the reported DF-Wild-Cup run.

Split ( $n$ )	Best regular	Best SWA	Final SWA
DF-Wild-Cup val. ( $3{,}072$ )	$95.67$	97.04	97.04
95% CI	[94.89, 96.34]	[96.38, 97.58]	[96.38, 97.58]
OpenForensics val. ( $39{,}428$ )	98.43	98.43	$97.67$
95% CI	[98.30, 98.55]	[98.30, 98.55]	[97.52, 97.81]
OpenForensics test ( $10{,}905$ )	$88.85$	$88.85$	94.76
95% CI	[88.24, 89.43]	[88.24, 89.43]	[94.33, 95.17]

TABLE V: Accuracy (%) of the three checkpoints on every evaluated split, at

\tau=0.5

, with

95\%

Wilson intervals. Best per split in bold.

			Training		Non-averaged val.		SWA val.
Epoch	Blocks	Trainable	Loss	Acc.	Loss	Acc.	Loss	Acc.	Gap
12	8	$33.15\%$	$0.1628$	$93.58$	$0.1677$	$95.70$	$0.1677$	$95.70$	—
13	8	$33.15\%$	$0.0802$	$97.01$	$0.1684$	$95.61$	$0.1684$	$95.61$	—
14	9	$37.30\%$	$0.1060$	$95.93$	$0.2552$	$91.05$	$0.1383$	$95.61$	$+4.56$
15	9	$37.30\%$	$0.0669$	$97.44$	$0.2005$	$93.88$	$0.1426$	$95.31$	$+1.43$
16	10	$41.44\%$	$0.0885$	$96.65$	$0.2344$	$92.22$	$0.1386$	$95.80$	$+3.58$

TABLE VI: Training log of the reported DF-Wild-Cup run over the averaged epochs (in-loop values). “Gap” is SWA minus non-averaged validation accuracy; the two coincide at epochs

12

–

13

because the first SWA update copies the iterate. Over epochs

12

–

19

the non-averaged validation accuracy spans

5.34

points (s.d.

2.13

) against

1.86

(s.d.

0.70

) for the average, with a mean change between consecutive epochs of

2.33

vs.

0.34

points. Accuracy before adaptation is

45.80\%

Fig. 2: Training dynamics of the reported DF-Wild-Cup run. (a) Validation accuracy of the non-averaged checkpoint oscillates across

5.34

points while the weight average stays within

1.86

points and is never below it; at epoch

17

the gap is

5.99

points. (b) Training loss falls by

78\%

while the non-averaged validation loss rises by a factor of

2.15

— overfitting of the adapted suffix — whereas the averaged model’s validation loss falls by

30\%

. Shading marks the number of unfrozen blocks.

Family	Model	Params	Acc. (%)
Spatial CNN	VGG-19 [ 66 ]	$144$ M	$62.25$
	ResNet-50 [ 67 ]	$25.6$ M	$69.00$
	MobileNet-V2	$3.5$ M	$78.12$
	ResNet-34 [ 67 ]	$21.3$ M	$80.32$
	Inception-V3	$26$ M	$84.38$
	ResNet-34 $+$ SE [ 32 ]	$21.3$ M	$88.90$

TABLE VII: Accuracy on the DF-Wild-Cup validation set (

n=3{,}072

). Baselines as reported in [ 9 ] ; ensemble rows are means over five seeds. All systems are trained on the DF-Wild-Cup training partition.

Metric	CAE-Net	PairCLIP-SWA
		$\tau=0.5$	$\tau^{\star}=0.680$ a
Accuracy (%)	$94.46$	97.04	$97.30$
95% CI	[93.60, 95.22]	[96.38, 97.58]	[96.66, 97.82]
Errors / $3{,}072$	$170$	91	$83$
AU-ROC (%) b	$97.60$	98.64
EER (%) b	$5.00$	4.07

TABLE VIII: Head-to-head on the DF-Wild-Cup validation set (

n=3{,}072

), fake class positive. CAE-Net values as published [ 9 ] . PairCLIP-SWA is the best-SWA checkpoint.

Fig. 3: Accuracy versus inference cost on DF-Wild-Cup; marker area is proportional to parameter count. Grey markers are the single models and ensembles of [ 9 ] .

Split	Checkpoint	Acc.	Prec.	Rec.	Spec.	$F_{1}$	MCC
Validation ( $39{,}428$ )	Best reg./SWA	98.43	$97.84$	99.03	$97.83$	98.43	0.969
Validation ( $39{,}428$ )	Final SWA	$97.67$	99.58	$95.73$	99.60	$97.61$	$0.954$
Test ( $10{,}905$ )	Best reg./SWA	$88.85$	$82.58$	98.67	$78.88$	$89.91$	$0.792$
Test ( $10{,}905$ )	Final SWA	94.76	94.97	$94.61$	94.92	94.79	0.895

TABLE IX: Complete metrics on OpenForensics at

\tau=0.5

, fake class positive, from the confusion matrices of Fig. 4 .

Fig. 4: OpenForensics confusion matrices for the validation-selected checkpoint (left) and the terminal average (right), on validation (top) and held-out test (bottom). Row percentages are per-class recall.

Fig. 5: (a) OpenForensics accuracy on validation (grey) and held-out test (blue). The validation-selected checkpoints lose

9.58

points in transfer; the terminal average loses

2.91

. (b) Test recall per class: the entire loss of the selected checkpoints is carried by the real class.

Panel of Figure 15 — Fig. 6: Ten randomly drawn OpenForensics test images under the validation-selected checkpoint (top) and the terminal average (bottom); same images, different order. Titles give label, prediction and softmax confidence; red borders mark errors. Both errors in the top panel are degraded genuine photographs — a heavily noised crowd scene ( $0.90$ ) and a portrait with an extreme magenta cast ( $0.99$ ) — rejected as fake. The average classifies both correctly, with its two lowest confidences ( $0.93$ and $0.62$ ).

Fig. 7: t-SNE embeddings of $1{,}000$ DF-Wild-Cup validation images ( $493$ fake, $507$ real) for the three checkpoints; green is real, red fake, crosses are centroids.

Corpus	Checkpoint	$\Delta$	$\sigma_{f}$	$\sigma_{r}$	$S$	Probe	Head
DF-Wild-Cup	Best regular	$43.90$	$10.70$	$10.90$	2.033	$96.00$	$95.67$
	Best SWA	$40.73$	$10.41$	$10.72$	$1.927$	97.20	97.04
	Final SWA	$39.08$	$10.32$	$10.07$	$1.917$	97.20	97.04
Open- Forensics	Best regular	$40.42$	$8.42$	$9.72$	$2.228$	$97.80$	$88.85^{\ddagger}$
	Best SWA	$40.42$	$8.42$	$9.72$	$2.228$	$97.80$	$88.85^{\ddagger}$
	Final SWA	$43.02$	$8.97$	$10.09$	2.257	$97.80$	$\textbf{94.76}^{\ddagger}$

TABLE X: Geometry and k-NN probe in t-SNE space. DF-Wild-Cup:

1{,}000

validation images; OpenForensics:

1{,}000

held-out images (

513

fake,

487

real). “Head” is classifier accuracy on the full split.

Variant	Train	Val.	Ctrl.
A. Input representation (P1)
Full released frame (reported run)	$98.48$	97.04	ref.
Face crop (dlib [ 69 ] ) $+$ OpenCLIP	$84.34$	$56.05$	—
FFT-domain input	$83.72$	$49.61$	—
Wavelet-coefficient input, 30 ep	$91.60$	$73.20$	—
B. Class balancing (P6), OpenCLIP ViT-L/14

TABLE XI: Ablations on the DF-Wild-Cup validation split (

n=3{,}072

). Rows marked ✓ differ from the reported configuration only in the stated factor; the others also differ in schedule or backbone and bound, rather than isolate, the effect. Train accuracy shows the memorization gap where logged.

Fig. 8: (a) In-house configurations evaluated on the identical DF-Wild-Cup validation split, coloured by backbone family; the dashed line is CAE-Net. (b) Train (open) and validation (filled) accuracy for alternative input representations and objectives; the gap is the memorization that each fails to convert into validation accuracy.

Configuration	Data	Init.	Val.
ImageNet-initialized CNNs
ResNet-50, gradual unfreezing, 40 ep	P	—	$71.70$
Xception, 5 ep (two runs)	S	—	$75.16$ , $75.79$
Xception, undersampled to $85{,}380$	U	—	$75.03$
Non-CLIP transformers
ViT, $0.69\%$ unfrozen (probe), 30 ep	P	$53.74$	$80.70$

TABLE XII: In-house configurations on the DF-Wild-Cup validation split, grouped by backbone. Data: S = SP-Cup training partition, P = Stage-1 subsample, U = random

1{:}1

undersample. “Init.” is accuracy with an untrained head. Schedules differ between rows.

Deepfake Detection

Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.

See all 246 papers

Sep 13, 2026

Lightweight Generalized DeepFake Face Detection with WAVIE: Wavelet Augmented Vision Intermediate Embeddings

Deepfake detection systems often exhibit significant performance degradation when deployed on unseen manipulation methods, limiting their reliability in real-world multimedia environments. This lack of generalization poses critical challenges for misinformation mitigation, digital forensics, and human-centric AI systems. Existing detectors perform well on the forgery methods they are trained on, but their accuracy drops sharply on unseen pipelines. To bridge this generalization gap, we propose WAVIE (Wavelet Augmented Vision Intermediate Embeddings), an end-to-end architecture that combines complementary spatial and frequency cues on top of a frozen CLIP backbone. WAVIE projects intermediate transformer embeddings through a lightweight learnable module, applies a three-level Daubechies-6 (db6) discrete wavelet transform (DWT), refines the low-frequency branch while preserving the high-frequency branch, reconstructs the feature via inverse DWT, and performs classification. Trained only on FaceForensics++, WAVIE achieves AUROC = 0.852 on Celeb-DF-v1, 0.852 on Celeb-DF-v2 and 0.831 on WildDeepFake (WDF) at the frame level, outperforming several state-of-the-art generalization baselines. Extensive ablation studies confirm the importance of both the wavelet module and the intermediate-feature aggregation for cross-dataset performance, highlighting the necessity of jointly leveraging spatial and frequency domains. These results position WAVIE as a strong baseline for deepfake detection in the wild.

Arya Pulkit, Aditya Ruhela, Akarshan Kapoor +1

School of Computing and Electrical Engineering, IIT Mandi, India · PHYTEC Messtechnik GmbH, Germany

Deepfake Detection

Ask PDF

Aug 28, 2026

FairReL: Deepfake Detection using Fairness-Aware Representation Learning

Although recent deepfake detectors achieve high overall accuracy, their errors remain unevenly distributed across demographic subgroups, with real faces from certain groups more often misclassified as fake. Existing fairness-aware detectors typically regularise the entire feature representation, without identifying or controlling the specific components that drive unfair predictions. Such coarse intervention can over-suppress useful forgery cues while leaving demographic structure in component-specific subspaces. To address this, we identify two subgroup-sensitive components: multi-scale spatial features, which encode local facial and forgery patterns, and fine-tuning-induced residual features, which adapt the backbone to the unfair training distribution. We propose FairReL, a fairness-aware representation-learning framework that targets both components with dedicated demographic supervision. FairReL uses an SVD-decomposed foundation-model backbone to isolate the fine-tuning-induced residual representation, and introduces two complementary losses. Group-Conditional Wavelet Decorrelation (GCWD) suppresses subgroup-imbalanced structure across spatial wavelet sub-bands, while Subspace-Localised Mean Alignment (SLMA) aligns subgroup means within each real/fake class in the residual representation. Experiments on FF++, Celeb-DF, DFD and DFDC show that, against the state-of-the-art fairness-aware detector, FairReL improves unseen-dataset AUC by 3.9% while reducing subgroup FPR disparity by 10.2%. Code is available at https://github.com/xiaoman89/FairReL .

Xiaoman Lu, Jiaqi Li, Shuntian Zheng +2

Department of Computer Science University of Warwick Coventry, United Kingdom · School of Computer Science University of Birmingham Birmingham, United Kingdom

Deepfake Detection Algorithmic Fairness Representation Learning

Ask PDF

Sep 7, 2026

Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection

The growing realism and accessibility of manipulated and generated faces threaten the trustworthiness of digital media. To detect such forgeries, deepfake detectors based on vision foundation models have shown promising performance, but they typically rely on a single pretrained representation and are prone to overfitting to particular training distributions. To improve generalization to unseen forgeries, we propose UCF-Net, an uncertainty-aware cascaded fusion network that harnesses CLIP's language-aligned semantic priors and DINO's self-supervised visual-structure priors. UCF-Net extracts hierarchical features across Transformer depths, uses layer-wise expert aggregation to adaptively combine each encoder's multi-level cues, and performs weighted fusion of the resulting representations based on entropy-derived uncertainty. We further consolidate public deepfake datasets into a unified benchmark of approximately 4M images and construct a separate cross-generator evaluation set with over 8K face images from eight recent generators. On the unified benchmark, UCF-Net achieves the best mean AUC among the evaluated methods in both in-domain and cross-domain evaluations. On the cross-generator set, it adapts effectively with limited target-domain data, although zero-shot transfer remains challenging.

Xuechao Zou, Yi Zhou, Kai Li +4

Beijing Jiaotong University · Tsinghua University · Ant Group

Deepfake Detection Multimodal Fusion

Ask PDF

Abstract

Deepfake Detection

Explore similar work

Lightweight Generalized DeepFake Face Detection with WAVIE: Wavelet Augmented Vision Intermediate Embeddings

FairReL: Deepfake Detection using Fairness-Aware Representation Learning

Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection