Beyond Frame-Level Accuracy: Analyzing Video-Level Stability in Transformer-Based Deepfake Forensics
Organizations: Department of Mathematics, Jashore University of Science and Technology, Jashore 7408, Bangladesh · School of Computing, Data and Mathematical Sciences, University of Stirling, Stirling, United Kingdom
Abstract
Deepfake videos generated by advanced generative models pose serious threats to digital security, misinformation control, and media forensics. Although numerous detection methods have been proposed, their reported performances are often based on heterogeneous preprocessing pipelines, inconsistent data splits, and model-specific training protocols, hindering fair architectural comparison. This work presents a controlled and reproducible comparative study of convolutional neural networks (ResNet-18 and EfficientNet-B0) and a transformer-based architecture (ViT-B/16) within a unified, face-centric deepfake detection framework. All models are trained using identical preprocessing, augmentation, optimization settings, and data splits on Celeb-DF v2 and FaceForensics++, and results are validated across three independent random seeds with bootstrapped confidence intervals. Experimental results show that while CNN-based models perform competitively at the frame level, their performance degrades after video-level aggregation. In contrast, ViT-B/16 demonstrates more stable video-level performance, achieving 92.8% [95% CI: 91.7--94.0] video-level accuracy and an AUC of 0.950 [95% CI: 0.934--0.964] on Celeb-DF v2 (3-seed bootstrapped). Cross-dataset evaluation across the same three random seeds reveals a large, asymmetric generalization gap (AUC 0.79 FF++Celeb-DF vs. 0.64 Celeb-DFFF++), indicating that in-domain video-level stability does not transfer to unseen manipulation domains. These findings indicate that global self-attention improves robustness and video-level stability for deepfake detection within a given data domain, while highlighting the need for further work on cross-domain generalization.
Figures & tables
| Dataset | Split | Videos | Real frames | Fake frames | Total frames | Fake ratio |
| FF++ | Train | 5,600 | 71,545 | 136,784 | 208,329 | 65.7% |
| FF++ | Val | 700 | 9,078 | 17,065 | 26,143 | 65.3% |
| FF++ | Test | 700 | 8,157 | 18,164 | 26,321 | 69.0% |
| Celeb-DF v2 | Train | 5,223 | 6,901 | 45,320 | 52,221 | 86.8% |
| Celeb-DF v2 | Val | 653 | 1,040 | 5,490 | 6,530 | 84.1% |
| Celeb-DF v2 | Test | 653 | 950 | 5,580 | 6,530 | 85.5% |
| Item | Value |
| Hardware | 1 Tesla T4 GPU, 14.56 GB VRAM |
| Software | PyTorch 2.10.0+cu128, timm 1.0.26 |
| Random seeds | 42, 123, 2024 (train + in-domain evaluation + both cross-dataset directions) |
| Data split | 80%/10%/10% (train/validation/test), split at video level |
| Image resolution | |
| Batch size | 16 (micro-batch) 2 (gradient accumulation) = 32 effective |
| Parameter | Specification |
| Patch Size | |
| Embedding Dimension | 768 |
| Layers | 12 |
| Attention Heads | 12 |
| Dropout | 0.1 |
| Pretraining | ImageNet-1k |
| Model | Dataset | Params | Acc (%) | F1 | AUC | FPS |
| EfficientNet-B0 | Celeb-DF | 5.30M | 60.71 | 0.670 | 0.763 | 18.3 |
| ResNet-18 | Celeb-DF | 11.18M | 88.04 | 0.907 | 0.949 | 270.3 |
| ViT-B/16 | Celeb-DF | 85.8M | 92.83 [91.68, 93.98] | 0.958 [0.951, 0.965] | 0.950 [0.934, 0.964] | 18.3 |
| EfficientNet-B0 | FF++ | 5.30M | 80.22 | 0.895 | 0.913 | 5.09 |
| ResNet-18 | FF++ | 11.18M | 82.00 | 0.837 | 0.955 | 15.8 |
| ViT-B/16 | FF++ | 85.8M | 77.41 [75.57, 79.10] | 0.835 [0.821, 0.849] | 0.840 [0.821, 0.859] | 18.3 |
| Dataset | Latency (ms) | FPS | Peak GPU Mem., Inference (GB) | Peak GPU Mem., Training (GB) |
| Celeb-DF | 13.57 | 73.70 | 0.668 | 2.835–2.839 |
| FF++ | 11.80 | 84.75 | 0.668 | 2.837–2.839 |
| Dataset | Seed | Frame AUC | Video AUC | Video Acc | Video F1 |
| FF++ | 42 | 0.8140 | 0.8416 | 0.7800 | 0.8392 |
| FF++ | 123 | 0.8102 | 0.8409 | 0.7757 | 0.8370 |
| FF++ | 2024 | 0.8074 | 0.8389 | 0.7671 | 0.8304 |
| Celeb-DF v2 | 42 | 0.9276 | 0.9539 | 0.9326 | 0.9606 |
| Celeb-DF v2 | 123 | 0.9203 | 0.9477 | 0.9280 | 0.9577 |
| Celeb-DF v2 | 2024 | 0.9229 | 0.9486 | 0.9250 | 0.9557 |
| Dataset | N videos | Acc (mean [95% CI]) | F1 (mean [95% CI]) | AUC (mean [95% CI]) |
| FF++ | 2100 | 0.7741 [0.7557, 0.7910] | 0.8353 [0.8207, 0.8489] | 0.8404 [0.8206, 0.8586] |
| Celeb-DF v2 | 1959 | 0.9283 [0.9168, 0.9398] | 0.9578 [0.9507, 0.9647] | 0.9499 [0.9340, 0.9643] |
| Direction | Seed | Frame AUC | Video AUC | Video Acc | Video F1 |
| FF++ Celeb-DF | 42 | 0.7352 | 0.7752 | 0.6570 | 0.7586 |
| FF++ Celeb-DF | 123 | 0.7531 | 0.7956 | 0.6708 | 0.7705 |
| FF++ Celeb-DF | 2024 | 0.7571 | 0.7967 | 0.6815 | 0.7783 |
| Celeb-DF FF++ | 42 | 0.6338 | 0.6475 | 0.3929 | 0.3222 |
| Celeb-DF FF++ | 123 | 0.6189 | 0.6301 | 0.4000 | 0.3375 |
| Celeb-DF FF++ | 2024 | 0.6269 | 0.6409 | 0.3700 | 0.2782 |
| Direction | N videos | Accuracy | F1 | AUC |
| FF++ Celeb-DF v2 | 1959 | 0.6691 [0.6483, 0.6901] | 0.7685 [0.7505, 0.7861] | 0.7881 [0.7617, 0.8133] |
| Celeb-DF v2 FF++ | 2100 | 0.3877 [0.3671, 0.4100] | 0.3133 [0.2851, 0.3407] | 0.6386 [0.6141, 0.6642] |
| Model | Epochs | Final Val Acc | Final Val AUC | Outcome |
| Mean-pool (baseline) | 10 | 0.749 (EMA, ep. 10) | 0.821 (EMA, ep. 10) | Learns normally |
| GRU temporal head | 8 | 0.697 (flat) | 0.50–0.52 (chance) | Collapsed to majority class |
| Paper | Model Type | Backbone / Method | AUC (%) |
| Chen et al. (2021) [ 1 ] | Lightweight ML | DefakeHop (Subspace Learning) | 97.45 |
| Prashnani et al. (2022) [ 14 ] | 3D CNN | PhaseForensics (Phase-based Motion Features) | 99.37 |
| Lacerda et al. (2022) [ 7 ] | CNN | EfficientNet-B4 | 95.52 |
| Xu et al. (2023) [ 20 ] | Multi-channel + Attention | MCX-API Framework | 98.48 |
| Rössler et al. (2019) [ 16 ] | CNN | XceptionNet | 99.26 |
| Nguyen et al. (2019) [ 11 ] | Capsule Network | Capsule-Forensics | 96.60 |
| Paper | Model Type | Backbone / Method | AUC (%) |
| Omar et al. (2024) [ 13 ] | CNN + Attention | Self-Attention CNN | 98.30 |
| Zhao et al. (2021) [ 22 ] | Multi-Attention CNN | Multi-Attentional Deepfake Detection | 99.40 |
| Li et al. (2020) [ 10 ] | CNN | Celeb-DF Benchmark (XceptionNet) | 73.40 |
| Wang et al. (2020) [ 19 ] | CNN | CNN-based Detector (Generalization Study) | 96.20 |
| Chen et al. (2021) [ 1 ] | Lightweight ML | DefakeHop (Subspace Learning) | 90.56 |
| Prashnani et al. (2022) [ 14 ] | 3D CNN | PhaseForensics (Phase-based Motion Features) | 92.40 |
Deepfake Detection
Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.
See all 264 papers