cs.CVMar 2, 2026
SaveDOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis
Organizations: University of North Carolina at Chapel Hill · University of California, Irvine · Washington University in St. Louis · Duke University ·
Abstract
We present DOCFORGE-BENCH, the first unified zero-shot benchmark for document forgery detection, evaluating 14 methods across eight datasets spanning text tampering, receipt forgery, and identity document manipulation. Unlike fine-tuning-oriented evaluations such as ForensicHub [Du et al., 2025], DOCFORGE-BENCH applies all methods with their published pretrained weights and no domain adaptation -- a deliberate design choice that reflects the realistic deployment scenario where practitioners lack labeled document training data. Our central finding is a pervasive calibration failure invisible under single-threshold protocols: methods achieve moderate Pixel-AUC (>=0.76) yet near-zero Pixel-F1. This AUC-F1 gap is not a discrimination failure but a score-distribution shift: tampered regions occupy only 0.27-4.17% of pixels in document images -- an order of magnitude less than in natural image benchmarks -- making the standard tau=0.5 threshold catastrophically miscalibrated. Oracle-F1 is 2-10x higher than fixed-threshold Pixel-F1, confirming that calibration, not representation, is the bottleneck. A controlled calibration experiment validates this: adapting a single threshold on N=10 domain images recovers 39-55% of the Oracle-F1 gap, demonstrating that threshold adaptation -- not retraining -- is the key missing step for practical deployment. Overall, no evaluated method works reliably out-of-the-box on diverse document types, underscoring that document forgery detection remains an unsolved problem. We further note that all eight datasets predate the era of generative AI editing; benchmarks covering diffusion- and LLM-based document forgeries represent a critical open gap on the modern attack surface.
Figures & tables
| Dataset | Domain | Year | Dataset Size | Eval. Images | Forgery Types | GT Masks | Used In |
| DocTamper | Document | 2023 | 170K | 1,000 (sampled) † | Text replace/insert/delete | ✓ | Both |
| T-SROIE | Receipt | 2022 | 360 | 360 | Text tampering | ✓ | Both |
| RealTextManip. | Real-world | 2022 | 9K | 1,000 (sampled) † | Text manipulation | ✓ | Both |
| Tampered-IC13 | Scene text | 2015 | 233 | 188 tampered | Text replacement | BBox ∗ | Both |
| Receipt Forgery | Receipt | 2023 | 988 | 35 forged | Price/qty forgery | BBox ∗ | Both |
| MixTamper | Document | 2024 | 30K | 1,000 (sampled) † | Copy-move, splice, gen., smear, erase | ✓(RGB) | Both |
Table 1: Overview of the eight document datasets evaluated in DocForge-Bench . The Eval. Images column shows the number of images used in our evaluation (see footnote for sampling details). ∗ Receipt Forgery and Tampered-IC13 provide bounding-box annotations; pixel masks are rasterized for localization evaluation.
| # | Method | Year | Venue | Category | Framework | I/O |
| General image forensics | ||||||
| 1 | TruFor | 2023 | CVPR | Image forensics | PyTorch | Image mask + score |
| 2 | ManTraNet | 2019 | CVPR | Image forensics | Keras/TF | Image heatmap |
| 3 | MVSS-Net | 2021 | ICCV | Image forensics | PyTorch | Image mask + edge |
| 4 | CAT-Net | 2022 | IJCV | JPEG forensics | PyTorch | JPEG mask |
| 5 | PSCC-Net | 2022 | TCSVT | Image forensics | PyTorch | Image mask + score |
Table 2: Overview of methods catalogued in DocForge-Bench .
Figure 1: Pixel-F1 (left) and Pixel-AUC (right) for all 14 evaluated methods—document-specific (above separator) and general forensic (below)—across eight document datasets. Pixel-AUC is consistently moderate to high while Pixel-F1 at fixed remains near zero for most (method, dataset) pairs. The pervasive AUC–F1 gap confirms calibration failure—not discriminative failure—as the dominant bottleneck: methods correctly rank tampered pixels above authentic ones but cannot identify a usable decision threshold in the document domain. ForensicHub methods (FFDN, CAFTB, TIFDM) achieve AUC 0.90 on multiple datasets where Pixel-F1 is below 0.05, confirming the calibration gap is not resolved by document-specific training. Appendix Fig. 6 shows Pixel-IoU and Oracle F1.
| Method | Mean F1 | Mean AUC | |
|---|---|---|---|
| Document-specific | |||
| CAFTB-Net | 0.34 | 0.33 | 0.85 |
| DocTamper | 0.30 | 0.31 | 0.81 |
| ASCFormer | 0.27 | 0.21 | 0.63 |
| TIFDM | 0.24 | 0.22 | 0.83 |
| ADCD-Net | 0.23 | 0.23 | 0.79 |
Table 3: Per-method summary: mean Pixel-F1 and Pixel-AUC across all seven datasets. = standard deviation of Pixel-F1 across datasets (consistency).
Figure 2: Mean cross-domain Pixel-F1 (left) and Pixel-AUC (right) for all 14 methods across the four cross-domain datasets (RealTextManipulation, Tampered-IC13, ReceiptForgery, FSTS-1.5k). Error bars show standard deviation. The dashed vertical line marks the mean across all general methods. Despite document-specific training, CAFTB-Net is the only doc-specific method that clearly outperforms both TruFor and CAT-Net on F1; on AUC, the two method families overlap substantially, confirming that calibration—not feature discrimination—distinguishes the groups.
Figure 5: Calibration failure across all 14 methods: Pixel-F1 @ (dark blue), Oracle F1 at best threshold (light blue), and Pixel-AUC (green), all averaged across the eight document datasets. The consistent ordering AUC Oracle F1 Pixel-F1 holds for every method, confirming that score-distribution shift—not feature discrimination—is the primary bottleneck. A dashed vertical line separates document-specific (left) from general methods (right).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Pixel-IoU (left) and Oracle F1 (right) for all 14 evaluated methods across eight document datasets. Pixel-IoU tracks Pixel-F1 closely ( ) and is included for comparison with prior work. Oracle F1 is the best achievable F1 at any threshold per image; the large gap between Oracle F1 and the fixed-threshold Pixel-F1 in Fig. 1 quantifies calibration error across the document domain.
| Method | Type | DocTamper | T-SROIE | RTM | IC13 | Receipt | FantasyID | MixTamper | FSTS-1.5k |
|---|---|---|---|---|---|---|---|---|---|
| DocTamper | Doc | 0.914 | 0.045 | 0.035 | 0.185 | 0.015 | 0.107 | 0.604 | 0.475 |
| DTD (FH) | Doc | 0.682 | 0.021 | 0.024 | 0.140 | 0.003 | 0.036 | 0.106 | 0.242 |
| FFDN | Doc | 0.736 | 0.043 | 0.025 | 0.156 | 0.019 | 0.068 | 0.072 | 0.219 |
| CAFTB-Net | Doc | 0.893 | 0.048 | 0.039 | 0.170 | 0.023 | 0.213 | 0.676 | 0.671 |
| TIFDM | Doc | 0.742 ‡ | 0.046 | 0.042 | 0.164 | 0.019 | 0.300 | 0.312 | 0.320 |
| ASCFormer | Doc | 0.180 | 0.779 | 0.265 | 0.022 | 0.260 | 0.222 | 0.314 | 0.133 |
Table 4: Document-specific ( ) vs. general forensic methods on 8 document datasets (zero-shot). Pixel-F1 scores (higher is better); best per column in bold . Full metrics (AUC, IoU) available in the benchmark repository. ‡ TIFDM DocTamper result may be in-domain; see Appendix B .
| Method | Type | DocTamper | T-SROIE | RTM | IC13 | Receipt | FantasyID | FSTS-1.5k | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AUC | Opt-F1 | AUC | Opt-F1 | AUC | Opt-F1 | AUC | Opt-F1 | AUC | Opt-F1 | AUC | Opt-F1 | AUC | Opt-F1 | ||
| DocTamper | Doc | 0.995 | 0.942 | 0.811 | 0.078 | 0.660 | 0.068 | 0.748 | 0.312 | 0.731 | 0.037 | 0.754 | 0.167 | 0.877 | 0.630 |
| DTD (FH) | Doc | 0.988 | 0.786 | 0.872 | 0.106 | 0.730 | 0.092 | 0.857 | 0.335 | 0.813 | 0.046 | 0.853 | 0.218 | 0.730 | 0.347 |
| FFDN | Doc | 0.991 | 0.806 | 0.904 | 0.155 | 0.741 | 0.090 | 0.895 | 0.352 | 0.842 | 0.088 | 0.880 | 0.307 | 0.736 | 0.337 |
| CAFTB-Net | Doc | 0.995 | 0.906 | 0.810 | 0.060 | 0.704 | 0.050 | 0.845 | 0.206 | 0.791 | 0.031 | 0.796 | 0.246 | 0.885 | 0.695 |
| TIFDM | Doc | 0.994 | 0.813 | 0.806 | 0.076 | 0.719 | 0.071 | 0.788 | 0.277 | 0.762 | 0.037 | 0.879 | 0.348 | 0.792 | 0.417 |
Table 5: Pixel-AUC (threshold-independent) and Oracle F1 for all evaluated methods. High AUC with low F1 reveals calibration failure independent of domain specificity.
Figure 7: Distribution of Pixel-F1 across eight datasets per method, shown as horizontal box plots sorted by median (descending). Individual dataset scores are overlaid as jittered points. Red methods are document-specific; blue are general forensics. The wide interquartile ranges confirm that no method generalises uniformly: a method can have the highest median while still scoring near zero on at least two datasets.
Figure 8: Pixel-F1 performance profiles for all 14 methods across eight datasets sorted by best-method F1 (descending difficulty left to right). Document-specific methods are shown as dashed lines; general methods as solid lines. The crossing of lines across datasets confirms that no single method dominates uniformly.
| Method | Val. Dataset | Ours | Ours | Ours | Published | Source | Notes |
|---|---|---|---|---|---|---|---|
| Pixel-F1 | AUC | Opt-F1 | (metric = value) | ||||
| TruFor | CASIAv1 | 0.687 | 0.946 | 0.789 | AUC = 0.793; Opt-F1 = 0.715 | Guillaro et al. [2023] (CVPR 2023) | Our AUC and Opt-F1 exceed the paper; same split, higher results consistent with implementation correctness. |
| MVSS-Net | CASIAv1 | 0.451 | 0.845 | 0.575 | AUC = 0.862 ‡ | Chen et al. [2021] (ICCV 2021) | 2 pp gap attributable to CASIAv1 vs. CASIAv1 + split difference; result is consistent. |
| PSCC-Net | IMD2020 | 0.132 | 0.708 | 0.326 | Pixel-F1 = 0.203 (pretrained) | Liu et al. [2022] (TCSVT 2022) | Both use frozen pretrained weights; 7 pp gap within the expected range for random sampling differences and test-set version. |
| IML-ViT | CASIAv1 | 0.694 | 0.942 | 0.776 | AUC = 0.836; Opt-F1 = 0.761 | Ma et al. [2023] (arXiv 2023) | Our numbers exceed the paper on both metrics, confirming correct implementation. |
| CAT-Net | CASIAv1 | 0.710 | 0.959 | 0.809 | AUC = 0.976; Opt-F1 = 0.781 | Kwon et al. [2022] (IJCV 2022) | 1.7 pp AUC gap; Opt-F1 slightly higher. Consistent with test-split variation reported across reproductions. |
Table 6: Validation of general image forensic methods against published benchmarks. Metrics computed on official test splits; CASIAv1 uses the standard (non-augmented) split. “Published” column reports the closest metric from the original paper or a reference reproduction; ‡ denotes results on CASIAv1 + (augmented) rather than CASIAv1.
| Method | Val. Dataset | Ours | Published | Source | Protocol | Notes |
|---|---|---|---|---|---|---|
| Pixel-F1 | Pixel-F1 | |||||
| DocTamper (model) | DocTamper test | 0.914 | 0.914 | Qu et al. [2023a] (CVPR 2023) | In-domain pretrained | Exact match to published figure. |
| ADCD-Net | DocTamper test | 0.762 | 0.806 | Wong et al. [2025] (ICCV 2025) | In-domain pretrained | 4 pp gap; attributable to our 1,000-image sample cap vs. full test set evaluation. |
| FFDN | DocTamper test | 0.736 | 0.821 | Du et al. [2025] (NeurIPS 2025) | Fine-tuned on DocTamper | Our frozen weights trail fine-tuned result by 8.5 pp, which is expected. |
| DTD | DocTamper test | 0.682 | 0.803 | Du et al. [2025] (NeurIPS 2025) | Fine-tuned on DocTamper | Our frozen weights trail fine-tuned result by 12 pp, which is expected. |
| CAFTB-Net | DocTamper test | 0.893 | 0.328 | Du et al. [2025] (NeurIPS 2025) | Retrained from scratch on DocTamper | Our result exceeds the ForensicHub baseline because the original CAFTB-Net weights were already pretrained on DocTamper; ForensicHub trains from scratch. |
Table 7: Validation of document-specific methods against published benchmarks. All evaluations use frozen pretrained weights (zero-shot protocol). “Published” column reports the closest comparable number from the source paper; where the source uses fine-tuning or retraining, this is noted explicitly.