AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents

Jiaqi Wu; Yuchen Zhou; Muduo Xu; Zisheng Liang; Simiao Ren; Jiayu Xue; Meige Yang; Siying Chen

cs.CVFeb 24, 2026

AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents

Authors: Jiaqi Wu, Yuchen Zhou, Muduo Xu, Zisheng Liang, Simiao Ren, Jiayu Xue, Meige Yang, Siying Chen,

Organizations: Duke University · New York University · University of North Carolina · University of Southern California

Abstract

We present AIForge-Doc, the first dedicated benchmark targeting exclusively diffusion-model-based inpainting in financial and form documents with pixel-level annotation. Existing document forgery datasets rely on traditional digital editing tools (e.g., Adobe Photoshop, GIMP), creating a critical gap: state-of-the-art detectors are blind to the rapidly growing threat of AI-forged document fraud. AIForge-Doc addresses this gap by systematically forging numeric fields in real-world receipt and form images using two AI inpainting APIs -- Gemini 2.5 Flash Image and Ideogram v2 Edit -- yielding 4,061 forged images from four public document datasets (CORD, WildReceipt, SROIE, XFUND) across nine languages, annotated with pixel-precise tampered-region masks in DocTamper-compatible format. We benchmark three representative detectors -- TruFor, DocTamper, and a zero-shot GPT-4o judge -- and find that all existing methods degrade substantially: TruFor achieves AUC=0.751 (zero-shot, out-of-distribution) vs. AUC=0.96 on NIST16; DocTamper achieves AUC=0.563 vs. AUC=0.98 in-distribution, with pixel-level IoU=0.020; GPT-4o achieves only 0.509 -- essentially at chance -- confirming that AI-forged values are indistinguishable to automated detectors and VLMs. These results demonstrate that AIForge-Doc represents a qualitatively new and unsolved challenge for document forensics.

Figures & tables

Figure 1: AIForge-Doc: AI-inpainted document forgeries pass visual inspection and are difficult to distinguish from authentic documents. Each row: authentic document with target field highlighted (left), AI-forged version (center), pixel-precise ground-truth mask (right). From top: CORD receipt (Ideogram v2 Edit), WildReceipt (Gemini 2.5 Flash Image), SROIE receipt (Gemini), XFUND multilingual form (Ideogram v2 Edit). The tampered region—a single numeric field—comprises a median of 0.9% of image pixels, yet contains the forensically critical edit.

Panel of Figure 2 — Figure 2: AIForge-Doc generation pipeline. Top : the overall dataset creation workflow, from source datasets through field selection and tool assignment (informed by a 320-trial prompt ablation study) to the final DocTamper-compatible dataset. Bottom : the per-image context-window inpainting technique—starting from a source document (a), we expand a context crop (b), create a binary inpainting mask (c), feed the crop and mask to the AI API (d), and paste only the field region back into the full image (e) to produce the forged output and pixel-precise ground-truth mask (f).

Figure 3: Prompt ablation: 5 representative outputs per rejected tool on WildReceipt image 000002699 ( Prod_item_key , “HULAHAWAIIANT1” $\to$ “HULAHAWAIIANT4”). Each row shows the Gemini 2.5 Flash reference (green border) alongside outputs from four prompt strategies spanning minimal, OCR-focused, step-by-step, and color-aware prompts. FLUX Fill Pro renders plausible digit shapes but consistently wrong values; GPT-Image-1 produces a black N/A patch or blurry wrong-font text at 512 px; SD 3.5 Medium renders prompt text literally or produces garbled symbols; SD 1.5 Inpainting yields illegible blurred patches. No prompt strategy succeeds for any rejected tool. Full 20-prompt-variant grids are in Figs. 6 , 7 , 8 and 9 (Appendix B ).

Dataset	Doc Type	Lang	Images	Fields
CORD v2 [ 20 ]	Receipt	id	1,000	price, total
WildReceipt [ 34 ]	Receipt	en	1,696	tel, addr, price
SROIE [ 10 ]	Receipt	en	946	total, date
XFUND [ 38 ]	Form	multi	419	answer (num)
Total	—	—	4,061	—

Table 1: Source datasets in AIForge-Doc.

Figure 4: Distribution of tampered pixel fraction across 4,061 AIForge-Doc images. Median: 0.92% (vertical blue line); IQR: [0.35%, 1.55%]. Over 99% of pixels in each image are unmodified—the tampered region is a small, localized field bbox.

Method	Image-Level			Pixel-Level
Method	AUC $\uparrow$	AP $\uparrow$	Acc@opt $\uparrow$	IoU $\uparrow$	F1 $\uparrow$	AUC ${}_{\text{px}}$ $\uparrow$
Random Baseline	0.500	–	0.500	–	–	0.500
TruFor [ 7 ]	0.751 [.726,.776]	0.709	0.731	0.358	0.434	0.916
DocTamper [ 22 ]	0.563 [.535,.591]	0.564	0.558	0.020	0.030	0.675
GPT-4o (zero-shot)	0.509 [.481,.537] §	0.516	0.535	–	–	– ‡
DocTamper (in-distribution) †	0.98	0.97	0.94	0.71	0.74	0.95

Table 2: Baseline detector performance on AIForge-Doc test set (zero-shot, out-of-distribution). All models are evaluated zero-shot (no fine-tuning on AIForge-Doc). Best results per column are bolded . “–” indicates metric not applicable or not comparable (see footnote). A random detector achieves AUC = 0.50. All three detectors use the full 1,624-image test split (812 forged + 812 authentic).

Inpainting Tool	AUC (img)	95% CI	# Test imgs
Gemini 2.5 Flash Image	0.778	[0.754, 0.802]	727
Ideogram v2 Edit	0.521	[0.447, 0.597] ∗	85
Overall	0.751	[0.726, 0.776]	812
∗ CI width 0.150 spans 0.50; interpret with caution (see text).

Table 3: Per-tool breakdown of TruFor image-level AUC on AIForge-Doc test set. Lower = harder to detect.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: NoisePrint++ heatmaps for two AIForge-Doc examples. Each row shows: original document crop (left), AI-forged version (center-left), ground-truth mask (center-right), and TruFor NoisePrint++ heatmap (right; hot colormap, 0 = authentic, 1 = forged). Top row (Gemini 2.5 Flash Image, TruFor score = 1.000) : TruFor assigns high confidence to this Gemini forgery; the heatmap shows elevated response near the tampered bbox (cyan rectangle), suggesting residual noise-level artifacts from Gemini’s generation process. Bottom row (Ideogram v2 Edit, TruFor score = 0.505) : TruFor is near-random on this Ideogram forgery; the heatmap is diffuse and uniform, indicating that Ideogram’s inpainting leaves no detectable sensor-level signature—consistent with its per-tool AUC of 0.521 (CI spans 0.50, Table 3 ). The per-tool gap ( $\Delta$ AUC = 0.257) shows that different AI generators produce forgeries of qualitatively different detectability.

Figure 6: FLUX Fill Pro: 20 prompt variants on WildReceipt 000002699. The model generates plausible digit shapes but consistently wrong numeric values across all prompt strategies, including chain-of-thought and typography-expert prompts.

Figure 7: GPT-Image-1 (DALL-E-2 fallback): 20 prompt variants on WildReceipt 000002699. Occasionally produces correct numerals but with wrong font weight and 512 $\times$ 512 px blurring artifacts that fail quality review under all prompt strategies.

Figure 8: SD 3.5 Medium: 20 prompt variants on WildReceipt 000002699. Renders prompt text literally (e.g., “Replace Text,” “Restore Beautifully”) or produces garbled characters and emoji, ignoring the inpainting task entirely.

Figure 9: SD 1.5 Inpainting: 20 prompt variants on WildReceipt 000002699. 512 $\times$ 512 native resolution yields uniform blurry patches regardless of prompt formulation; text is completely illegible at document scale.

Text Forgery Detection

See all 11 papers

DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis
Mar 2, 2026Zengqi Zhao, Weidi Xia, En Wei +7Text Forgery Detection Digital Forensics
University of North Carolina at Chapel Hill · University of California, Irvine · Washington University in St. Louis +4
When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documents
Apr 28, 2026Jiaqi Wu, Yuchen Zhou, Dennis Tsang Ng +5Image Editing Evaluation Image Forgery Detection
arXiv:2604.25213v1
GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics
Mar 12, 2026Yan Zhang, Simiao Ren, Ankit Raj +6Digital Image Forensics AI-Generated Text Detection

Mar 2, 2026J/K move · Enter open · S save

DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis

Zengqi Zhao, Weidi Xia, En Wei, Yan Zhang+6

University of North Carolina at Chapel Hill · University of California, Irvine · Washington University in St. Louis+4

Text Forgery Detection Digital Forensics Digital Image Forensics Model Calibration

Open paper AskPDF arXiv

We present DOCFORGE-BENCH, the first unified zero-shot benchmark for document forgery detection, evaluating 14 methods across eight datasets spanning text tampering, receipt forgery, and identity document manipulation. Unlike fine-tuning-oriented evaluations such as ForensicHub [Du et al., 2025], DOCFORGE-BENCH applies all methods with their published pretrained weights and no domain adaptation -- a deliberate design choice that reflects the realistic deployment scenario where practitioners lack labeled document training data. Our central finding is a pervasive calibration failure invisible under single-threshold protocols: methods achieve moderate Pixel-AUC (>=0.76) yet near-zero Pixel-F1. This AUC-F1 gap is not a discrimination failure but a score-distribution shift: tampered regions occupy only 0.27-4.17% of pixels in document images -- an order of magnitude less than in natural image benchmarks -- making the standard tau=0.5 threshold catastrophically miscalibrated. Oracle-F1 is 2-10x higher than fixed-threshold Pixel-F1, confirming that calibration, not representation, is the bottleneck. A controlled calibration experiment validates this: adapting a single threshold on N=10 domain images recovers 39-55% of the Oracle-F1 gap, demonstrating that threshold adaptation -- not retraining -- is the key missing step for practical deployment. Overall, no evaluated method works reliably out-of-the-box on diverse document types, underscoring that document forgery detection remains an unsolved problem. We further note that all eight datasets predate the era of generative AI editing; benchmarks covering diffusion- and LLM-based document forgeries represent a critical open gap on the modern attack surface.

Abstract

Text Forgery Detection

Explore similar work

DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis

When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documents

GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics