Rethinking VLMs for Image Forgery Detection and Localization

Shaofeng Guo; Jiequan Cui; Richang Hong

cs.CVMar 13, 2026

Rethinking VLMs for Image Forgery Detection and Localization

Authors: Shaofeng Guo, Jiequan Cui, Richang Hong

Organizations: Hefei University of Technology Key Laboratory of Knowledge Engineering with Big Data, Ministry of Education

Abstract

With the rapid rise of Artificial Intelligence Generated Content (AIGC), image manipulation has become increasingly accessible, posing significant challenges for image forgery detection and localization (IFDL). In this paper, we study how to fully leverage vision-language models (VLMs) to assist the IFDL task. In particular, we observe that priors from VLMs hardly benefit the detection and localization performance and even have negative effects due to their inherent biases toward semantic plausibility rather than authenticity. Additionally, the location masks explicitly encode the forgery concepts, which can serve as extra priors for VLMs to ease their training optimization, thus enhancing the interpretability of detection and localization results. Building on these findings, we propose a new IFDL pipeline named IFDL-VLM. To demonstrate the effectiveness of our method, we conduct experiments on 9 popular benchmarks and assess the model performance under both in-domain and cross-dataset generalization settings. The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.Code is available at: https://github.com/sha0fengGuo/IFDL-VLM.

Figures & tables

Figure 1: Comparison with the existing IFDL pipeline. (a) Existing IFDL algorithms enhance the interpretability with language explanations from VLMs. Meanwhile, the $<SEG>$ token and the $<CLS>$ token for localization and detection are derived from the responses of VLMs. (b) Our IFDL method decouples the optimization of language explanations and detection&localization. Moreover, the detection&localization results are used as extra inputs to assist the optimization of language explanation with VLMs.

Figure 2: VLMs exhibit a bias toward semantic plausibility rather than authenticity . As a result, they are often insensitive to semantically consistent or subtle forgeries, such as the replacement of a cat instance in (a) or the addition of a new person in (b). This inherent bias ultimately hinders IFDL performance.

Figure 3: Our IFDL-VLM framework. IFDL-VLM method decouples the optimization of detection&localization from the language explanation generation. (a) Detection&localization expert training with VIT and SAM. (b) Region-aware visual feature enhancement technique for language explanation generation.

Methods	Year	Real		Fully synthetic		Tampered		Overall
Methods	Year	Acc	F1	Acc	F1	Acc	F1	Acc	F1
AntifakePrompt	2024	0.65	0.79	0.94	0.97	0.31	0.47	0.63	0.74
CnnSpott	2021	0.80	0.89	0.40	0.57	0.07	0.13	0.42	0.53
FreDect	2020	0.84	0.91	0.17	0.29	0.12	0.21	0.37	0.47
Fusing	2022	0.85	0.92	0.34	0.51	0.03	0.05	0.41	0.49
Gram-Net	2020	0.70	0.82	0.94	0.97	0.01	0.02	0.55	0.60

Table 1: Comparisons with previous methods regarding detection performance on SID-Set. Numbers are borrowed from SIDA [ 18 ] . The best results are in bold. The overall accuracy and F1 are calculated as the average of the values from the three classification categories.

Method	Year	AUC	F1	IoU
MVSS-Net	2023	0.49	0.32	0.24
HiFi-Net	2023	0.64	0.46	0.21
PSCC-Net	2022	0.82	0.71	0.36
LISA-7B-v1	2024	0.78	0.69	0.33
SIDA-7B	2024	0.87	0.74	0.44
IFDL-VLM (Ours)	-	0.99	0.87	0.65

Table 2: Comparison with previous methods regarding localization results on SID-Set.

Figure 4: Qualitative comparison on localization mask and language explanations.

Method	CASIA1+		IMD2020		Columbia		NIST		DSO		Korus		DeepFake		AIGC-Editing		Avg
Method	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1
SPAN	0.11	0.14	0.09	0.14	0.14	0.20	0.16	0.21	0.14	0.24	0.06	0.10	0.04	0.06	0.09	0.12	0.10	0.15
ManTraNet	0.09	0.13	0.10	0.16	0.04	0.07	0.14	0.20	0.08	0.13	0.02	0.05	0.03	0.05	0.07	0.12	0.07	0.11
OSN	0.47	0.51	0.38	0.47	0.58	0.69	0.25	0.33	0.32	0.45	0.14	0.19	0.11	0.13	0.07	0.09	0.29	0.37
HiFi-Net	0.13	0.18	0.09	0.14	0.06	0.11	0.09	0.13	0.18	0.29	0.01	0.06	0.07	0.11	0.13	0.22	0.10	0.15
PSCC-Net	0.36	0.46	0.22	0.32	0.64	0.74	0.18	0.26	0.22	0.33	0.15	0.22	0.12	0.18	0.10	0.15	0.24	0.33

Table 3: Comparative results of tamper localization capabilities between competing IFDL methods and our IFDL-VLM, tested on MMTD-Set (Photoshop, DeepFake, AIGC-Editing). All numerical results for baseline methods are borrowed from FakeShield [ 45 ] . Results marked with an asterisk (*) are reproduced using the official checkpoint released by the authors.

Section	SIDA-13B-description	IFDL-VLM (Ours)
Mask	1.22 $\pm$ 1.10	2.28 $\pm$ 1.68
Type	2.96 $\pm$ 2.44	3.70 $\pm$ 2.21
Areas	1.67 $\pm$ 1.40	2.45 $\pm$ 1.69
Tampered Content	1.14 $\pm$ 1.44	1.98 $\pm$ 1.80
Visual Inconsistencies	2.40 $\pm$ 1.04	2.91 $\pm$ 1.22
Summary	2.48 $\pm$ 2.15	3.31 $\pm$ 2.11

Table 4: Quantitative comparison between SIDA-13B-description and our model on six evaluation dimensions with GPT-5. The overall score is computed as the average of the Mask score and the Text score.

Figure 5: Overall preference distribution in human evaluation.

Section	SIDA-13B-description	Ours
Type	$0.83$	$\mathbf{0.87}$
Areas	$0.61$	$\mathbf{0.67}$
Tampered Content	$0.44$	$\mathbf{0.49}$
Visual Inconsistencies	$0.67$	$\mathbf{0.70}$
Summary	$0.80$	$\mathbf{0.84}$
CSS(weighted)	$0.57$	$\mathbf{0.62}$

Table 5: Section-wise CSS scores.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$ Value	Type	Areas	Tampered Content	Visual Inconsistencies	Summary
0.0	0.79	0.64	0.54	0.59	0.77
0.3	0.83	0.66	0.58	0.65	0.80
0.5	0.87	0.67	0.60	0.70	0.84
0.7	0.80	0.68	0.55	0.62	0.79
1.0	0.75	0.65	0.50	0.60	0.77

Appendix

Table 6: Ablation study on

\alpha

for region-aware visual feature enhancement technique with CSS scores.

Method	AUC	F1	IoU	CSS	ROUGE-L
SIDA-7B	0.87	0.74	0.44	0.66	0.38
SIDA-13B	0.94	0.72	0.54	0.80	0.41
SIDA-7B-Unfrozen-CLIP	0.94	0.63	0.40	0.65	0.39
SIDA-13B-Unfrozen-CLIP	0.95	0.72	0.50	0.73	0.40
IFDL-VLM (Ours)	0.99	0.87	0.65	0.84	0.43

Appendix

Table 7: Comparison with SIDA (unfrozen CLIP).

Metric	FakeShield	IFDL-VLM	SIDA
Params (B)	22.0	14.3	14.0
FLOPs (T)	6.3	6.2	6.1
Peak Mem. (GB)	27.1	27.5	29.5

Appendix

Table 8: Efficiency comparison.

Figure 6: Attention details in the IFDL-VLM framework.

Method	BLEU-1	ROUGE-L	METEOR	CIDEr
SIDA-13B	0.548	0.406	0.272	0.016
IFDL-VLM (Ours)	0.580	0.434	0.290	0.030

Appendix

Table 9: Quantitative comparison on standard text similarity metrics.

Figure 7: The full multimodal prompt used to instruct GPT-5 for pairwise evaluation of IFDL models. The prompt specifies scoring criteria, output format, and input modalities (image, masks, texts).

Setting	BLEU-1	ROUGE-L	METEOR	CIDEr	CSS
PM (Clean)	0.579	0.434	0.290	0.030	0.842
PM (Boundary-Pert.)	0.574	0.431	0.292	0.020	0.842
GT	0.565	0.428	0.291	0.026	0.853

Appendix

Table 10: Quantitative evaluation of explanation robustness under boundary-perturbed masks. “Boundary-Pert.” denotes applying morphological operations to the predicted masks.

Figure 8: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 9: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 10: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 11: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 12: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 13: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 14: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 15: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 16: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 17: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 18: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 19: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 20: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 21: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Figure 22: Qualitative comparison example. We highlight the accurate descriptions from GT in red , while the areas where the model fails to identify the modified regions correctly are marked in blue .

Image Forgery Detection

Image Forgery Detection studies how to identify manipulated or synthetic images and locate the visual evidence of tampering.

See all 126 papers

Can Vision-Language Models Reason about AI Edits in Images?
Jul 30, 2026Darsha Udayanga, Pin-Yu Chen, Payel Das +1Image Forgery Detection Image Forgery Localization
1Rensselaer Polytechnic Institute · 2IBM Research
Venus-DeFakerOne: Unified Fake Image Detection & Localization
May 13, 2026GuangJian TeamImage Forgery Detection Digital Image Forensics
Ant Group
UC-VLM: Consistency-Driven Learning for AI-Generated Image Detection with Vision-Language Large Models
Aug 15, 2026Lei Tan, Shuwei Li, Mohan Kankanhalli +1AI-Generated Image Detection Large Vision-Language Models
National University of Singapore, Singapore

Jul 30, 2026J/K move · Enter open · S save

Can Vision-Language Models Reason about AI Edits in Images?

Darsha Udayanga, Pin-Yu Chen, Payel Das, Qiang Ji

1Rensselaer Polytechnic Institute · 2IBM Research

Image Forgery Detection Image Forgery Localization VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Detection and localization of AI-tampered images are critical for trustworthy AI, yet modern generative models have made such manipulations increasingly difficult to identify. While traditional binary classifiers can detect image tampering, they lack interpretability and generalization. Vision-Language Models (VLMs) offer a promising alternative due to their strong visual understanding and reasoning capabilities; however, existing approaches typically rely on supervised finetuning with curated explanations rather than exploiting their inherent reasoning capabilities. In this work, we investigate whether VLMs can be trained to reason about AI-generated image edits using reinforcement learning (RL) rather than explicit reasoning supervision. Motivated by the success in Group Relative Policy Optimization (GRPO), an RL technique that incentivizes the model to reason by generating thinking traces prior to giving the final answer, we propose a GRPO-based training framework that utilizes simple accuracy and format rewards. Given an input image, the model produces a structured reasoning trace and predicts whether the image has been tampered with. A lightweight segmentation model is then guided by the reasoning output to generate pixel-level localization masks. Experiments across multiple image manipulation datasets demonstrate that our approach achieves competitive detection and localization performance compared to state-of-the-art image forgery detectors, despite requiring substantially weaker supervision. We introduce effective intersection over union (eff-IoU), a unified metric to jointly evaluate detection and localization. These results suggest that reinforcement learning provides an effective and scalable mechanism for teaching VLMs to reason about AI-generated content.

Abstract

Image Forgery Detection

Explore similar work

Can Vision-Language Models Reason about AI Edits in Images?

Venus-DeFakerOne: Unified Fake Image Detection & Localization

UC-VLM: Consistency-Driven Learning for AI-Generated Image Detection with Vision-Language Large Models