GAP-SAM: A Global Artifact Prior for Generalizable AI-Generated Image Manipulation Localization

Haozhen Yan; Siyuan Shan; Zijian Yu; Youqi Wang; Yan Hong; Jun Lan; Jianfu Zhang

cs.CVAug 21, 2026

GAP-SAM: A Global Artifact Prior for Generalizable AI-Generated Image Manipulation Localization

Authors: Haozhen Yan, Siyuan Shan, Zijian Yu, Youqi Wang, Yan Hong, Jun Lan, Jianfu Zhang

Organizations: Shanghai Jiao Tong University · Ant Group · Shenzhen University

Abstract

AI-generated image manipulation localization identifies edited pixels, but its OOD performance lags behind image-level detection partly because pixel supervision entangles forensic evidence with dataset-specific mask geometry and semantic boundaries. Extending image-level distribution alignment to localization, we construct COCO-ControlNet with source-image Canny edges and depth maps to align semantics and geometry, improving OOD performance across multiple localizers. Yet tighter Mask-VAE Reconstruction Alignment (Mask-VAE) underperforms COCO-ControlNet, showing that VAE reconstruction artifacts transfer poorly to local diffusion-inpainting artifacts. We also identify boundary adhesion, where fine-tuned segmentation models snap predictions to semantic object contours rather than true manipulation boundaries. These findings motivate GAP-SAM, which encodes an image and its frozen VAE reconstruction into a global artifact token and injects it into SAM3's feature pyramid via zero-gated FiLM before pixel decoding. Without prescribing a spatial region, this token modulates dense decoding to preserve localization while suppressing semantic-boundary shortcuts. Across six datasets, GAP-SAM averages 79.8 Pixel-F1, outperforming the strongest prior method by 12.6 points. It also performs best at every tested severity of JPEG compression, Gaussian blur, and resizing.

Figures & tables

Figure 1: Relative IID-to-OOD F1 drops on OpenSDI; localization drops more than classification, indicating weaker OOD performance.

Figure 2: COCO-ControlNet data construction pipeline. Starting from a real MS-COCO image, the pipeline combines an annotation-based or randomly generated mask, an optional text prompt, and either Canny or depth conditioning to produce an inpainted region or a fully generated image.

Figure 3: Pixel-F1 comparison between official weights and checkpoints retrained on COCO-ControlNet.

Method	Training data	COCO-CN	OOD Avg.	All Avg.
CoDE	Mask-VAE	41.0/25.8	44.6/29.8	44.0/29.1
	COCO-CN	92.7/86.4	61.4/46.5	66.6/53.1
MaskCLIP	Mask-VAE	35.6/21.7	48.7/33.6	46.6/31.6
	COCO-CN	91.3/84.1	55.2/41.7	61.2/48.8

Table 1: Mask-VAE versus COCO-ControlNet training under matched COCO sources and masks (F1/IoU

\uparrow

Figure 4: Boundary adhesion in SAM3 directly fine-tuned on COCO-ControlNet. From left to right: manipulated image, ground-truth mask (green), and predicted mask (red). The predictions partially overlap the ground-truth regions but expand or shrink to the full contours of salient semantic objects rather than following the true manipulation boundaries.

Figure 5: Overview of GAP-SAM. Adaptive and frozen SAM3 encoders extract features from the input image and its VAE reconstruction, respectively, and the Paired Artifact Encoder fuses the pooled features into a Global Artifact Prior Token supervised by an auxiliary artifact classifier. Zero-gated FiLM uses this token to modulate the input-image feature pyramid before the native SAM3 localization pipeline predicts the manipulation mask.

	COCO-CN		OpenSDID		SID-Set		AutoSplice		UltraEdit		DiffSeg30K		Avg.
Method	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU
SparseViT ( Su et al. 2025 )	89.0	80.1	49.6	33.0	74.9	59.8	61.2	44.1	9.6	5.1	45.3	29.3	54.9	41.9
IML-ViT ( Ma et al. 2024 )	77.5	63.3	61.4	44.2	37.0	22.7	47.6	31.3	41.4	26.1	18.3	10.1	47.2	32.9
PSCC-Net ( Liu et al. 2022 )	84.3	72.9	68.4	52.0	67.1	50.5	56.9	39.8	31.4	18.7	60.2	43.1	61.4	46.2
TruFor ( Guillaro et al. 2023 )	86.8	76.6	75.8	61.1	77.2	62.9	64.5	47.6	49.0	32.5	49.7	33.1	67.2	52.3
CoDE ( Peng et al. 2024 )	92.7	86.4	75.1	60.2	83.7	71.9	62.3	45.2	36.2	22.1	49.5	32.9	66.6	53.1

Table 2: Pixel-level localization after training on COCO-ControlNet, with COCO-ControlNet as the in-domain test set and all other test sets as OOD benchmarks. Best and second-best results are bolded and underlined, respectively.

Figure 6: Qualitative comparison: GAP-SAM reduces semantic boundary adhesion and localizes arbitrary manipulation regions.

Setting (F1/IoU $\uparrow$ )	COCO-CN	OOD Avg.	All Avg.
Baseline	95.4/91.1	68.5/54.4	73.0/60.6
+ Original-only prior	95.7/92.1	69.4/55.1	73.8/61.3
+ Reconstruction-paired prior	97.3/94.8	73.5/59.7	77.5/65.6
+ Artifact supervision (Full)	98.3/96.7	76.1/63.9	79.8/69.3

Table 3: Incremental module ablation.

Setting (F1/IoU $\uparrow$ )	COCO-CN	OOD Avg.	All Avg.
Gated concatenation	96.6/93.7	75.0/62.4	78.6/67.6
Residual addition	95.4/94.2	74.7/61.4	78.2/66.9
FiLM	98.3/96.7	76.1/63.9	79.8/69.3

Table 4: Conditioning operator ablation.

Injection site (F1/IoU $\uparrow$ )	COCO-CN	OOD Avg.	All Avg.
Object queries	96.6/93.4	72.2/60.6	76.3/66.1
Post-decoder features	97.4/95.1	74.4/62.0	78.2/67.5
Instance embedding	98.5/97.1	74.7/62.8	78.7/68.5
Pre-decoder FPN	98.3/96.7	76.1/63.9	79.8/69.3

Table 5: Injection-site ablation with query-mask output fixed; the last row is the default GAP-SAM configuration.

Reconstruction Encoder
Setting (F1/IoU $\uparrow$ )	COCO-CN	OOD Avg.	All Avg.
Shared LoRA	98.7/97.5	73.0/61.3	77.3/67.3
Frozen	98.3/96.7	76.1/63.9	79.8/69.3
Artifact-Prior Representation
Spatial delta	99.0/98.1	74.6/62.9	78.7/68.8
Spatial cross-attn.	97.6/95.4	74.4/60.9	78.3/66.6

Table 6: Reconstruction encoder/GAP encoding ablation.

Figure 7: Robustness analysis on COCO-ControlNet under JPEG compression, resizing, and Gaussian blur.

Image Forgery Localization

Image forgery localization identifies manipulated or AI-generated regions in images, often alongside image-level authenticity detection.

See all 79 papers

Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization
Apr 14, 2026Xiaojie Liang, Zhimin Chen, Ziqi Sheng +1Image Forgery Localization Digital Image Forensics
Sun Yat-sen University, School of Computer Science and Engineering, Guangzhou, Guangdong, China
FUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization
Aug 28, 2026Anton Nuzhdin, Marcel Worring, Ivona NajdenkoskaAI-Generated Image Detection Image Forgery Detection
Informatics Institute, University of Amsterdam Amsterdam, The Netherlands
Progressive Decision-Making for Localizing Open-Ended AI-Generated Image Forgeries
Jul 31, 2026Jingyi Hou, Xiaoxia Chen, Leyu Zhou +2Image Segmentation Image Forgery Localization
School of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China · Institute of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China · Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education, University of Science and Technology Beijing, Beijing 100083, China

Apr 14, 2026J/K move · Enter open · S save

Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization

Xiaojie Liang, Zhimin Chen, Ziqi Sheng, Wei Lu

Sun Yat-sen University, School of Computer Science and Engineering, Guangzhou, Guangdong, China

Image Forgery Localization Digital Image Forensics

Open paper AskPDF arXiv

As generative image editing advances, image manipulation localization (IML) must handle both traditional manipulations with conspicuous forensic artifacts and diffusion-generated edits that appear locally realistic. Existing methods typically rely on either low-level forensic cues or high-level semantics alone, leading to a fundamental micro--macro gap. To bridge this gap, we propose FASA, a unified framework for localizing both traditional and diffusion-generated manipulations. Specifically, we extract manipulation-sensitive frequency cues through an adaptive dual-band DCT module and learn manipulation-aware semantic priors via patch-level contrastive alignment on frozen CLIP representations. We then inject these priors into a hierarchical frequency pathway through a semantic-frequency side adapter for multi-scale feature interaction, and employ a prototype-guided, frequency-gated mask decoder to integrate semantic consistency with boundary-aware localization for tampered region prediction. Extensive experiments on OpenSDI and multiple traditional manipulation benchmarks demonstrate state-of-the-art localization performance, strong cross-generator and cross-dataset generalization, and robust performance under common image degradations.

Abstract

Image Forgery Localization

Explore similar work

Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization

FUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization

Progressive Decision-Making for Localizing Open-Ended AI-Generated Image Forgeries