From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering

Xinyi Shang; Yi Tang; Jiacheng Cui; Ahmed Elhagry; Salwa K. Al Khatib; Sondos Mahmoud Bsharat; Jiacheng Liu; Xiaohan Zhao

cs.CVMar 20, 2026

From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering

Authors: Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao,

Organizations: Mohamed bin Zayed University of Artificial Intelligence · University College London

Abstract

Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.

Figures & tables

Figure 1: Pitfalls of Current Benchmarks and Our Remedy. (a) They contain unrealistic samples. (b)–(d) Their widely adopted mask-based label contains large regions of not aligned pixels with the true generative pixels. In contrast, our pixel label is precisely aligned with the true generative pixels.

Figure 2: Visualization of our pixel-level label under different $\tau$ . Small $\tau$ emphasizes sensitivity to pixel edits, while larger $\tau$ emphasizes semantic changes.

Dataset	Year	Task	Multi-Object	Fidelity Check	Ground Truth
ArtiFact ( Rahman et al., 2023 )	2023	B-Cla.	✗	✗	-
TrainFors ( Nandi et al., 2023 )	2024	B-Cla. & T-Loc.	✗	✗	Mask
SIDBench ( Schinas and Papadopoulos, 2024 )	2024	B-Cla.	✗	✗	-
DiFF ( Cheng et al., 2024 )	2024	B-Cla.	✗	✗	-
M3Dsynth ( Zingarini et al., 2024 )	2024	B-Cla. & T-Loc.	✗	✗	Mask
SemiTruths ( Pal et al., 2024 )	2024	B-Cla. & T-Loc.	✗	✗	Mask

Table 1: Details of recent publicly available tampering benchmarks. The Task column indicates the evaluation type, where “B-Cla.” refers to binary classification ( “real” vs. “tampered”), and “T-Loc.” denotes tampering localization. “Multi-Object” means multiple objects within one image can be tampered with.

Figure 3: Overview of our PIXAR generation pipeline. The pipeline consists of four stages. Stage 1: Image Generation: We generate images with 8 tampering types. Stage 2: Tampering Effectiveness Checks: We filter out ineffective tampered images. Stage 3: Image Fidelity Assessment: The generated images are first assessed by Qwen3 ( Yang et al., 2025 ) and then reviewed by human annotators. Stage 4: Label Construction: We manually annotate semantic labels and construct faithful pixel-level annotations.

Panel of Figure 4 — Figure 4: Visualization of tampered images with varying (a) tampered sizes and (b) tampered complexities.

Figure 5: Visualization of various tampering types in PIXAR . For each type, from top-left to bottom-right, the images show: original image with red shading indicating the modified object, tampered image, pixel-difference map overlaid on the tampered image, and our pixel-level label.

Figure 6: Comparison of fidelity between SID-Set and our PIXAR .

Figure 7: Examples of pixel–semantic inconsistency.

Figure 8: Composition of PIXAR . Each entry includes four images with detailed metadata.

Figure 9: Overview of training framework.

Methods	Semantic Classification		Pixel Localization
	Top-1 Acc	Top-5 Acc	Recall	F1-Score	AUC	g-IoU	IoU
LISA-7B ( Lai et al., 2024 )	27.1	71.6	10.0	15.4	55.0	7.7	8.3
SIDA-7B ( Huang et al., 2025 )	27.1	71.9	15.0	21.1	55.0	10.7	11.8
PIXAR -7B-Lite (Ours)	28.2	75.0	26.4	26.1	55.2	14.3	15.0
PIXAR -7B (Ours)	36.2	77.0	29.8	30.6	62.2	16.1	18.1
LISA-13B ( Lai et al., 2024 )	30.6	75.1	11.4	17.3	55.6	9.0	9.5

Table 2: Pixel-level Tampered Region Localization Results and Associated Semantic Prediction Accuracy. PIXAR -Lite models are fine-tuned on a subset of our training data that contains masks to guide the generation of tampered images. In this setting, LISA and SIDA utilize these masks as ground-truth, whereas our model is supervised by the pixel-difference map with a threshold of

\tau=0.05

. All methods use the same backbone model, either LISA-7B or LISA-13B as specified.

Methods	Pixel Localization
	Recall	F1-Score	AUC	g-IoU	IoU
FakeShield	10.7	17.0	52.0	8.4	9.3
PIXAR -7B	29.8	30.6	62.2	16.1	18.1
PIXAR -13B	33.6	32.3	62.2	17.8	19.3

Table 3: Performance comparison between PIXAR and FakeShield.

Detector	Backbone	Data Distribution		Precision			Recall			F1-Score
		GANs	Diffusion	Real	Fake	Overall	Real	Fake	Overall	Real	Fake	Overall
CnnSpot ( Wang et al., 2020b )	ResNet-50 ( He et al., 2016 )	✓	✗	15.2	84.4	49.8	99.4	0.6	50.0	26.4	1.2	13.8
AntifakePrompt ( Chang et al., 2023 )	InstructBLIP ( Dai et al., 2023 )	✓	✓	4.6	84.8	44.7	0.01	99.9	50.0	0.03	91.7	45.9
SIDA-7B ( Huang et al., 2025 )	LISA-7B ( Lai et al., 2024 )	✗	✓	15.8	86.7	51.3	88.2	13.6	50.9	26.8	23.4	25.1
SIDA-13B ( Huang et al., 2025 )	LISA-13B ( Lai et al., 2024 )	✗	✓	16.5	86.3	51.4	62.0	42.8	52.4	26.0	57.2	41.6
PIXAR -7B	SIDA-7B ( Huang et al., 2025 )	✗	✓	39.1	97.6	68.4	89.9	74.9	82.4	54.5	84.8	69.7

Table 4: Comparison of existing deepfake detection methods and our models evaluated on the PIXAR test set.

Methods	Semantic Classification		Pixel Localization
	Top-1 Acc	Top-5 Acc	Recall	F1-Score	AUC	g-IoU	IoU
GPT-Image-1.5 ( OpenAI, 2026 )	26.0	70.5	16.5	20.9	52.3	11.3	11.7
Seedream 4.5 ( Seedream et al., 2025 )	29.9	73.1	27.8	26.8	57.8	13.9	15.5
Gemini 3 ( DeepMind, 2025 )	33.8	75.7	34.5	26.7	67.0	14.6	15.4
Gemini 2.5 ( Comanici et al., 2025a )	34.0	76.9	23.7	29.5	70.2	16.1	17.3
Flux.2 ( Labs, 2025 )	37.1	79.7	25.7	31.3	56.3	16.7	18.6

Table 5: Comparison of detection performance across different generative sources.

Table 15

	Semantic Classification		Pixel Localization
	Top-1 Acc	Top-5 Acc	Recall	F1-Score	AUC	g-IoU	IoU
$\lambda_{sem}=0.1$	35.1	76.2	29.7	30.3	62.2	15.9	17.9
$\lambda_{sem}=0.5$	36.2	77.0	29.8	30.6	62.2	16.1	18.1
$\lambda_{sem}=1.0$	35.2	76.2	29.7	30.5	62.0	16.0	18.0

Table 8: Impact of Semantic Loss Weight.

	Semantic Classification		Pixel Localization		Text Quality
	Top-1 Acc	Top-5 Acc	g-IoU	IoU	Css
$\lambda_{text}=2.0$	35.6	76.7	16.1	18.1	0.51
$\lambda_{text}=3.0$	36.2	77.0	16.1	18.1	0.75
$\lambda_{text}=4.0$	35.9	76.9	16.1	18.1	0.75

Table 9: Impact of Text Loss Weight.

	Semantic Classification		Pixel Localization
	Top-1 Acc	Top-5 Acc	Recall	F1-Score	AUC	g-IoU	IoU
$\lambda_{dice}=0.0$	35.3	76.3	12.9	19.5	61.6	7.4	10.8
$\lambda_{dice}=0.5$	36.0	76.6	22.8	27.3	62.0	13.5	15.8
$\lambda_{dice}=1.0$	36.2	77.0	29.8	30.6	62.2	16.1	18.1

Table 10: Impact of Dice Loss Weight

\lambda_{dice}

Panel of Figure 10 — Figure 10: Visualization comparison of prediction results between PIXAR and SIDA ( Huang et al., 2025 ) . The red dashed boxes show different prediction focuses compared to SIDA.

	Binary Classification			Localization
	Precision	Recall	F1-Score	Recall	F1-Score	IOU
Human	22.2	55.5	31.0	17.4	18.8	10.7

Table 11: Results of user study.

Figure 11: Visualization of pixel-level labels under different threshold values $\tau$ .

Figure 12: Visualization of tampered images generated by state-of-the-art open-source generative models.

Figure 13: Visualization of various tampering types in PIXAR . For each type, from top-left to bottom-right, the images show: the original image (with the tampering mask shown in red, if applicable), the generated tampered image, the pixel-difference map overlaid on the generated image, and our pixel-level label.

Panel of Figure 14 — Figure 14: Analysis of PIXAR Generated Images. (a) shows the distribution of different manipulation types. (b) shows the distribution of tampered sizes. We set the size categorization as: small indicates areas with fewer than $23000$ pixels, medium indicates areas between $23000$ and $50000$ pixels, and large indicates areas with at least $50000$ pixels.

Training set				Test set
Manipulation	#	%	Mask	Manipulation	#	%	Mask
intra-class replacement	101206	26.10	$\checkmark$	intra-class replacement	5601	13.41	$\times$
inter-class replacement	89415	23.06	$\checkmark$	inter-class replacement	5965	14.27	$\times$
object removal	1059	0.27	$\checkmark$	object removal	5694	13.63	$\times$
object addition	39651	10.22	$\times$	object addition	4947	11.84	$\times$
color change	36687	9.46	$\times$	color change	4456	10.67	$\times$

Table 12: Dataset composition by manipulation type (including a multi-edit subset). Train total

N=387{,}810

, Test total

N=41{,}781

. Mask indicates whether a mask is used to guide the generation of tampered images. The three mask-conditioned training manipulations ( intra-class replacement , inter-class replacement , and object removal ; marked with

\checkmark

) form the Lite training subset, as referenced in Tab. 2 .

Figure 15: Tampering Effectiveness Checks and Label Reliability Checks. (a) Geometric Rectification; (b) Edit Magnitude Check; (c) Spatial Concentration

Figure 16: Visualization of samples filtered and retained by human evaluation. Images with an orange background indicate filtered samples, while those with a green background indicate retained images.

Figure 17: Visualization of examples under different overlap ratios. For each ratio, two examples are shown. For each example, from left to right: the original image, input mask, generated image, and our pixel label at $\tau=0.05$ .

$r_{\text{grid}}$	$r_{\text{dens}}$	$r_{\text{grid}}(1-r_{\text{dens}})$	Label
$r_{\text{grid}}\leq 0.20$	—	—	Concentrated
$r_{\text{grid}}\geq 0.50$	—	—	Diverse
$0.20<r_{\text{grid}}<0.50$	$r_{\text{dens}}\geq 0.35$	—	Concentrated
$0.20<r_{\text{grid}}<0.50$	$r_{\text{dens}}\leq 0.25$	—	Diverse
$0.20<r_{\text{grid}}<0.50$	$0.25<r_{\text{dens}}<0.35$	$r_{\text{grid}}(1-r_{\text{dens}})\leq 0.25$	Concentrated
$0.20<r_{\text{grid}}<0.50$	$0.25<r_{\text{dens}}<0.35$	$r_{\text{grid}}(1-r_{\text{dens}})>0.25$	Diverse

Table 13: Spatial Concentration Check decision cases. Each row specifies one case for classifying a pixel label map using

(r_{\text{grid}},r_{\text{dens}})

and the tie-break score

r_{\text{grid}}(1-r_{\text{dens}})

. Diverse samples are discarded in the pipeline.

Manipulation type	Template	Example
background change	The background was changed while keeping the foreground unchanged.	The background was changed while keeping the foreground unchanged.
object removal	The {orig} was removed from the image.	The car was removed from the image.
object addition	A {cat} was added to the image.	A bicycle was added to the image.
intra-class repl.	The {orig} was replaced with a different-looking {orig}.	The dog was replaced with a different-looking dog.
inter-class repl.	The {orig} was replaced with a {repl}.	The chair was replaced with a sofa.
color change	The color of the {cat} was changed.	The color of the shirt was changed.

Table 14: Prompt templates for description construction. We deterministically map structured metadata to single-sentence tampering description. For multi-edit samples, we form a compositional explanation by concatenating the corresponding single-edit descriptions in the applied order.

Model	#	%
Qwen-Image ( Wu et al., 2025 )	8185	19.59
GPT-Image-1.5 ( OpenAI, 2026 )	7016	16.79
Flux.2 ( Labs, 2025 )	6651	15.92
Gemini 2.5 ( Comanici et al., 2025a )	6636	15.88
Gemini 3 ( DeepMind, 2025 )	6716	16.07
Seedream 4.5 ( Seedream et al., 2025 )	6577	15.74

Table 15: Test set composition by generation model. Total

N=41{,}781

Figure 18: Dataset statistics in PIXAR. The upper and lower rows visualize the training and test partitions, respectively. In each row, from left to right, we report the distributions over (a) manipulation size, (b) manipulated object classes, and (c) manipulation types.

Top 25%			Top 50%			Top 75%			Bottom 25%
Class	#	%	Class	#	%	Class	#	%	Class	#	%
person	1812	4.52	elephant	586	1.46	bird	464	1.16	hot dog	370	0.92
car	1308	3.26	tv	578	1.44	suitcase	464	1.16	sports ball	367	0.91
chair	1240	3.09	clock	572	1.43	sink	462	1.15	tie	362	0.90
dining table	784	1.95	potted plant	568	1.42	cell phone	459	1.14	remote	348	0.87
bottle	748	1.86	sheep	562	1.40	fire hydrant	456	1.14	donut	345	0.86

Table 16: Test set class distribution (single-edit subset). Total

N=40{,}113

. We report per-class counts and percentages.

Image Forgery Detection

Image Forgery Detection studies how to identify manipulated or synthetic images and locate the visual evidence of tampering.

See all 126 papers

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs
Jul 20, 2026Yi Tang, Xinyi Shang, Jiacheng Cui +12Vision-Language Models Image Forgery Detection
Mohamed bin Zayed University of Artificial Intelligence · University College London · Weizmann Institute of Science
Can Vision-Language Models Reason about AI Edits in Images?
Jul 30, 2026Darsha Udayanga, Pin-Yu Chen, Payel Das +1Image Forgery Detection Image Forgery Localization
1Rensselaer Polytechnic Institute · 2IBM Research
Semantic Manipulation Localization
Apr 11, 2026Zhenshan Tan, Chenhan Lu, Yuxiang Huang +6Image Forgery Localization Digital Image Forensics
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology, Nanjing, 210044, China · College of Computer Science and Technology, Zhejiang University, and KOKONI3D, Moxin (Hangzhou) Technology Company Ltd., Hangzhou 310027, China

Jul 20, 2026J/K move · Enter open · S save

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

Yi Tang, Xinyi Shang, Jiacheng Cui, Sondos Mahmoud Bsharat+11

Mohamed bin Zayed University of Artificial Intelligence · University College London · Weizmann Institute of Science

Vision-Language Models Image Forgery Detection Domain Generalization OOD Generalization Image Forgery Localization

Open paper AskPDF arXiv

Modern vision-language models (VLMs) have significantly improved image generation and editing capabilities, making pixel-level image tampering detection increasingly important yet challenging under cross-model and out-of-distribution shifts. This work studies domain generalization for pixel-level image tampering detection in modern VLMs like ChatGPT, Gemini, Qwen-Image, etc., aiming to learn tampering localization models that remain robust across diverse VLM-generated manipulation distributions. We propose a simple yet effective domain-generalized training framework built on two practical strategies. First, we introduce a balanced minibatch sampling scheme that strategically samples tampered and real images in each minibatch, preventing biased optimization toward either manipulated artifacts or clean-image priors and avoiding training collapse, ensuring that each optimization step receives proper sampled gradient signals. Second, we adopt a simple late-injection strategy, where the detector is first trained on large-scale base data until stable convergence, and then exposed to a small amount of newly selected supporting data from emerging VLM distributions, improving adaptability without overfitting to limited new domains. Together, these components provide a simple yet strong recipe for improving pixel-level tampering localization and OOD robustness across modern VLMs. Despite the conceptual simplicity, our framework outperforms the prior state-of-the-art PIXAR by a large margin of 26.1% and 26.8% relative improvement in average gIoU and cIoU, respectively, across OOD VLMs of GPT-Images-2.0, Gemini-3.1, FLUX.2, and Seedream 4.5. Our code is available at https://github.com/VILA-Lab/PIXAR-DG

Abstract

Image Forgery Detection

Explore similar work

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

Can Vision-Language Models Reason about AI Edits in Images?

Semantic Manipulation Localization