Organizations: East China Normal University · University of Science and Technology of China · Shanghai Development Center of Computer Software Technology · Shanghai Artificial Intelligence Laboratory
Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image-caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.
Figures & tables
Figure 1: A canonical photograph of John F. Kennedy with his face replaced by a synthetic one. The model rejects the replacement face in isolation, yet infers Kennedy from the full scene and judges the edited photograph authentic.
K1 People
K5 Event attribute
Karl May (died in 1912) inserted
Zidane’s shirt recolored blue (France wore white)
Scene: Casablanca (1942)
Scene: 2006 World Cup final
K2 Objects
K6 Artwork
Jet airliner (first flew in 1949) inserted
Extra figure added (not in the canonical composition)
Scene: McKinley campaign print (1900)
Scene: Leonardo’s The Last Supper
K3 Place
K0 Control
Table 1: Mandela-Bench item taxonomy. Full visual examples and the six-domain distribution are shown in Appendix Figure 5 .
Figure 2: Construction of Mandela-Bench for one 1945 Yalta Conference seed. Stage 1 anchors the seed to a knowledge-base entry. Stage 2 generates a knowledge-anchored edit and matched K0 control, automatically checks candidates, and retries failures. Stage 3 applies manual review, pixel-detector auditing, and deduplication before release. Here the first proposal failed the face check.
Table 2: Main results on the release set (1,507 forged, 474 authentic; strict scoring) for the 15 main-text models; all 36 are in Appendix Table 5 . KGR, Det, Auth, and Bal denote knowledge-grounded rate, forgery detection, authentic-image accuracy, and balanced accuracy. Left: L0, L1, and L-adv results. Right: L0 KGR by knowledge family. Colors show changes from the corresponding baseline (L1 and L-adv vs. L0; K1–K6 vs. K0): blue better, red worse; Δ KGR = L1 − L0 is defined as elicitation gap; † Judge model. Rows are sorted by L1 KGR.
Full
Full+Caption
Crop
Crop+Caption
Absent-celebrity (L0)
Derived
Model
Det
Bal
KGR
Det
Bal
KGR
Det
Bal
KGR
Det
Bal
KGR
Erase
Swap
K0
Named
& Real
Cap.
Pen.
Proprietary
Gemini-3.8-Flash
73.1
78.9
35.2
77.9
75.2
33.1
79.3
76.4
46.2
86.2
79.1
44.8
53.0
71.8
2.4
74.2
32.6
+4.8
+0.2
GPT-5.6
53.8
69.2
14.5
57.2
71.5
13.1
56.6
67.3
13.1
69.7
76.8
16.6
17.4
52.6
2.4
23.5
19.7
+3.4
+7.6
Open weights
GLM-5.3-Flash
52.4
67.5
7.6
47.6
68.7
4.8
62.8
68.8
14.5
59.3
74.5
10.3
32.6
65.4
9.8
52.3
37.1
− 4.8
+7.0
Table 3: People-family analyses. Det, Bal, KGR as in Table 2 . Left: 145 peripheral forgeries under L1 in the four de-recognition conditions. Right: 132 absent-celebrity items under the default probe L0. Named denotes answers naming the erased person; Real denotes naming that person while judging the image authentic. Colors show changes from Full (Erase from Swap); bold Det indicates McNemar p<0.05 . Cap. = Det(Full+Caption) − Det(Full); Pen. = Bal(Crop+Caption) − Bal(Full). ¶ Recognizes fewer than half of the images. All models: Appendix Table 6 .
Figure 4: Results across all 36 models. (a) Prompting increases detection more than KGR. (points above y=x ) (b) Recognition penalty is positive for models recognizing at least half of the de-recognition images and near zero otherwise (median +0.4). (c) Models that often name the erased person show larger erase–swap gaps. Wedges mark off-axis points.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Mandela-Bench examples for each knowledge family, an anchor-free control (K0), and an untouched authentic image. Center: distribution of 1,507 forged items across six image domains.
Figure 6: One seed and its derived benchmark items (top), and the de-recognition conditions applied to the knowledge-anchored forgery (bottom). Top: (a) authentic original; (b) knowledge-anchored forgery with George Orwell inserted, for which the ideal verdict is fake, grounded in the recorded factual contradiction; (c) matched anchor-free control (K0) with an unnamed face inserted at the same position by the same editor; (d) absent-celebrity item with Maradona replaced by a synthetic face. Bottom: (e) Full; (f) Full+Caption; (g) Crop; (h) Crop+Caption. The crop preserves the edited region while removing much of the recognizable composition, and the box marks the crop region. (f) and (h) receive the same event-and-date context.
Human
E0
E1
E2
E3
E4
n
exact
±1
E0
46
1
2
0
2
51
90.2
92.2
E1
0
29
5
0
0
34
85.3
100.0
E2
0
9
30
0
0
39
76.9
100.0
E3
0
0
4
30
1
35
85.7
100.0
E4
0
2
2
8
35
47
74.5
91.5
Appendix
Table 4: Human label (rows) against the judge’s level (columns) on the 206 calibration answers; the right-hand columns give exact and within-one agreement by human level.
Default (L0)
Prompted (L1)
L-adv
KGR by knowledge family (L0)
Derived
Model
KGR
Det
Auth
Bal
KGR
Det
Auth
Bal
Auth
K0
K1
K2
K3
K4
K5
K6
Δ KGR
Proprietary
Gemini-3.8-Flash
46.6
66.3
94.7
80.5
55.1
80.5
88.2
84.3
99.4
0.7
27.5
79.2
58.4
56.5
66.2
66.0
+8.5
GPT-5.6
19.8
40.9
95.8
68.3
36.0
61.0
91.6
76.3
99.2
0.0
5.8
54.8
19.1
26.9
20.7
17.9
+16.2
Open weights
GLM-5.3-Flash
29.7
53.8
92.6
73.2
34.8
60.6
85.9
73.2
99.4
2.0
7.6
62.2
40.5
55.6
40.0
41.7
+5.1
Appendix
Table 5: Table 2 for all thirty-six models (same columns and notation). Family sizes: K0 148, K1 567, K2 294, K3 89, K4 108, K5 145, K6 156. Colour saturates at a change of 20 points (L1 vs. L0), 50 (L-adv vs. L0), and 60 (families vs. K0). Rows sorted by KGR under L1 within each group.
Full
Full+Caption
Crop
Crop+Caption
Absent-celebrity (L0)
Derived
Model
Det
Bal
KGR
Det
Bal
KGR
Det
Bal
KGR
Det
Bal
KGR
Erase
Swap
K0
Named
& real
Cap.
Pen.
Proprietary
Gemini-3.8-Flash
73.1
78.9
35.2
77.9
75.2
33.1
79.3
76.4
46.2
86.2
79.1
44.8
53.0
71.8
2.4
74.2
32.6
+4.8
+0.2
GPT-5.6
53.8
69.2
14.5
57.2
71.5
13.1
56.6
67.3
13.1
69.7
76.8
16.6
17.4
52.6
2.4
23.5
19.7
+3.4
+7.6
Open weights
GLM-5.3-Flash
52.4
67.5
7.6
47.6
68.7
4.8
62.8
68.8
14.5
59.3
74.5
10.3
32.6
65.4
9.8
52.3
37.1
− 4.8
+7.0
Appendix
Table 6: Table 3 for all thirty-six models (same columns and notation; ¶ recognizes fewer than half of the images under Full). Colour saturates at a change of 25 points from Full (40 for Erase vs. Swap). Authentic accuracy under every condition is in Table 17 ; parsed-only rates in Table 18 .
Model
Full
Mirror
Crop
n
Model
Full
Mirror
Crop
n
Qwen3.5-122B-A10B
88.5
85.6
48.8
902
Qwen3-Omni-30B-A3B
66.3
64.6
33.6
902
Kimi-K3
87.8
86.9
31.3
902
Qwen3.6-35B-A3B
62.0
58.2
25.2
902
GPT-5.6
87.8
86.8
42.1
902
Qwen3.8-27B †
60.2
51.4
17.0
902
Gemini-3.8-Flash
87.4
87.5
43.2
902
Gemma-4-31B
57.5
52.7
19.4
902
Qwen3.5-35B-A3B
86.3
85.8
48.4
902
Qwen3-VL-8B
56.4
52.5
11.8
902
Qwen3.5-27B
83.6
80.5
47.1
902
Qwen3-VL-235B-A22B
54.4
46.3
13.3
902
Appendix
Table 7: Self-reported recognition (%, RECOG probe) of the 902 peripheral forgeries with a crop variant, under Full, Mirror (the full image flipped horizontally), and Crop (the edited region and its neighbours). Rows sorted by recognition under Full; n=902 for every model (release set, complete sweeps). Detection, KGR, and authentic accuracy under the same conditions are in Table 19 .
TEXT (%)
L2 (%)
L0 detection by L2 answer (%)
FACE
Model
knows
names
cannot
named
cannot
pass
full real (%)
named (%)
Proprietary
Gemini-3.8-Flash
83.0
68.2
12.1
53.3 (90)
62.5 (16)
81/119
29.6
49.4
GPT-5.6
80.0
38.6
42.4
17.6 (51)
16.1 (56)
4/119 r
75.0
75.0
Open weights
GLM-5.3-Flash
83.5
43.2
13.6
32.7 (55)
27.8 (18)
—
—
—
Appendix
Table 8: Text-only, identity, and face probes for all models. TEXT reports the share of 435 named-insertion forgeries for which the model, given only the name, event, and date, correctly rules out the person’s presence. The remaining columns use the 132 verified absent-celebrity items. L2 reports whether the erased person is named or whether the model says someone cannot be identified; L0 detection is then stratified by those L2 responses. FACE reports, for five models, how often the original face is accepted and the replacement rejected when shown in isolation (119 pairs), followed by full-image authenticity and naming rates on the pass set. r GPT-5.6 declines isolated-face identification, so its FACE row reflects refusal rather than perception; — indicates that the probe was not run. † Also the judge.
Model
pairs
Full
Crop
only Full / only Crop
p
Proprietary
Gemini-3.8-Flash
116
92.2
44.0
56/0
< 0.0001
GPT-5.6 §
66
56.0
7.8
37/1
< 0.0001
Open weights
GLM-5.3-Flash
95
56.0
6.9
52/1
< 0.0001
Kimi-K3
116
84.5
18.1
78/1
< 0.0001
Appendix
Table 9: The remembered people follow the composition. On the 116 peripheral people-family forgeries whose scene has named principal figures, the identity probe (L2) is asked on the full image and on the crop around the edited region. Full / Crop: share of items whose answer names at least one principal figure of the scene (strict denominators); only Full / only Crop: items named under one condition only; p : exact McNemar test. Rows in the order of Table 2 . † Also the judge.
RECOG
People family (K1)
Class-level families (K2–K6)
Model
Full (%)
recognized
not
Δ
OR
p
recognized
not
OR
p
Proprietary
Gemini-3.8-Flash
78.2
75.4 (391)
87.7 (203)
− 12.2
0.67
0.1766
79.7 (788)
89.6 (125)
0.29
0.0529
GPT-5.6
76.8
37.9 (116)
63.8 (69)
− 25.8
0.58
0.3597
66.7 (129)
40.0 (5)
3.36
0.0860
Open weights
GLM-5.3-Flash
74.2
60.0 (365)
56.3 (229)
+3.7
1.15
0.5612
61.9 (753)
61.9 (160)
1.30
0.5536
Appendix
Table 10: Prompted-verification detection as a function of self-reported scene recognition (RECOG under Full), shown per model on the release set with strict scoring. For the people family (K1), we report detection on recognized and unrecognized scenes, their difference, and the recognition coefficient from a logistic regression controlling for saliency, family, and domain with scene-clustered errors. The same quantities are reported for class-level families (K2–K6). Anchor-free controls are assigned to the family of their edit specification. ¶ Recognizes fewer than half of the images and is excluded from the pooled model; — denotes separated or empty cells. † Also the judge.
Full
Full+Caption
Full+Wrong caption
p vs. Full
Wrong caption
Model
Det
Auth
Bal
KGR
Det
Auth
Bal
KGR
Det
Auth
Bal
KGR
Caption
Wrong
uptake
dispute
Gemini-3.8-Flash
73.1
84.9
79.0
35.2
77.9
73.1
75.5
33.1
93.8
19.4
56.6
31.0
0.1671
< 0.0001
80.8
30.1
GPT-5.6
53.8
84.9
69.4
14.5
57.2
86.0
71.6
13.1
91.0
16.1
53.6
9.7
0.4421
< 0.0001
84.1
10.9
Appendix
Table 11: Wrong-caption control on the 145 peripheral people-family forgeries and 93 authentic originals under prompted verification and strict scoring. We compare Full, Full+Caption, and Full+Wrong caption, where the wrong caption names a different same-era event from the same domain. Det, Auth, Bal, and KGR are defined as in the main text; p is the exact McNemar test against Full. Uptake and dispute report whether the model adopts or explicitly rejects the wrong event.
Det
KGR
Cap.
Model
Fam.
n
F
FC
C
CC
F
FC
C
CC
Δ
p
pCC
Gemini-3.8-Flash
K1
145
73.1
77.9
79.3
86.2
35.2
33.1
46.2
44.8
+4.8
0.1671
0.0009
K2
294
96.3
96.3
94.6
100.0
84.7
83.3
85.4
91.5
0.0
1.00
0.0010
K3
89
85.4
88.8
91.0
97.8
69.7
62.9
79.8
88.8
+3.4
0.4531
0.0010
K4
108
90.7
96.3
75.9
90.7
87.0
94.4
66.7
84.3
+5.6
0.1094
1.00
K5
145
86.2
90.3
83.4
95.9
77.9
85.5
73.1
87.6
+4.1
0.1796
0.0013
Appendix
Table 12: Detection and KGR under the four de-recognition conditions by knowledge family for the two proprietary models (peripheral forgeries on the release set; prompted verification; strict scoring). n : forgeries; F, FC, C, CC: Full, Full+Caption, Crop, Crop+Caption; Cap.: Det(FC) − Det(F) with its exact McNemar p ; pCC : McNemar p of Det(CC) against Det(F).
Authentic accuracy, L1 (%)
By recognition (L1)
Model
All
A1
A2
A3
A4
A5
A6
RECOG
recognized
not
Δ
Proprietary
Gemini-3.8-Flash
88.2
93.3
83.3
88.6
91.5
89.6
41.7
91.4
89.4
75.6
+13.8
GPT-5.6
91.6
95.2
95.0
91.8
91.5
91.0
41.7
86.1
91.9
89.4
+2.5
Open weights
GLM-5.3-Flash
85.9
92.3
86.7
83.7
85.1
82.1
83.3
83.3
89.4
68.4
+21.0
Appendix
Table 13: Accuracy on authentic images (%, prompted verification, strict scoring) by domain (A1 historical photographs 104, A2 film stills 60, A3 sports 184, A4 paintings 47, A5 political news 67, A6 meme templates 12 authentic images), and by whether the model recognizes the image (RECOG under Full: share recognized, accuracy on recognized and unrecognized images, difference). † Also the judge.
Detection
KGR
Model
C1
C2
C3
C1 − C3
C1
C2
C3
Proprietary
Gemini-3.8-Flash
81.5
98.6
78.2
+3.3
42.0
93.1
61.4
GPT-5.6
56.4
97.2
61.2
− 4.8
22.6
81.9
41.7
Open weights
GLM-5.3-Flash
58.2
90.3
59.8
− 1.6
20.2
79.2
41.6
Appendix
Table 14: Detection and KGR (%, prompted verification, strict scoring) by saliency of the edit over all families: C1 main figure (605 forgeries), C2 secondary figure (72), C3 background (830). † Also the judge.
L0
L1
K1 Det (L1)
K1 KGR (L1)
Model
E3+
E4
E3+
E4
all
no border
all
no border
Proprietary
Gemini-3.8-Flash
46.6
28.4
55.1
37.2
81.1
80.0
33.7
33.7
GPT-5.6
19.8
8.3
36.0
26.3
62.1
61.4
18.7
18.6
Open weights
GLM-5.3-Flash
29.7
22.5
34.8
29.6
60.0
59.1
11.1
10.8
Appendix
Table 15: E4 rate next to KGR (E3+), and the people family without threshold-borderline items (%, strict scoring). E3+ / E4 under the default (L0) and prompted (L1) probes over all forgeries; K1 Det / KGR under L1 on all people-family forgeries and after dropping the 41 items whose face-similarity score lies within ±0.05 of the acceptance threshold. † Also the judge.
Model
K0
K1
K2
K3
K4
K5
K6
Gemini-3.8-Flash
11
65
92
74
62
70
69
GPT-5.6
5
40
73
43
40
30
28
GLM-5.3-Flash
13
56
75
64
61
50
53
Kimi-K3
4
48
67
45
67
37
29
Kimi-K2.6
5
40
65
33
54
39
18
Qwen3.5-122B-A10B
22
49
69
52
45
48
39
Appendix
Table 16: Detection rate (%) on forgeries under the default probe by knowledge family, over items with a parsed verdict. Family sizes (forged items): K0 148, K1 567, K2 294, K3 89, K4 108, K5 145, K6 156. Rows in the order of Table 2 . KGR by family is in Table 5 .
Detection on forgeries (%)
Authentic accuracy (%)
Balanced accuracy (%)
KGR (%)
Model
F
FC
C
CC
F
FC
C
CC
F
FC
C
CC
F
FC
C
CC
Gemini-3.8-Flash
73.1
77.9
79.3
86.2
84.7
72.4
73.5
72.0
78.9
75.2
76.4
79.1
35.2
33.1
46.2
44.8
GPT-5.6
53.8
57.2
56.6
69.7
84.7
85.7
78.1
83.9
69.2
71.5
67.3
76.8
14.5
13.1
13.1
16.6
GLM-5.3-Flash
52.4
47.6
62.8
59.3
82.7
89.8
74.9
89.7
67.5
68.7
68.8
74.5
7.6
4.8
14.5
10.3
Kimi-K3
35.9
26.9
82.8
62.8
92.9
95.9
57.1
86.5
64.4
61.4
69.9
74.6
3.4
2.1
12.4
8.3
Kimi-K2.6
28.3
29.7
67.6
63.4
89.8
87.8
75.0
89.2
59.0
58.7
71.3
76.3
4.1
2.8
20.0
15.2
Appendix
Table 17: All four de-recognition conditions for every model with condition variants, using the same population, probe, and strict scoring as Table 3 . We report detection, authentic accuracy, balanced accuracy, and KGR under Full (F), Full+Caption (FC), Crop (C), and Crop+Caption (CC). Each sweep covers 145 forgeries and the 98 authentic originals from the same scenes.
Detection on forgeries (%)
Authentic accuracy (%)
Balanced accuracy (%)
Model
F
FC
C
CC
F
FC
C
CC
F
FC
C
CC
Gemini-3.8-Flash
73.1
77.9
79.3
86.2
84.7
72.4
73.5
72.0
78.9
75.2
76.4
79.1
GPT-5.6
53.8
57.2
56.6
69.7
84.7
85.7
78.1
83.9
69.2
71.5
67.3
76.8
GLM-5.3-Flash
59.4
50.0
72.2
61.0
92.0
96.7
83.1
93.0
75.7
73.4
77.7
77.0
Kimi-K3
35.9
27.3
82.8
63.6
92.9
98.9
57.3
86.8
64.4
63.1
70.0
75.2
Kimi-K2.6
28.3
29.9
68.1
63.9
89.8
87.8
75.7
89.4
59.0
58.8
71.9
76.6
Appendix
Table 18: De-recognition conditions of Table 3 under parsed-only scoring (items without a parsed verdict excluded from the denominator). Conditions abbreviated as in Table 17 . KGR is unchanged by the convention and is omitted.
Detection
KGR
Authentic accuracy
Model
Full
Mirror
Crop
b / c
p
Full
Mirror
Crop
Full
Mirror
Crop
Qwen3.5-122B-A10B
59.5
57.6
66.1
193/117
< 0.0001
38.6
29.5
40.5
77.4
60.5
55.7
Kimi-K3
61.2
57.8
81.4
275/62
< 0.0001
43.7
36.6
51.9
92.2
84.2
74.3
GPT-5.6
64.1
62.6
66.3
153/120
0.0526
44.9
40.5
45.0
91.6
82.1
82.4
Gemini-3.8-Flash
79.8
80.9
80.7
91/93
0.9413
64.0
55.5
65.5
88.2
68.1
80.2
Qwen3.5-35B-A3B
45.6
47.0
50.2
180/151
0.1237
28.8
24.3
30.7
58.4
35.0
28.5
Appendix
Table 19: Detection, KGR, and authentic accuracy under Full, Mirror, and Crop for the 902 peripheral forgeries in Table 7 and all 474 authentic images. b/c counts forgeries detected only under Crop or only under Mirror, and p is the exact McNemar test between them. Because this population includes visible-text forgeries (K4), for which mirroring itself changes the text, the paired recognition analysis in Table 3 is restricted to the people family.
Contrast, recorded
Contrast, measured
Swap coef. vs. K0
Model
n
log-odds
p
log-odds
p
base
full
shrink
Gemini-3.8-Flash
248
0.688
0.1188
0.946
0.0450
4.580
4.623
− 0.9%
GLM-5.3-Flash
239
1.449
0.0012
1.605
0.0020
3.072
3.056
0.5%
Kimi-K3
248
1.430
0.0019
1.552
0.0021
4.556
4.165
8.6%
Kimi-K2.6
248
1.596
< 0.0001
1.644
0.0001
3.581
3.260
9.0%
Qwen3.5-122B-A10B
242
1.520
0.0004
1.548
0.0005
2.046
2.132
− 4.2%
Appendix
Table 20: Per-model logistic regressions predicting detection from edit condition, with domain fixed effects, log footprint, and both pixel-detector scores; standard errors are clustered by seed. We report the swap-versus-erase log-odds contrast under both footprint definitions, and the swap coefficient relative to K0 before and after adding the covariates. “Shrink” is the change in that coefficient from the base to the full specification; negative values indicate an increase. Rows follow Table 2 .
Model
Erase (%)
Swap (%)
Ratio
E4 (%)
Outcome
Gemini-3.8-Flash
53.0
71.8
0.739
7.6
intermediate
GPT-5.6
17.4
52.6
0.331
0.0
collapse
GLM-5.3-Flash
32.6
65.4
0.498
2.3
collapse
Kimi-K3
30.3
70.5
0.430
2.3
collapse
Kimi-K2.6
21.2
59.0
0.360
1.5
collapse
Qwen3.5-122B-A10B
29.5
61.5
0.480
0.0
collapse
Appendix
Table 21: Application of the preregistered rule (default probe, strict scoring). Ratio: erase detection over swap detection. E4: share of the erase items whose verdict states the contradiction. Models detecting fewer than one swap in ten are listed without a ratio. Rows in the order of Table 2 .
Forensic analysis of AI-edited images requires more than binary real-versus-fake prediction: a useful system should localize the edit, identify its semantic type, and ground its decisions in visual evidence. Existing image-forensics datasets typically emphasize detection or localization, while reasoning-supervised vision-language datasets rarely target image manipulation and often rely on LLM-generated rationales whose faithfulness is difficult to verify. We introduce EditSleuth, a dataset of 257,725 image-edit triplets constructed from existing image-editing corpora for grounded image-edit forensic reasoning. Each example includes an edited image, its source image, a binary edit mask, a 12-class edit taxonomy label, a difficulty score, and a six-step reasoning chain. EditSleuth chains are generated deterministically from triplet-grounded upstream artifacts, with each statement tied to a specific computable source of evidence. Our analysis reveals that a naive four-component difficulty formulation suffers from a rank-2 correlation collapse among magnitude features; a simplified three-component formulation substantially increases score dispersion on both Pico-Banana and MagicBrush. Difficulty also varies meaningfully within most edit categories, indicating that the score is not a proxy for edit type. As an initial learning study, we fine-tune Qwen2-VL-2B with LoRA and find that chain-as-target supervision matches a label-only baseline on classification accuracy among parseable answers, while additionally yielding grounded explanatory prose that label-only supervision cannot produce. We release the dataset, the deterministic construction pipeline, and pilot training scripts.
With the rapid advancement of generative models, powerful image editing methods now enable diverse and highly realistic image manipulations that far surpass traditional deepfake techniques, posing new challenges for manipulation detection. Existing image manipulation detection and localization (IMDL) benchmarks suffer from limited content diversity, narrow generative-model coverage, and insufficient interpretability, which hinders the generalization and explanation capabilities of current manipulation detection methods. To address these limitations, we introduce ManipBench, a large-scale benchmark for image manipulation detection and localization focusing on AI-edited images. ManipBench contains over 450K manipulated images produced by 25 state-of-the-art image editing models across 12 manipulation categories, among which 100K images are further annotated with bounding boxes, judgment cues, and textual explanations to support interpretable detection. Building upon ManipBench, we propose ManipShield, an all-in-one model based on a Multimodal Large Language Model (MLLM) that leverages contrastive LoRA fine-tuning and task-specific decoders to achieve unified image manipulation detection, localization, and explanation. Extensive experiments on ManipBench and several public datasets demonstrate that ManipShield achieves state-of-the-art performance and exhibits strong generality to unseen manipulation models. The project is available at https://github.com/IntMeGroup/ManipShield.
Zitong Xu, Huiyu Duan, Xiaoyu Wang +4
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University · University of Electronic and Science Technology of China
Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or vision--language model judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 Scalable Vector Graphics (SVG) graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.