Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or vision--language model judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 Scalable Vector Graphics (SVG) graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.
Figures & tables
Figure 1: Overview of VeriEdit-Bench. Examples from SVG graphics, charts, web interfaces, and slides; each task specifies a controlled edit that preserves the surrounding context.
Figure 2: From editable sources to verifiable evaluation. (a) Source edits, large language model (LLM) screening, and author review curate tasks; repairs are re-rendered and rechecked. (b) The editor receives only the input image and instruction. Input, target, edit region, and output yield four scores and their harmonic mean, shown for a GPT-Image-2 output.
Figure 3: Dataset composition. Left: case shares and case/source counts. Right: operation counts, sorted within each asset type on a shared scale; instruction variants are grouped by operation.
Metric
GPT- Image-2
FireRed- Edit
Qwen-Image- Edit-2511
Qwen-Image- Edit
FLUX.2- dev
BAGEL
Step1X- Edit
FLUX.1- Kontext
FLUX.2- klein
OmniGen2
Instruct Pix2Pix
Fidelity F
37.10
13.42
12.22
9.38
8.16
6.66
5.24
3.78
4.85
5.16
2.36
Preservation P
84.52
85.09
87.16
82.06
82.61
68.75
82.78
91.64
78.81
60.52
39.19
Localization L
56.92
44.29
46.64
44.00
43.30
42.38
40.09
28.16
38.81
35.52
42.30
Magnitude M
84.09
68.76
66.92
60.53
61.65
61.38
59.47
46.69
60.34
56.80
64.16
Cell-mean H
47.13
21.85
20.47
16.26
13.11
10.81
9.78
8.78
8.42
7.32
3.83
Table 1: Performance on 1,740 cases (0–100, higher is better). Models are columns; the four axes are case-weighted means. Cell-mean H averages per-case harmonic composites within each of the 22 asset–operation cells and weights cells equally. CI denotes the 95% bootstrap confidence interval for cell-mean H . Bold : best overall; underlined : best open-weight, per metric.
Figure 4: Fixed-operation comparison on 455 recoloring cases and 5,005 outputs from Table 1 . Left: all eleven models’ mean H by asset type. The color bar shows the shared score scale. Right: the FLUX.2-dev–FireRed-Edit gap with source-group bootstrap intervals; hollow diamonds give the gap after equally weighting edits covering at most 10% and more than 10% of the image.
Figure 5: Complete images and per-output scores. PaintBench rows average CIE76 tolerances 0–10; shading highlights localization and magnitude differences. Larger views and full accuracy curves appear in Appendix Figures 19 and 20 .
Figure 6: Attribute execution and precise editing on identical outputs. Left: all 6,831 eligible outputs, showing color attainment against preservation. Transparency reduces overlap; hollow points mark GPT-Image-2. Right: component scores below 50 among the color-score ≥50 outputs of GPT-Image-2, FLUX.2-dev, and FireRed-Edit; the denominators are printed beside their names. Components can overlap. Appendix A.10 reports every model and alternative color cutoffs.
Metric
GPT- Image-2
FireRed- Edit
Qwen-Image- Edit-2511
Qwen-Image- Edit
FLUX.2- dev
BAGEL
Step1X- Edit
FLUX.1- Kontext
FLUX.2- klein
OmniGen2
Instruct Pix2Pix
Color attainment
64.92
2.98
4.65
2.25
6.41
1.00
0.85
0.26
3.40
0.60
0.45
Table 2: Independent color-attainment experiment on 621 color-editing cases. Values average the covered asset–operation cells (0–100). Color attainment is not a component of the four-axis composite. Model order matches Table 1 ; bold : best model; underlined : best open-weight model.
Figure 7: Fixed-output ablations. Left: changing the aggregation of identical axis scores changes the reward for unbalanced edits. Right: rank correlation across all eleven models after changing the scoring configuration. Complete model scores and single-parameter sensitivity are in Appendix A.5 .
Appendix figures & tables41 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Four-axis composite scores for all eleven models across the 22 asset–operation combinations. Each cell averages the corresponding cases.
Metric
GPT- Image-2
FireRed- Edit
Qwen-Image- Edit-2511
Qwen-Image- Edit
FLUX.2- dev
BAGEL
Step1X- Edit
FLUX.1- Kontext
FLUX.2- klein
OmniGen2
Instruct Pix2Pix
Cell-mean H
47.13
21.85
20.47
16.26
13.11
10.81
9.78
8.78
8.42
7.32
3.83
Case-mean H
44.90
17.76
16.29
11.49
11.34
9.09
7.31
5.73
7.02
6.12
3.66
Appendix
Table 3: Effect of case weighting on the same 1,740 cases. Both rows average per-case harmonic composites. Cell-mean H weights the 22 asset–operation cells equally; case-mean H weights individual cases equally. Model order follows Table 1 ; bold marks the best model and underlining the best open-weight model.
Figure 9: Full eleven-model axis profiles across all 22 asset–operation cells. Each panel averages one axis within each cell, with a shared 0–100 scale.
Figure 10: SVG recoloring. Only the chimney and door should change color; the roof and surrounding content remain protected.
Figure 11: Chart palette change. The four bar series should take the new muted palette while axes, labels, and surrounding content remain protected. GPT-Image-2 applies the palette; FireRed-Edit produces a partial recolor with hex-code text artifacts; FLUX.1-Kontext leaves the chart unchanged.
Figure 12: Web heading hiding. The specified heading should disappear while its space and all surrounding content remain unchanged. GPT-Image-2 and FireRed-Edit remove the heading; FLUX.1-Kontext retains it.
Figure 13: SVG element hiding. The large teal body should disappear while every other element remains unchanged. GPT-Image-2 removes it; FireRed-Edit instead recolors it; FLUX.1-Kontext leaves it in place.
Figure 14: Detected-change area relative to the allowed edit region. Top: each model’s distribution across ratio bins. Bottom: median area ratio (left) and mean composite H (right) for each model, grouped by allowed-region size. All 19,140 outputs enter the analysis, including zero-score outputs.
Model
Harmonic
Geometric
Arithmetic
GPT-Image-2
47.13
52.26
66.55
FireRed-Edit
21.85
26.65
54.60
Qwen-Image-Edit-2511
20.47
25.85
54.71
Qwen-Image-Edit
16.26
20.11
50.53
FLUX.2-dev
13.11
17.11
49.47
Appendix
Table 4: Aggregation of identical four-axis measurements on all 19,140 outputs. Each column applies the same 22-cell macro average; bold marks the best model within that column. The columns represent different scoring choices, not model improvements.
Model
Default
CIEDE2000
Half: bicubic
Half: nearest
Tighter
Looser
GPT-Image-2
47.13
50.92
45.03
45.64
40.81
51.31
FireRed-Edit
21.85
26.15
20.52
20.82
19.45
25.20
Qwen-Image-Edit-2511
20.47
25.68
18.97
19.44
17.56
24.92
Qwen-Image-Edit
16.26
19.72
15.14
15.43
14.34
18.63
FLUX.2-dev
13.11
18.36
12.19
12.47
10.22
16.44
Appendix
Table 5: Macro composites under six scoring configurations on identical outputs. Bold marks the best model in each column. Color distance, prediction resolution, and tolerance choices can change absolute scores and close model orderings.
Parameter
Default
Settings
Fidelity tolerances
{5,10,20}
1–4
Preservation tolerances
{2,5,10,20}
5–8
Effective-change threshold
6
9–12
Global-fit reference-separation cap
8
13–16
Minimum relative change / fit residual
0.05
17–20
Minimum local coherence
0.25
21–24
Appendix
Table 6: Single-parameter sweep. Each default is multiplied by 0.5, 0.75, 1.5, and 2, in that order. Setting numbers identify the points in Figure 15 . All other constants remain fixed.
Figure 15: Scoring sensitivity on identical outputs. Left: composite scores of the five models leading under the default configuration as fidelity tolerances are scaled; a multiplier of one is the default. Right: eleven-model rank correlation for each of the 44 single-parameter settings. All other parameters and predictions stay fixed.
Figure 16: Response to controlled rendered changes on 101 fixed cases. Outputs are clip(I+α(T−I),0,255) , using six change multipliers and the reference target at α=1 . Curves average cases equally; color attainment covers the same 50 eligible cases at every setting. α controls pixel-change amplitude, not the fraction of a semantic task completed.
α
F
P
L
M
H
C
0.3
5.85
100.00
94.02
70.49
16.57
0.00
0.5
14.05
100.00
98.16
80.80
33.55
0.00
0.7
27.86
100.00
99.09
91.40
52.59
11.46
0.85
50.05
100.00
99.69
99.11
74.74
20.30
1
100.00
100.00
100.00
100.00
100.00
100.00
1.15
63.42
100.00
100.00
99.36
82.73
41.80
Appendix
Table 7: Complete means for controlled change amplitudes. F, P, L, M and H each cover 101 cases; color attainment C covers the same 50 cases for every multiplier. No best-model marking applies to constructed inputs.
Constructed output
F
P
L
M
Half of target-region pixels
50.17
100.00
66.65
75.03
Correct target; background RGB +20
93.51
75.89
54.86
100.00
Half change amplitude
19.89
100.00
93.72
82.24
1.5 times change amplitude
49.07
99.84
100.00
93.51
Target, JPEG quality 95
90.10
98.89
87.21
99.73
Target, Gaussian σ=0.4
99.99
99.61
98.89
99.99
Appendix
Table 8: Controlled changes on 110 fixed development cases (five per asset–operation cell). Means retain all 110 cases per row. Shifted differences modify the source by a translated target-minus-source field; whole-image shifts also move protected content.
Construction
F
P
L
M
H
Exact target
100.00
100.00
100.00
100.00
100.00
Unchanged input
0.00
100.00
0.00
0.00
0.00
Omit one group
49.48
100.00
69.79
75.97
62.36
Edit another group
0.00
71.66
0.00
0.00
0.00
Edit an extra group
96.40
71.67
73.45
99.92
71.44
Swap target colors
2.89
100.00
99.90
68.80
6.78
Appendix
Table 9: Source-rendered controls on eight fixed assets (two per type). Values average each per-asset score equally. Groups are defined by source color declarations; they need not be semantic objects.
Condition
First trigger
Independent
Empty region
0
0
Negligible change
1122
1122
Blur match
95
997
Local cancellation
114
117
Global color transformation
61
384
Passes all checks
17748
—
Appendix
Table 10: Shared-check triggers on all 19,140 outputs. First-trigger counts follow the execution order and are mutually exclusive. Independent counts evaluate each condition separately and can overlap. Passing is not proof of correct target appearance.
Model
Rejected
H=0
ΔH
GPT-Image-2
1.3
21.0
0.00
FireRed-Edit
5.5
52.5
0.00
Qwen-Image-Edit-2511
6.7
54.7
0.02
Qwen-Image-Edit
8.0
65.9
0.01
FLUX.2-dev
7.1
66.3
0.01
BAGEL
6.4
73.6
0.00
Appendix
Table 11: Effect of the shared check. Rejected and H=0 are percentages of 1,740 outputs per model; ΔH is the change in the 22-cell mean when the check is removed.
Model
H=0
Check rejected
Zero P
Zero F only
Other
GPT-Image-2
365
23
20
322
0
FireRed-Edit
914
96
13
800
5
Qwen-Image-Edit-2511
951
116
15
816
4
Qwen-Image-Edit
1146
139
21
982
4
FLUX.2-dev
1154
124
9
1016
5
BAGEL
1281
111
29
1141
0
Appendix
Table 12: Composition of zero composites among each model’s 1,740 outputs. Categories are mutually exclusive and assigned in order: rejected by the shared check ( F=L=M=0 ); zero preservation ( P=0 ); zero fidelity only ( F=0 with L,M>0 ), meaning agreement with the target does not exceed the unchanged-input baseline; and other combinations. The rejected total equals the first-trigger rejections in Table 10 .
Model
F
P
L
M
H
ΔH
GPT-Image-2
37.10
84.52
56.97
84.60
47.14
+0.002
FireRed-Edit
13.42
85.09
44.57
70.78
21.86
+0.005
Qwen-Image-Edit-2511
12.23
87.16
47.21
69.29
20.49
+0.015
Qwen-Image-Edit
9.39
82.06
44.78
63.23
16.27
+0.006
FLUX.2-dev
8.17
82.61
44.26
64.36
13.12
+0.012
BAGEL
6.66
68.75
43.21
63.84
10.81
+0.003
Appendix
Table 13: Without the shared check. Identical 19,140 outputs; axes are case means and H is the 22-cell macro mean. ΔH is relative to the fixed default, with all other rules unchanged.
Model
F
P
L
M
H
ΔH
GPT-Image-2
37.10
84.52
53.17
84.09
44.75
-2.389
FireRed-Edit
13.42
85.09
39.80
68.76
20.03
-1.824
Qwen-Image-Edit-2511
12.22
87.16
43.23
66.92
19.04
-1.434
Qwen-Image-Edit
9.38
82.06
39.58
60.53
14.40
-1.864
FLUX.2-dev
8.16
82.61
38.38
61.65
12.08
-1.029
BAGEL
6.66
68.75
34.34
61.38
9.48
-1.327
Appendix
Table 14: Without the localization background cap. Identical 19,140 outputs; axes are case means and H is the 22-cell macro mean. ΔH is relative to the fixed default, with all other rules unchanged.
Model pair
ΔH
95% interval
Gap change interval
GPT-Image-2 / FireRed-Edit
+24.72
[22.65, 26.66]
[-0.91, -0.21]
FireRed-Edit / Qwen-Image-Edit-2511
+0.99
[-0.21, 2.20]
[-0.69, -0.06]
Qwen-Image-Edit-2511 / Qwen-Image-Edit
+4.64
[3.41, 5.98]
[-0.03, 0.92]
Qwen-Image-Edit / FLUX.2-dev
+2.32
[0.85, 3.83]
[-1.32, -0.32]
FLUX.2-dev / BAGEL
+2.60
[1.03, 4.23]
[-0.26, 0.81]
BAGEL / Step1X-Edit
+0.53
[-0.97, 1.86]
[-1.03, 0.02]
Appendix
Table 15: Pairwise gaps without the localization cap. Pairs follow the default ordering; intervals use 10,000 paired source-group resamples. The final column measures the change in each gap relative to the default.
Figure 17: Shared-check rejection rates for every model–operation combination. Each cell uses all outputs in that combination as its denominator. Values are percentages, rounded for display. Rejection leaves preservation independently measured.
Figure 18: Generation controls on their actual subsets: four seeds on 94 cases for FireRed-Edit and FLUX.2-dev, and two instruction forms on 169 translation cases for FLUX.2-klein and FLUX.1-Kontext. Seed scores use cell-macro aggregation; wording scores use case-weighted means.
Model
Seed 0
Seed 1
Seed 2
Seed 3
FireRed-Edit
23.60
24.12
22.18
23.85
FLUX.2-dev
17.39
19.79
17.11
16.50
Appendix
Table 16: All four seed runs on the same 94 cases. Scores average the covered operation cells equally; bold marks the larger score within each seed.
Model
Pixel instruction
With percentage
Change
FLUX.2-klein
2.37
2.48
+0.11
FLUX.1-Kontext
9.97
9.55
-0.42
Appendix
Table 17: Matched wording control on 169 cases per model. The same source image and requested edit are retained; the alternative adds displacement as a percentage of the canvas size. Scores are case-mean H; bold marks the larger model score for each instruction form.
Asset
Model
H [95% CI]
F
P
L
M
SVG (82)
GPT-Image-2
76.78 [70.18, 82.43]
64.99
87.73
85.47
95.78
FireRed-Edit
15.53 [10.55, 20.93]
8.78
88.67
69.30
74.07
Qwen-Image-Edit-2511
15.98 [10.21, 21.82]
9.23
89.81
66.88
80.64
FLUX.2-dev
8.43 [4.53, 12.61]
5.48
70.88
59.28
76.37
Qwen-Image-Edit
11.99 [7.27, 17.27]
7.74
76.24
55.87
71.59
BAGEL
6.35 [3.06, 9.84]
3.39
70.65
56.15
69.89
Appendix
Table 18: Recoloring on the same operation across asset types. Entries give mean H [95% source-cluster bootstrap interval] and the four mean axes. All 455 cases and eleven models are retained. Bold marks the highest mean per asset and measurement.
Coarse area strata
Fine shared area strata
Asset
≤10%
>10%
≤1%
1–5%
5–10%
>25%
SVG
51 / 28
31 / 19
19 / 14
23 / 16
9 / 8
21 / 14
Chart
30 / 8
24 / 6
8 / 2
14 / 4
8 / 2
8 / 2
Web
151 / 36
32 / 6
57 / 17
60 / 14
34 / 7
18 / 3
Slide
105 / 29
31 / 13
53 / 15
27 / 10
25 / 7
18 / 9
Appendix
Table 19: Cases / source assets in the recoloring area strata. Coarse strata retain all 455 cases; the four shared fine strata retain 402. An asset can contribute cases to more than one stratum.
Two coarse strata (all 455 cases)
Four fine strata (402 cases)
Model
SVG
Chart
Web
Slide
SVG
Chart
Web
Slide
GPT-Image-2
78.19
70.56
30.48
62.33
75.29
64.74
28.13
56.35
FireRed-Edit
16.63
5.12
10.30
11.99
14.97
4.52
9.07
10.69
Qwen-Image-Edit-2511
17.78
6.81
10.59
15.44
14.59
5.28
11.78
17.35
FLUX.2-dev
8.88
18.15
12.85
12.08
7.11
16.89
9.61
12.49
Appendix
Table 20: Mean H after equal weighting of the area strata defined in Table 19 . Best model per asset and weighting is bold. These comparisons describe area-standardized performance, not a causal effect of asset type.
Color ≥32
Color ≥50
Color ≥80
Model
n
F
P
L
M
n
F
P
L
M
n
F
P
L
M
GPT-Image-2
449
134
60
95
5
401
97
50
71
5
270
43
27
44
0
FireRed-Edit
20
14
5
9
0
13
9
4
5
0
7
4
1
2
0
Qwen-Image-Edit-2511
38
27
7
15
2
30
21
6
11
2
18
12
3
5
0
FLUX.2-dev
52
32
14
21
2
37
22
12
16
1
15
8
2
6
0
Appendix
Table 21: Conditional diagnosis on the same 621 color cases per model. For each color-attainment cutoff c , n is the number of outputs with color score ≥c ; subsequent columns count F/P/L/M<50 . These are descriptive slices, not new pass criteria.
Model
Attainment
Zero (%)
≥50 (%)
L
M
L,M>0 (%)
GPT-Image-2
61.47
8.2
64.6
69.94
93.46
99.5
FireRed-Edit
3.04
74.7
2.1
58.96
73.81
95.2
Qwen-Image-Edit-2511
5.48
68.8
4.8
58.50
75.55
96.1
Qwen-Image-Edit
2.32
74.7
1.3
51.54
65.67
93.9
FLUX.2-dev
6.44
63.4
6.0
60.13
79.47
97.7
BAGEL
0.93
82.8
0.6
52.19
67.30
98.2
Appendix
Table 22: Color attainment and edit execution on the same 621 color-editing cases. Columns give the case-mean color attainment (Table 2 instead averages asset–operation cells), the percentage of outputs with zero attainment and with attainment of at least 50, the case-mean localization and magnitude, and the percentage of outputs with both L>0 and M>0 .
Model
mIoU
Edit acc.
Pres. acc.
F
P
L
M
GPT-Image-2
8.20
26.67
77.96
36.89
90.36
50.99
84.84
FireRed-Edit
2.80
16.11
82.19
15.93
88.66
50.97
72.32
Qwen-Image-Edit-2511
2.48
17.11
81.83
15.36
89.01
47.74
70.61
Qwen-Image-Edit
2.18
15.38
78.57
13.88
87.18
45.39
64.98
FLUX.2-dev
1.36
9.11
76.45
9.57
86.10
45.30
69.70
Appendix
Table 23: Same aligned outputs, official PaintBench measurements and our four-axis measurements. Means over 101 fixed diagnostic cases per model (0–100), covering 11 models. These are measurements of this panel, not scores on PaintBench's original benchmark. Best values are bold within each column.
Pairs
Difference
Min.
Q1
Median
Q3
Max.
All pairs
∣ΔmIoU∣
0.00
0.08
1.00
6.30
96.00
All pairs
Maximum curve gap
0.00
38.70
69.29
92.27
100.00
All pairs
max(∣ΔL∣,∣ΔM∣)
0.00
12.25
27.49
47.32
100.00
Matched pairs
∣ΔmIoU∣
0.00
0.00
0.00
0.02
0.85
Matched pairs
Maximum curve gap
0.00
0.02
0.40
1.10
1.99
Matched pairs
max(∣ΔL∣,∣ΔM∣)
0.00
7.06
23.79
44.91
100.00
Appendix
Table 24: Complete difference distributions on the fixed 607-case panel: 33,385 same-case pairs, including 416 matched pairs. Maximum curve gap considers both accuracy curves at all eleven tolerances. All differences are on a 0–100 scale.
dI
dA
dLM
Similar pairs
Differing pairs
Cases
0.5
1
10
213
123
48
0.5
1
20
213
92
38
0.5
1
30
213
71
31
0.5
2
10
379
219
66
0.5
2
20
379
164
56
Appendix
Table 25: Sensitivity to the numerical similarity window. dI bounds the mIoU gap; dA bounds each mean edit/preservation-accuracy gap; dLM is the minimum localization or magnitude gap. Counts concern 5,555 same-case model pairs on the 101-case panel. The stricter rule used in the main text bounds each accuracy gap at every tolerance by two points, with dI=1 : 78 pairs qualify, and 28 differ by at least 20 points in localization or magnitude, spanning 13 cases.
Figure 19: Complete task and instruction, with the input, reference target, and FLUX.2-dev/Step1X-Edit outputs used in the comparison.
Figure 20: Complete PaintBench accuracy curves for the FLUX.2-dev/Step1X-Edit pair in Figure 5 . Each curve retains all eleven CIE76 tolerances. The maximum between-model accuracy gaps are 0.07 points for editing and 1.87 points for preservation, both within the two-point matching bound at every tolerance.
Asset
Operation
Original H
2511 H
n
Δ [95% CI]
Chart
Color
3.22
6.48
54
+3.26 [-0.34, +7.60]
Font size
7.11
20.32
42
+13.21 [+2.18, +21.57]
Grid
9.75
9.27
48
-0.49 [-1.44, -0.13]
Line width
9.68
17.50
96
+7.81 [+3.85, +13.98]
Palette
4.95
8.62
76
+3.66 [-2.47, +8.56]
Slide
Border
2.94
4.82
78
+1.87 [+0.08, +5.15]
Appendix
Table 26: Qwen-Image-Edit version comparison by operation. The two H columns use all cases in each cell. All 1,740 cases use identical actual source images and instructions; Δ is Qwen-Image-Edit-2511 minus Qwen-Image-Edit, with pointwise 95% source-group BCa intervals. Bold marks the higher full-set mean, not a significance test. The final row weights cells equally.
Model pair
ΔH
95% interval
GPT-Image-2 / FireRed-Edit
+25.28
[23.29,27.38]
FireRed-Edit / Qwen-Image-Edit-2511
+1.38
[0.02,2.61]
Qwen-Image-Edit-2511 / Qwen-Image-Edit
+4.21
[2.95,5.46]
Qwen-Image-Edit / FLUX.2-dev
+3.15
[1.72,4.65]
FLUX.2-dev / BAGEL
+2.30
[0.53,3.91]
BAGEL / Step1X-Edit
+1.03
[−0.58,2.70]
Appendix
Table 27: Paired gaps in cell-mean H between adjacent models of Table 1 under the default scoring configuration. ΔH is the first model minus the second; intervals are 95% BCa intervals from 10,000 source-group bootstrap draws shared across models. An interval excluding zero separates the pair.
Figure 21: Source-level descriptors. Each point is one source; horizontal bars mark medians. Multiple edits of a source contribute only one point. Color entropy describes palette dispersion, and edge density describes visible detail under fixed preprocessing.
Model
Checkpoint
Revision
Steps
Guidance
GPT-Image-2
—
API
—
—
FireRed-Edit
FireRed-Edit-1.1
3bc3f2a1
40
True CFG 4.0
Qwen-Image-Edit-2511
Qwen-Image-Edit-2511
6f3ccc0b
40
True CFG 4.0
FLUX.2-dev
FLUX.2-dev
26afe3a7
50
4.0
Qwen-Image-Edit
Qwen-Image-Edit
ac7f9318
50
True CFG 4.0
BAGEL
BAGEL-7B-MoT
5019f57d
50
T/I: 4.0/1.5
Appendix
Table 28: Inference settings for the evaluated editors. Checkpoint gives the exact public release evaluated; for open-weight models, Revision denotes the public repository commit. Guidance values follow each model’s own parameterization; T/I denotes text/image guidance.
Design assets
Verification
Dataset
Other visual domains
SVG artwork
Charts
Web
Slides
Source target
Regional pixels
MagicBrush ( Zhang et al., 2023 )
Natural images
—
—
—
—
—
—
GIE-Bench ( Qian et al., 2025 )
Photographs; illustrations
—
—
—
—
—
—
PaintBench ( Xu et al., 2026 )
Shape canvases; network graphs †
—
✓†
—
—
✓
✓
StructBench ( Zhuo et al., 2025 )
Math, science, puzzles, graphs, tables
—
✓
—
—
✓
—
FigEdit ( Li et al., 2025 )
—
—
✓
—
—
✓
—
Appendix
Table 29: Asset coverage, editing operations, and verification in related datasets. ✓ indicates coverage documented in the paper’s task taxonomy or examples; — means not explicitly reported there. ✓† denotes PaintBench’s TinyGrafixBench extension. Presence does not imply equal task counts or coverage of every domain–operation combination.
While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (R2=0.91, p<0.001). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.
Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7
Nanyang Technological University · StepFun · Southeast University
Diffusion-based image editing has achieved strong visual fidelity under natural language instructions, yet most existing systems still operate at the level of surface instruction following, without reasoning about the implicit contextual constraints embedded in real user requests. This often leads to visually plausible but logically inconsistent edits. In this work, we introduce RE-Edit, a benchmark for REasoning-aware image Editing that evaluates image editing systems across five complementary reasoning dimensions: physical, environmental, cultural, causal, and referential. RE-Edit comprises 1,000 carefully curated samples, each designed such that visual plausibility alone is insufficient and correct editing requires satisfying implicit logical constraints. To support fine-grained analysis, we establish dimension-aligned evaluation criteria and conduct a comprehensive study of ten open-source and two commercial image editing models. Our results show that even advanced systems frequently struggle with implicit multi-dimensional reasoning despite producing high-quality visuals. We further present a lightweight reasoning-guided post-edit baseline as an initial exploration, illustrating how inserting explicit reasoning can help mitigate such failures in a model-agnostic manner.