Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets

Muyao Wang; Chen Zhu; Shiqi Yang; DongHyun Hwan; Hideki Nakayama

cs.CVOct 3, 2026

Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets

Authors: Muyao Wang, Chen Zhu, Shiqi Yang, DongHyun Hwan, Hideki Nakayama

Organizations: SB Intuitions · University of Tokyo

Abstract

Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or vision--language model judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 Scalable Vector Graphics (SVG) graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.

Figures & tables

Figure 1: Overview of VeriEdit-Bench. Examples from SVG graphics, charts, web interfaces, and slides; each task specifies a controlled edit that preserves the surrounding context.

Figure 2: From editable sources to verifiable evaluation. (a) Source edits, large language model (LLM) screening, and author review curate tasks; repairs are re-rendered and rechecked. (b) The editor receives only the input image and instruction. Input, target, edit region, and output yield four scores and their harmonic mean, shown for a GPT-Image-2 output.

Figure 3: Dataset composition. Left: case shares and case/source counts. Right: operation counts, sorted within each asset type on a shared scale; instruction variants are grouped by operation.


Metric	GPT- Image-2	FireRed- Edit	Qwen-Image- Edit-2511	Qwen-Image- Edit	FLUX.2- dev	BAGEL	Step1X- Edit	FLUX.1- Kontext	FLUX.2- klein	OmniGen2	Instruct Pix2Pix
Fidelity $F$	37.10	13.42	12.22	9.38	8.16	6.66	5.24	3.78	4.85	5.16	2.36
Preservation $P$	84.52	85.09	87.16	82.06	82.61	68.75	82.78	91.64	78.81	60.52	39.19
Localization $L$	56.92	44.29	46.64	44.00	43.30	42.38	40.09	28.16	38.81	35.52	42.30
Magnitude $M$	84.09	68.76	66.92	60.53	61.65	61.38	59.47	46.69	60.34	56.80	64.16
Cell-mean $H$	47.13	21.85	20.47	16.26	13.11	10.81	9.78	8.78	8.42	7.32	3.83

Table 1: Performance on 1,740 cases (0–100, higher is better). Models are columns; the four axes are case-weighted means. Cell-mean

H

averages per-case harmonic composites within each of the 22 asset–operation cells and weights cells equally. CI denotes the 95% bootstrap confidence interval for cell-mean

H

. Bold : best overall; underlined : best open-weight, per metric.

Figure 4: Fixed-operation comparison on 455 recoloring cases and 5,005 outputs from Table 1 . Left: all eleven models’ mean $H$ by asset type. The color bar shows the shared score scale. Right: the FLUX.2-dev–FireRed-Edit gap with source-group bootstrap intervals; hollow diamonds give the gap after equally weighting edits covering at most 10% and more than 10% of the image.

Figure 5: Complete images and per-output scores. PaintBench rows average CIE76 tolerances 0–10; shading highlights localization and magnitude differences. Larger views and full accuracy curves appear in Appendix Figures 19 and 20 .

Figure 6: Attribute execution and precise editing on identical outputs. Left: all 6,831 eligible outputs, showing color attainment against preservation. Transparency reduces overlap; hollow points mark GPT-Image-2. Right: component scores below 50 among the color-score $\geq 50$ outputs of GPT-Image-2, FLUX.2-dev, and FireRed-Edit; the denominators are printed beside their names. Components can overlap. Appendix A.10 reports every model and alternative color cutoffs.


Metric	GPT- Image-2	FireRed- Edit	Qwen-Image- Edit-2511	Qwen-Image- Edit	FLUX.2- dev	BAGEL	Step1X- Edit	FLUX.1- Kontext	FLUX.2- klein	OmniGen2	Instruct Pix2Pix
Color attainment	64.92	2.98	4.65	2.25	6.41	1.00	0.85	0.26	3.40	0.60	0.45

Table 2: Independent color-attainment experiment on 621 color-editing cases. Values average the covered asset–operation cells (0–100). Color attainment is not a component of the four-axis composite. Model order matches Table 1 ; bold : best model; underlined : best open-weight model.

Figure 7: Fixed-output ablations. Left: changing the aggregation of identical axis scores changes the reward for unbalanced edits. Right: rank correlation across all eleven models after changing the scoring configuration. Complete model scores and single-parameter sensitivity are in Appendix A.5 .

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Four-axis composite scores for all eleven models across the 22 asset–operation combinations. Each cell averages the corresponding cases.


Metric	GPT- Image-2	FireRed- Edit	Qwen-Image- Edit-2511	Qwen-Image- Edit	FLUX.2- dev	BAGEL	Step1X- Edit	FLUX.1- Kontext	FLUX.2- klein	OmniGen2	Instruct Pix2Pix
Cell-mean $H$	47.13	21.85	20.47	16.26	13.11	10.81	9.78	8.78	8.42	7.32	3.83
Case-mean $H$	44.90	17.76	16.29	11.49	11.34	9.09	7.31	5.73	7.02	6.12	3.66

Appendix

Table 3: Effect of case weighting on the same 1,740 cases. Both rows average per-case harmonic composites. Cell-mean

H

weights the 22 asset–operation cells equally; case-mean

H

weights individual cases equally. Model order follows Table 1 ; bold marks the best model and underlining the best open-weight model.

Figure 9: Full eleven-model axis profiles across all 22 asset–operation cells. Each panel averages one axis within each cell, with a shared 0–100 scale.

Figure 10: SVG recoloring. Only the chimney and door should change color; the roof and surrounding content remain protected.

Figure 11: Chart palette change. The four bar series should take the new muted palette while axes, labels, and surrounding content remain protected. GPT-Image-2 applies the palette; FireRed-Edit produces a partial recolor with hex-code text artifacts; FLUX.1-Kontext leaves the chart unchanged.

Figure 12: Web heading hiding. The specified heading should disappear while its space and all surrounding content remain unchanged. GPT-Image-2 and FireRed-Edit remove the heading; FLUX.1-Kontext retains it.

Figure 13: SVG element hiding. The large teal body should disappear while every other element remains unchanged. GPT-Image-2 removes it; FireRed-Edit instead recolors it; FLUX.1-Kontext leaves it in place.

Figure 14: Detected-change area relative to the allowed edit region. Top: each model’s distribution across ratio bins. Bottom: median area ratio (left) and mean composite

H

(right) for each model, grouped by allowed-region size. All 19,140 outputs enter the analysis, including zero-score outputs.


Model	Harmonic	Geometric	Arithmetic
GPT-Image-2	47.13	52.26	66.55
FireRed-Edit	21.85	26.65	54.60
Qwen-Image-Edit-2511	20.47	25.85	54.71
Qwen-Image-Edit	16.26	20.11	50.53
FLUX.2-dev	13.11	17.11	49.47

Appendix

Table 4: Aggregation of identical four-axis measurements on all 19,140 outputs. Each column applies the same 22-cell macro average; bold marks the best model within that column. The columns represent different scoring choices, not model improvements.


Model	Default	CIEDE2000	Half: bicubic	Half: nearest	Tighter	Looser
GPT-Image-2	47.13	50.92	45.03	45.64	40.81	51.31
FireRed-Edit	21.85	26.15	20.52	20.82	19.45	25.20
Qwen-Image-Edit-2511	20.47	25.68	18.97	19.44	17.56	24.92
Qwen-Image-Edit	16.26	19.72	15.14	15.43	14.34	18.63
FLUX.2-dev	13.11	18.36	12.19	12.47	10.22	16.44

Appendix

Table 5: Macro composites under six scoring configurations on identical outputs. Bold marks the best model in each column. Color distance, prediction resolution, and tolerance choices can change absolute scores and close model orderings.

Parameter	Default	Settings
Fidelity tolerances	$\{5,10,20\}$	1–4
Preservation tolerances	$\{2,5,10,20\}$	5–8
Effective-change threshold	6	9–12
Global-fit reference-separation cap	8	13–16
Minimum relative change / fit residual	0.05	17–20
Minimum local coherence	0.25	21–24

Appendix

Table 6: Single-parameter sweep. Each default is multiplied by 0.5, 0.75, 1.5, and 2, in that order. Setting numbers identify the points in Figure 15 . All other constants remain fixed.

Figure 15: Scoring sensitivity on identical outputs. Left: composite scores of the five models leading under the default configuration as fidelity tolerances are scaled; a multiplier of one is the default. Right: eleven-model rank correlation for each of the 44 single-parameter settings. All other parameters and predictions stay fixed.

Figure 16: Response to controlled rendered changes on 101 fixed cases. Outputs are

\mathrm{clip}(I+\alpha(T-I),0,255)

, using six change multipliers and the reference target at

\alpha=1

. Curves average cases equally; color attainment covers the same 50 eligible cases at every setting.

\alpha

controls pixel-change amplitude, not the fraction of a semantic task completed.

$\alpha$	F	P	L	M	H	C
0.3	5.85	100.00	94.02	70.49	16.57	0.00
0.5	14.05	100.00	98.16	80.80	33.55	0.00
0.7	27.86	100.00	99.09	91.40	52.59	11.46
0.85	50.05	100.00	99.69	99.11	74.74	20.30
1	100.00	100.00	100.00	100.00	100.00	100.00
1.15	63.42	100.00	100.00	99.36	82.73	41.80

Appendix

Table 7: Complete means for controlled change amplitudes. F, P, L, M and H each cover 101 cases; color attainment C covers the same 50 cases for every multiplier. No best-model marking applies to constructed inputs.

Constructed output	F	P	L	M
Half of target-region pixels	50.17	100.00	66.65	75.03
Correct target; background RGB +20	93.51	75.89	54.86	100.00
Half change amplitude	19.89	100.00	93.72	82.24
1.5 times change amplitude	49.07	99.84	100.00	93.51
Target, JPEG quality 95	90.10	98.89	87.21	99.73
Target, Gaussian $\sigma=0.4$	99.99	99.61	98.89	99.99

Appendix

Table 8: Controlled changes on 110 fixed development cases (five per asset–operation cell). Means retain all 110 cases per row. Shifted differences modify the source by a translated target-minus-source field; whole-image shifts also move protected content.

Construction	$F$	$P$	$L$	$M$	$H$
Exact target	100.00	100.00	100.00	100.00	100.00
Unchanged input	0.00	100.00	0.00	0.00	0.00
Omit one group	49.48	100.00	69.79	75.97	62.36
Edit another group	0.00	71.66	0.00	0.00	0.00
Edit an extra group	96.40	71.67	73.45	99.92	71.44
Swap target colors	2.89	100.00	99.90	68.80	6.78

Appendix

Table 9: Source-rendered controls on eight fixed assets (two per type). Values average each per-asset score equally. Groups are defined by source color declarations; they need not be semantic objects.

Condition	First trigger	Independent
Empty region	0	0
Negligible change	1122	1122
Blur match	95	997
Local cancellation	114	117
Global color transformation	61	384
Passes all checks	17748	—

Appendix

Table 10: Shared-check triggers on all 19,140 outputs. First-trigger counts follow the execution order and are mutually exclusive. Independent counts evaluate each condition separately and can overlap. Passing is not proof of correct target appearance.

Model	Rejected	H=0	$\Delta H$
GPT-Image-2	1.3	21.0	0.00
FireRed-Edit	5.5	52.5	0.00
Qwen-Image-Edit-2511	6.7	54.7	0.02
Qwen-Image-Edit	8.0	65.9	0.01
FLUX.2-dev	7.1	66.3	0.01
BAGEL	6.4	73.6	0.00

Appendix

Table 11: Effect of the shared check. Rejected and

H=0

are percentages of 1,740 outputs per model;

\Delta H

is the change in the 22-cell mean when the check is removed.

Model	$H=0$	Check rejected	Zero $P$	Zero $F$ only	Other
GPT-Image-2	365	23	20	322	0
FireRed-Edit	914	96	13	800	5
Qwen-Image-Edit-2511	951	116	15	816	4
Qwen-Image-Edit	1146	139	21	982	4
FLUX.2-dev	1154	124	9	1016	5
BAGEL	1281	111	29	1141	0

Appendix

Table 12: Composition of zero composites among each model’s 1,740 outputs. Categories are mutually exclusive and assigned in order: rejected by the shared check (

F=L=M=0

); zero preservation (

P=0

); zero fidelity only (

F=0

with

L,M>0

), meaning agreement with the target does not exceed the unchanged-input baseline; and other combinations. The rejected total equals the first-trigger rejections in Table 10 .

Model	$F$	$P$	$L$	$M$	$H$	$\Delta H$
GPT-Image-2	37.10	84.52	56.97	84.60	47.14	+0.002
FireRed-Edit	13.42	85.09	44.57	70.78	21.86	+0.005
Qwen-Image-Edit-2511	12.23	87.16	47.21	69.29	20.49	+0.015
Qwen-Image-Edit	9.39	82.06	44.78	63.23	16.27	+0.006
FLUX.2-dev	8.17	82.61	44.26	64.36	13.12	+0.012
BAGEL	6.66	68.75	43.21	63.84	10.81	+0.003

Appendix

Table 13: Without the shared check. Identical 19,140 outputs; axes are case means and

H

is the 22-cell macro mean.

\Delta H

is relative to the fixed default, with all other rules unchanged.

Model	$F$	$P$	$L$	$M$	$H$	$\Delta H$
GPT-Image-2	37.10	84.52	53.17	84.09	44.75	-2.389
FireRed-Edit	13.42	85.09	39.80	68.76	20.03	-1.824
Qwen-Image-Edit-2511	12.22	87.16	43.23	66.92	19.04	-1.434
Qwen-Image-Edit	9.38	82.06	39.58	60.53	14.40	-1.864
FLUX.2-dev	8.16	82.61	38.38	61.65	12.08	-1.029
BAGEL	6.66	68.75	34.34	61.38	9.48	-1.327

Appendix

Table 14: Without the localization background cap. Identical 19,140 outputs; axes are case means and

H

is the 22-cell macro mean.

\Delta H

is relative to the fixed default, with all other rules unchanged.

Model pair	$\Delta H$	95% interval	Gap change interval
GPT-Image-2 / FireRed-Edit	+24.72	[22.65, 26.66]	[-0.91, -0.21]
FireRed-Edit / Qwen-Image-Edit-2511	+0.99	[-0.21, 2.20]	[-0.69, -0.06]
Qwen-Image-Edit-2511 / Qwen-Image-Edit	+4.64	[3.41, 5.98]	[-0.03, 0.92]
Qwen-Image-Edit / FLUX.2-dev	+2.32	[0.85, 3.83]	[-1.32, -0.32]
FLUX.2-dev / BAGEL	+2.60	[1.03, 4.23]	[-0.26, 0.81]
BAGEL / Step1X-Edit	+0.53	[-0.97, 1.86]	[-1.03, 0.02]

Appendix

Table 15: Pairwise gaps without the localization cap. Pairs follow the default ordering; intervals use 10,000 paired source-group resamples. The final column measures the change in each gap relative to the default.

Figure 17: Shared-check rejection rates for every model–operation combination. Each cell uses all outputs in that combination as its denominator. Values are percentages, rounded for display. Rejection leaves preservation independently measured.

Figure 18: Generation controls on their actual subsets: four seeds on 94 cases for FireRed-Edit and FLUX.2-dev, and two instruction forms on 169 translation cases for FLUX.2-klein and FLUX.1-Kontext. Seed scores use cell-macro aggregation; wording scores use case-weighted means.

Model	Seed 0	Seed 1	Seed 2	Seed 3
FireRed-Edit	23.60	24.12	22.18	23.85
FLUX.2-dev	17.39	19.79	17.11	16.50

Appendix

Table 16: All four seed runs on the same 94 cases. Scores average the covered operation cells equally; bold marks the larger score within each seed.

Model	Pixel instruction	With percentage	Change
FLUX.2-klein	2.37	2.48	+0.11
FLUX.1-Kontext	9.97	9.55	-0.42

Appendix

Table 17: Matched wording control on 169 cases per model. The same source image and requested edit are retained; the alternative adds displacement as a percentage of the canvas size. Scores are case-mean H; bold marks the larger model score for each instruction form.

Asset	Model	$H$ [95% CI]	$F$	$P$	$L$	$M$
SVG (82)	GPT-Image-2	76.78 [70.18, 82.43]	64.99	87.73	85.47	95.78
	FireRed-Edit	15.53 [10.55, 20.93]	8.78	88.67	69.30	74.07
	Qwen-Image-Edit-2511	15.98 [10.21, 21.82]	9.23	89.81	66.88	80.64
	FLUX.2-dev	8.43 [4.53, 12.61]	5.48	70.88	59.28	76.37
	Qwen-Image-Edit	11.99 [7.27, 17.27]	7.74	76.24	55.87	71.59
	BAGEL	6.35 [3.06, 9.84]	3.39	70.65	56.15	69.89

Appendix

Table 18: Recoloring on the same operation across asset types. Entries give mean

H

[95% source-cluster bootstrap interval] and the four mean axes. All 455 cases and eleven models are retained. Bold marks the highest mean per asset and measurement.


	Coarse area strata		Fine shared area strata
Asset	$\leq 10\%$	$>10\%$	$\leq 1\%$	1–5%	5–10%	$>25\%$
SVG	51 / 28	31 / 19	19 / 14	23 / 16	9 / 8	21 / 14
Chart	30 / 8	24 / 6	8 / 2	14 / 4	8 / 2	8 / 2
Web	151 / 36	32 / 6	57 / 17	60 / 14	34 / 7	18 / 3
Slide	105 / 29	31 / 13	53 / 15	27 / 10	25 / 7	18 / 9

Appendix

Table 19: Cases / source assets in the recoloring area strata. Coarse strata retain all 455 cases; the four shared fine strata retain 402. An asset can contribute cases to more than one stratum.


	Two coarse strata (all 455 cases)				Four fine strata (402 cases)
Model	SVG	Chart	Web	Slide	SVG	Chart	Web	Slide
GPT-Image-2	78.19	70.56	30.48	62.33	75.29	64.74	28.13	56.35
FireRed-Edit	16.63	5.12	10.30	11.99	14.97	4.52	9.07	10.69
Qwen-Image-Edit-2511	17.78	6.81	10.59	15.44	14.59	5.28	11.78	17.35
FLUX.2-dev	8.88	18.15	12.85	12.08	7.11	16.89	9.61	12.49

Appendix

Table 20: Mean

H

after equal weighting of the area strata defined in Table 19 . Best model per asset and weighting is bold. These comparisons describe area-standardized performance, not a causal effect of asset type.


	Color $\geq 32$					Color $\geq 50$					Color $\geq 80$
Model	$n$	$F$	$P$	$L$	$M$	$n$	$F$	$P$	$L$	$M$	$n$	$F$	$P$	$L$	$M$
GPT-Image-2	449	134	60	95	5	401	97	50	71	5	270	43	27	44	0
FireRed-Edit	20	14	5	9	0	13	9	4	5	0	7	4	1	2	0
Qwen-Image-Edit-2511	38	27	7	15	2	30	21	6	11	2	18	12	3	5	0
FLUX.2-dev	52	32	14	21	2	37	22	12	16	1	15	8	2	6	0

Appendix

Table 21: Conditional diagnosis on the same 621 color cases per model. For each color-attainment cutoff

c

n

is the number of outputs with color score

\geq c

; subsequent columns count

F/P/L/M<50

. These are descriptive slices, not new pass criteria.

Model	Attainment	Zero (%)	$\geq 50$ (%)	$L$	$M$	$L,M>0$ (%)
GPT-Image-2	61.47	8.2	64.6	69.94	93.46	99.5
FireRed-Edit	3.04	74.7	2.1	58.96	73.81	95.2
Qwen-Image-Edit-2511	5.48	68.8	4.8	58.50	75.55	96.1
Qwen-Image-Edit	2.32	74.7	1.3	51.54	65.67	93.9
FLUX.2-dev	6.44	63.4	6.0	60.13	79.47	97.7
BAGEL	0.93	82.8	0.6	52.19	67.30	98.2

Appendix

Table 22: Color attainment and edit execution on the same 621 color-editing cases. Columns give the case-mean color attainment (Table 2 instead averages asset–operation cells), the percentage of outputs with zero attainment and with attainment of at least 50, the case-mean localization and magnitude, and the percentage of outputs with both

L>0

and

M>0


Model	mIoU	Edit acc.	Pres. acc.	$F$	$P$	$L$	$M$
GPT-Image-2	8.20	26.67	77.96	36.89	90.36	50.99	84.84
FireRed-Edit	2.80	16.11	82.19	15.93	88.66	50.97	72.32
Qwen-Image-Edit-2511	2.48	17.11	81.83	15.36	89.01	47.74	70.61
Qwen-Image-Edit	2.18	15.38	78.57	13.88	87.18	45.39	64.98
FLUX.2-dev	1.36	9.11	76.45	9.57	86.10	45.30	69.70

Appendix

Table 23: Same aligned outputs, official PaintBench measurements and our four-axis measurements. Means over 101 fixed diagnostic cases per model (0–100), covering 11 models. These are measurements of this panel, not scores on PaintBench's original benchmark. Best values are bold within each column.

Pairs	Difference	Q1	Median	Q3	Max.
All pairs	$\|\Delta\mathrm{mIoU}\|$	0.08	1.00	6.30	96.00
All pairs	Maximum curve gap	38.70	69.29	92.27	100.00
All pairs	$\max(\|\Delta L\|,\|\Delta M\|)$	12.25	27.49	47.32	100.00
Matched pairs	$\|\Delta\mathrm{mIoU}\|$	0.00	0.00	0.02	0.85
Matched pairs	Maximum curve gap	0.02	0.40	1.10	1.99
Matched pairs	$\max(\|\Delta L\|,\|\Delta M\|)$	7.06	23.79	44.91	100.00

Appendix

Table 24: Complete difference distributions on the fixed 607-case panel: 33,385 same-case pairs, including 416 matched pairs. Maximum curve gap considers both accuracy curves at all eleven tolerances. All differences are on a 0–100 scale.


$d_{I}$	$d_{A}$	$d_{LM}$	Similar pairs	Differing pairs	Cases
0.5	1	10	213	123	48
0.5	1	20	213	92	38
0.5	1	30	213	71	31
0.5	2	10	379	219	66
0.5	2	20	379	164	56

Appendix

Table 25: Sensitivity to the numerical similarity window.

d_{I}

bounds the mIoU gap;

d_{A}

bounds each mean edit/preservation-accuracy gap;

d_{LM}

is the minimum localization or magnitude gap. Counts concern 5,555 same-case model pairs on the 101-case panel. The stricter rule used in the main text bounds each accuracy gap at every tolerance by two points, with

d_{I}=1

: 78 pairs qualify, and 28 differ by at least 20 points in localization or magnitude, spanning 13 cases.

Figure 19: Complete task and instruction, with the input, reference target, and FLUX.2-dev/Step1X-Edit outputs used in the comparison.

Figure 20: Complete PaintBench accuracy curves for the FLUX.2-dev/Step1X-Edit pair in Figure 5 . Each curve retains all eleven CIE76 tolerances. The maximum between-model accuracy gaps are 0.07 points for editing and 1.87 points for preservation, both within the two-point matching bound at every tolerance.

Asset	Operation	Original $H$	2511 $H$	$n$	$\Delta$ [95% CI]
Chart	Color	3.22	6.48	54	+3.26 [-0.34, +7.60]
	Font size	7.11	20.32	42	+13.21 [+2.18, +21.57]
	Grid	9.75	9.27	48	-0.49 [-1.44, -0.13]
	Line width	9.68	17.50	96	+7.81 [+3.85, +13.98]
	Palette	4.95	8.62	76	+3.66 [-2.47, +8.56]
Slide	Border	2.94	4.82	78	+1.87 [+0.08, +5.15]

Appendix

Table 26: Qwen-Image-Edit version comparison by operation. The two

H

columns use all cases in each cell. All 1,740 cases use identical actual source images and instructions;

\Delta

is Qwen-Image-Edit-2511 minus Qwen-Image-Edit, with pointwise 95% source-group BCa intervals. Bold marks the higher full-set mean, not a significance test. The final row weights cells equally.

Model pair	$\Delta H$	95% interval
GPT-Image-2 / FireRed-Edit	$+25.28$	$[23.29,\,27.38]$
FireRed-Edit / Qwen-Image-Edit-2511	$+1.38$	$[0.02,\,2.61]$
Qwen-Image-Edit-2511 / Qwen-Image-Edit	$+4.21$	$[2.95,\,5.46]$
Qwen-Image-Edit / FLUX.2-dev	$+3.15$	$[1.72,\,4.65]$
FLUX.2-dev / BAGEL	$+2.30$	$[0.53,\,3.91]$
BAGEL / Step1X-Edit	$+1.03$	$[-0.58,\,2.70]$

Appendix

Table 27: Paired gaps in cell-mean

H

between adjacent models of Table 1 under the default scoring configuration.

\Delta H

is the first model minus the second; intervals are 95% BCa intervals from 10,000 source-group bootstrap draws shared across models. An interval excluding zero separates the pair.

Figure 21: Source-level descriptors. Each point is one source; horizontal bars mark medians. Multiple edits of a source contribute only one point. Color entropy describes palette dispersion, and edge density describes visible detail under fixed preprocessing.

Model	Checkpoint	Revision	Steps	Guidance
GPT-Image-2	—	API	—	—
FireRed-Edit	FireRed-Edit-1.1	3bc3f2a1	40	True CFG 4.0
Qwen-Image-Edit-2511	Qwen-Image-Edit-2511	6f3ccc0b	40	True CFG 4.0
FLUX.2-dev	FLUX.2-dev	26afe3a7	50	4.0
Qwen-Image-Edit	Qwen-Image-Edit	ac7f9318	50	True CFG 4.0
BAGEL	BAGEL-7B-MoT	5019f57d	50	T/I: 4.0/1.5

Appendix

Table 28: Inference settings for the evaluated editors. Checkpoint gives the exact public release evaluated; for open-weight models, Revision denotes the public repository commit. Guidance values follow each model’s own parameterization; T/I denotes text/image guidance.

		Design assets				Verification
Dataset	Other visual domains	SVG artwork	Charts	Web	Slides	Source target	Regional pixels
MagicBrush ( Zhang et al., 2023 )	Natural images	—	—	—	—	—	—
GIE-Bench ( Qian et al., 2025 )	Photographs; illustrations	—	—	—	—	—	—
PaintBench ( Xu et al., 2026 )	Shape canvases; network graphs †	—	$\checkmark^{\dagger}$	—	—	$\checkmark$	$\checkmark$
StructBench ( Zhuo et al., 2025 )	Math, science, puzzles, graphs, tables	—	$\checkmark$	—	—	$\checkmark$	—
FigEdit ( Li et al., 2025 )	—	—	$\checkmark$	—	—	$\checkmark$	—

Appendix

Table 29: Asset coverage, editing operations, and verification in related datasets.

\checkmark

indicates coverage documented in the paper’s task taxonomy or examples; — means not explicitly reported there.

\checkmark^{\dagger}

denotes PaintBench’s TinyGrafixBench extension. Presence does not imply equal task counts or coverage of every domain–operation combination.

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

PaintBench: Deterministic Evaluation of Precise Visual Editing
May 29, 2026Kai Xu, Ellis Brown, Shrikar Madhu +3Image Editing Evaluation Text-Guided Image Editing
New York University
GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing
Apr 16, 2026Yixuan Ding, Wei Huang, Ruijie Quan +2Image Editing Evaluation Text-Guided Image Editing
Zhejiang University · The University of Hong Kong

May 29, 2026J/K move · Enter open · S save

PaintBench: Deterministic Evaluation of Precise Visual Editing

Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus+2

New York University

Image Editing Evaluation Text-Guided Image Editing Image Editing

Open paper AskPDF arXiv

While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (

R^2 = 0.91

p < 0.001

). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.

Abstract

Image Editing

Explore similar work

PaintBench: Deterministic Evaluation of Precise Visual Editing

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing