ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

Aymen Lassoued; Angelique Loesch; Romaric Audigier

cs.CVOct 20, 2026

ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

Authors: Aymen Lassoued, Angelique Loesch, Romaric Audigier

Organizations: Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France

Abstract

Instruction-based image editing models can now add, remove a specified number of objects or change their category while preserving the scene. Yet these models often fail to produce the correct number, and these failures grow sharply as scenes become more complex and the requested count increases. Current benchmarks largely miss this problem: they score edits with holistic similarity metrics or with vision-language model (VLM) judges that are themselves poor at counting, so numerical errors go unnoticed or are graded incorrectly. As a result, the true numerical ability of today's editing models has never been accurately measured. We address this issue with ABACUS, a new benchmark, evaluation protocol and dedicated scoring metric for numerical image editing. Our protocol is judge-free: no VLM or MLLM scores an edit or counts objects (a VLM only confirms the category of rare ambiguous crops). Instead, it verifies each edit at the instance level against panoptic ground truth confirmed by detector consensus, checking both that the exact requested count was achieved and that the rest of the scene was preserved, across the add, remove and change operations. By evaluating leading image editors, our method reveals substantial numerical editing failures that worsen with increasing scene complexity and object count; failures that prior benchmarks often overlooked or misjudged.

Figures & tables

Figure 1: Numerical editing requires a specialized evaluator. Given an instruction to generate an exact number of objects, existing evaluation methods, including VLM judges (VIEScore [ 19 ] , I2EBench [ 25 ] , and RefEdit [ 30 ] ) and the agentic evaluator EdiVal-Agent [ 9 ] , incorrectly mark the edit as successful because they fail to verify numerical correctness. In contrast, ABACUS correctly identifies the edit as a failure. (The red boxes indicate the target categories that were either modified or removed during editing.)

Table 1: Overview of ABACUS.

Figure 2: Benchmark creation pipeline. Each {image, category} pair from COCONut panoptic ground truth: a detector-verified removability audit, an addition-plausibility check, an optimal addable-count estimate, instruction generation, and a final human-verification pass (Sec. 3.1 ).

Figure 3: ‘Add’/‘Remove’ pipeline. Addition and removal share one pipeline (Sec. 4.1 ); four stages map $I_{o}$ , $I_{e}$ , and the panoptic ground truth to the confirmed new instances $N^{\mathrm{added}}$ and the unmatched source instances $N^{\mathrm{unmatched}}$ : novelty discovery , clean-image construction , dual-detector detection , and VLM ambiguity resolution . The two outputs swap roles between addition and removal.

	Add			Remove			Change			Overall ( $\mathrm{Score}=\mathrm{CF}\cdot\mathrm{SC}\cdot\mathrm{OC}$ )
Model	E	M	H	E	M	H	E	M	H	CF	SC	OC	Score
Gemini 2.5 Flash Image [ 12 ]	0.52	0.39	0.26	0.45	0.30	0.18	0.31	0.22	0.12	0.59	0.77	0.64	0.29
Seedream 4.0 [ 36 ]	0.55	0.42	0.28	0.46	0.31	0.20	0.38	0.25	0.14	0.60	0.79	0.66	0.31
GPT-Image-1 [ 27 ]	0.51	0.36	0.25	0.38	0.24	0.17	0.31	0.18	0.11	0.56	0.75	0.62	0.26
Qwen-Image-Edit [ 40 ]	0.35	0.23	0.13	0.26	0.18	0.08	0.18	0.11	0.05	0.42	0.69	0.55	0.16
FLUX.1 Kontext [ 4 ]	0.30	0.20	0.12	0.24	0.14	0.08	0.14	0.09	0.05	0.39	0.67	0.53	0.14

Table 2: Editing-model leaderboard on ABACUS. E/M/H: cell Score

\mathrm{CF}_{c}\cdot\mathrm{SC}_{c}\cdot\mathrm{OC}_{c}

(Eq. 3 ), each factor averaged over the edits of that operation

\times

tier cell. Overall: CF, SC, OC averaged over all

10{,}944

edits; Score is their product, not the mean of the cell Scores. Bold: best per column; the three proprietary models’

95\%

bootstrap intervals overlap (App. D.1 ). Details: App. C.1 .

Evaluator	Easy	Medium	Hard	Overall
Grounding-DINO [ 23 ]	0.37	0.26	0.23	0.28
Qwen3.6-27B (thinking)	0.71	0.54	0.45	0.55
ABACUS (ours)	0.89	0.82	0.76	0.82

Table 3: Exact-count agreement with the human count (

\uparrow

), per difficulty tier, on the 600-image study.

Editor	ABACUS	Solo VLM	Single det.
Seedream 4.0 [ 36 ]	0.83	0.54	0.25
Gemini 2.5 [ 12 ]	0.81	0.53	0.24
GPT-Image-1 [ 27 ]	0.81	0.51	0.23
Qwen-Image-Edit [ 40 ]	0.80	0.47	0.22
FLUX.1 Kontext [ 4 ]	0.80	0.43	0.20
Step1X-Edit [ 24 ]	0.80	0.41	0.18

Table 4: ABACUS is stable across editors. Exact-count agreement with the human count (

200

edited images per editor, drawn from the same source images).

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Correlation of the three split features (Spearman $\rho$ ). The scoring axes, scene occupancy (fg_ratio), target visibility (target_fg_ratio), and edit capacity (max_count), are only weakly inter-correlated ( $|\rho|\!\leq\!0.32$ ), so they capture complementary aspects of scene difficulty rather than restating one another.

Figure S2 : The construction axes separate the tiers. Per-tier distributions of the three axes and of the composite difficulty score, with Cliff’s $\delta$ effect sizes (reported instead of $p$ -values, which are uninformative at this sample size). Every axis shifts monotonically from easy to hard, and the difficulty score is strongly and cleanly ordered.

Figure S3 : The tiers are non-circular. Per-tier CDFs of three held-out scene attributes that do not enter the difficulty score. They still separate easy/medium/hard (Cliff’s $\delta$ shown), so the split reflects real scene complexity, not the axes that define it. Crowd ratio ( $\delta=-0.09$ ) separates least, as expected, since it is the most orthogonal to the construction.

Figure S4 : The difficulty cube. Benchmark pairs in the three-axis space, colored by tier. Easy pairs (green) cluster in the low-clutter, large-target, low-capacity corner; hard pairs (red) fill the opposite corner, with medium (orange) in between.

Figure S5 : Benchmark examples. One add , one remove , and one change sample, each with the instruction shown beneath the image pair.

Figure S6 : Difficulty tiers. Real examples for easy, medium, and hard. Difficulty is a tercile split of a composite score over three axes: foreground coverage, target count, and target-object scale. This makes precise what a “complex scene” is.

Op.	Pipeline output	Symbol	Role in the Score
Add	$N^{\mathrm{added}}$	$d$	new instances; CF (Eq. 4 )
	$N^{\mathrm{unmatched}}$	$N_{\mathrm{rem}}$	originals destroyed; OC (Eq. 5 )
Remove	$N^{\mathrm{unmatched}}$	$r$	removed instances; CF (Eq. 9 )
	$N^{\mathrm{added}}$	$a$	spurious additions; OC (Eq. 9 )
Change	$N^{\mathrm{changed}}$	$t$	in-place changes; CF (Eq. 10 )
	$N_{\mathrm{cls1}}-N_{\mathrm{cls1}}^{\mathrm{ip}}$	$f_{1}$	$\mathrm{cls1}$ deleted, not replaced; OC

Appendix

Table S1 : Notation. Pipeline outputs named in the figures (Figs. 3 and S14 ) and the corresponding symbols in the scoring equations of Sec. 4.4 . For change, the OC denominator is

N_{\mathrm{GT}}^{\mathrm{cls1}}

Figure S7 : Addition pipeline on a real edit (“add 4 backpacks”). Row 1: source $+$ GT, NLPB-inpainted background, GT instances, and the Gemini edit. Row 2: presence-gated SAM-3 segmentation, one-to-one matching (grey $=$ matched, green $=$ new), clean-image construction, and the two detectors’ boxes. Row 3: SAM-confirmed candidates, consensus fallback, VLM candidates, and the final verdict. Row 4: the super-resolved crops sent to the constrained VLM verifier with its yes/no answers. The editor adds five backpacks for $K\!=\!4$ ; ABACUS recovers the true count and penalizes it.

Panel of Figure 16 — Figure S8 : Correct additions (I). “Add a zebra” (top, one added for $K\!=\!1$ ) and “add 2 people” (bottom, two for $K\!=\!2$ ). The added zebra sits behind the grazing animals, yet it is still segmented, isolated by inpainting, and confirmed by both detectors; ABACUS credits the edit.

Panel of Figure 17 — Figure S9 : Correct additions (II). “Add 2 motorcycles” (top) and “add 2 bottles” (bottom), both correctly counted at $K\!=\!2$ . The new motorcycles sit behind the foreground one and are still recovered, while the bottles are added among the existing desk clutter.

Panel of Figure 18 — Figure S10 : Caught over-additions. “Add 2 baseball bats” (top) and “add 2 spoons” (bottom) each yield three instances; ABACUS counts the surplus and the count-fidelity term penalizes the miscount, which a holistic or VLM-judge score would overlook.

Panel of Figure 19 — Figure S11 : High-count addition (“add 10 bottles”). A stress case at the top of the count range. The editor packs in fifteen bottles; presence-gated segmentation and dual-detector confirmation resolve the dense, overlapping instances, and ABACUS reports $15$ against $K\!=\!10$ , a miscount a holistic or VLM-judge metric would overlook.

Panel of Figure 20 — Figure S13 : Matching catches changes a judge cannot see. Top: source (left) and edited (right) images for “add 2 cars”; the two added cars blend in so well that the before/after difference is easy to miss, and a direct VLM judge cannot tell what changed. Bottom: ABACUS’s geometry-and-appearance matcher resolves it exactly, marking the pre-existing car matched (e1) and the two inserted cars new (e0, e2). The one-to-one matching is what turns a near-imperceptible edit into a precise, grounded count.

Figure S15 : Change pipeline on a real edit (“change 2 horses to zebras”). Each instance is colored by its verdict as the pipeline proceeds: the source horses are segmented and matched, the edited zebras are detected and confirmed by the two detectors, and the removed-source and added-target positions are matched so each change is credited only where a horse originally stood. This position-aware coupling is what prevents a delete-here-paint-there edit from passing as a change.

		Add			Remove			Change
Model	Factor	E	M	H	E	M	H	E	M	H
	$N_{c}$	1,159	1,363	1,374	1,146	1,340	1,308	1,000	1,128	1,126
Seedream 4.0 [ 36 ]	CF	0.762	0.662	0.579	0.684	0.563	0.513	0.677	0.541	0.469
	SC	0.922	0.852	0.776	0.901	0.800	0.724	0.839	0.726	0.607
	OC	0.782	0.745	0.633	0.745	0.686	0.530	0.677	0.624	0.476
	$\mathrm{CF}\!\cdot\!\mathrm{SC}\!\cdot\!\mathrm{OC}$	0.5494	0.4202	0.2844	0.4591	0.3090	0.1968	0.3845	0.2451	0.1355

Appendix

Table S2 : Per-cell factors behind Table 2 . Mean count fidelity (CF), scene consistency (SC), and original-category conservation (OC) per editor, operation, and difficulty tier, with the cell size

N_{c}

and the cell Score

\mathrm{CF}_{c}\cdot\mathrm{SC}_{c}\cdot\mathrm{OC}_{c}

; rounding the exact product half-up to two decimals gives the E/M/H columns of Table 2 .

Variant	Acc. vs. human
Full ABACUS	0.82
w/o ambiguity block (VLM verifier)	0.79
single detector (avg. RF-only, Co-only)	0.71
w/o inpainting (clean-image isolation)	0.70
w/o ground-truth completion	0.69
w/o one-to-one alignment	0.36

Appendix

Table S3 : Ablation of the evaluation protocol. Exact-count accuracy against the human count, removing one component at a time (600-image set; analysis in Sec. 5.4 ).

Figure S16 : Composition of the 600-image human-evaluation set. (a) Difficulty-tier distribution (easy/medium/hard). (b) Target-count $K$ distribution, concentrated at the low-to-mid counts and thinning at the hardest high counts. (c) Category coverage: 79 of 80 COCO thing-categories.

Figure S17 : Counting agreement with the human reference, by tier (600-image study). ABACUS tracks the human count across easy, medium, and hard; the solo VLM (Qwen3.6-27B) degrades with difficulty; the single Grounding-DINO detector is the least reliable.

Figure S18 : Counting agreement with human vs. scene count. As the number of objects in the edited image grows, ABACUS degrades gradually and levels off near $0.6$ on the busiest scenes, the solo VLM drops sharply but stays non-zero, and the single Grounding-DINO detector collapses to zero past ten objects. ABACUS stays well above both single-tool counters across the whole range.

Tier	$n$ (count-correct)	ABACUS recall	False-neg. rate
Easy	200	0.97	0.03
Medium	160	0.91	0.09
Hard	120	0.86	0.14
Overall	480	0.92	0.08

Appendix

Table S4 : Recall audit on human-confirmed count-correct edits. ABACUS recognises the correct count about

92\%

of the time, and the false-negative rate is roughly editor-independent, so a weak editor that does produce the right number is not scored as a failure. The overall is weighted by

n

(count-correct edits skew easy, so it exceeds the unweighted tier mean).

Factor of the ABACUS Score	Agreement with humans
Count fidelity (exact count)	$0.82$
Original-category conservation	$0.97$ (bal. acc. $0.90$ )
In-place change-correspondence	$0.95$
Scene consistency	$\rho=0.87$
Combined ABACUS Score	$\rho=0.88$ (ordering $\rho=0.96$ )

Appendix

Table S5 : Every factor of the Score is validated against humans (600-image study). Discrete-verdict factors report agreement rates; continuous factors report Spearman

\rho

Figure S19 : Scene consistency vs. human rating. ABACUS’s continuous scene-consistency score is monotone in the discrete human rating; the five-point human scale caps the rank correlation at $\rho=0.87$ . Human ratings are integers; points are jittered vertically for visibility only, and $\rho$ is computed on the integer ratings.

Threshold	Governs	Max $\|\Delta\|$ agreement
$\tau_{p}=0.6$	presence gate (SAM-3)	$\leq 0.02$
$\tau_{c}=0.7$	containment	$\leq 0.02$
$\tau_{d}=0.3$	centroid-distance gate	$\leq 0.02$
$\tau_{a}=0.15$	appearance confirmation	$\leq 0.02$
$\tau_{m}=0.1$	matcher keep-margin	$0.03$

Appendix

Table S6 : Threshold sweep (600-image set). Largest change in exact-count agreement as each threshold is swept over a wide range. Four thresholds are inert;

\tau_{m}

is the most sensitive, changing agreement by at most

0.03

Aggregation	Best real editor	Do-nothing editor
Arithmetic mean of CF, SC, OC	$0.684$	$0.667$ (nearly ties)
Product (ours)	$\mathbf{0.31}$	$\mathbf{0.00}$ (rejected)

Appendix

Table S7 : Product vs. mean. An arithmetic mean lets a do-nothing editor nearly tie the best model; the product rejects it while keeping all three factors.

Component	Role	Version / model ID
Seedream 4.0	editor	seedream-4-0-250828 (hosted API, queried 2026)
Gemini 2.5 Flash Image	editor	gemini-2.5-flash-image-preview
GPT-Image-1	editor	gpt-image-1
Qwen-Image-Edit	editor	Qwen/Qwen-Image-Edit
FLUX.1 Kontext	editor	black-forest-labs/FLUX.1-Kontext-dev
Step1X-Edit	editor	stepfun-ai/Step1X-Edit

Appendix

Table S8 : Components and versions. Proprietary editors are hosted APIs queried on fixed dates (

8

–

14

June and

15

August

2026

), reported by API model name and subject to provider-side updates. Open editors and all evaluation components are pinned to the listed Hugging Face repositories or checkpoints and released with the benchmark.

Evaluation detector set	Agreement	Ordering
RF-DETR + Co-DETR (ViT-L), default	$0.82$	—
RF-DETR + DINO (Swin-L)	$0.81$	preserved
RF-DETR + YOLO26-X	$0.81$	preserved
YOLO26-X + DINO (Swin-L)	$0.80$	preserved

Appendix

Table S9 : Detector-swap robustness. Re-scoring the 600-image validation set with perturbed detector pairs: exact-count agreement with the human count changes by at most

0.02

(largest for the fully disjoint pair) and the editor ordering never changes. The swapped-in DINO is the closed-set object-detection DINO, not the open-vocabulary Grounding-DINO.

Agreement	Overall	Hard
Human vs. human, Krippendorff $\alpha$	$0.88$	$0.79$
Human vs. human, raw	$0.92$	$0.85$
ABACUS vs. human, raw	$0.82$	$0.76$
Gap to the raw human ceiling	$0.10$	$0.09$

Appendix

Table S10 : Inter-annotator agreement and the human ceiling (re-annotated 600-image set). ABACUS stays within about

0.10

of the human-human ceiling, and the gap is flat across tiers.

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 91 papers

PaintBench: Deterministic Evaluation of Precise Visual Editing
May 29, 2026Kai Xu, Ellis Brown, Shrikar Madhu +3Image Editing Evaluation Text-Guided Image Editing
New York University
GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets
Oct 3, 2026Muyao Wang, Chen Zhu, Shiqi Yang +2Image Editing Image Editing Evaluation
SB Intuitions · University of Tokyo

May 29, 2026J/K move · Enter open · S save

PaintBench: Deterministic Evaluation of Precise Visual Editing

Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus+2

New York University

Image Editing Evaluation Text-Guided Image Editing Image Editing

Open paper AskPDF arXiv

While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (

R^2 = 0.91

p < 0.001

). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.

Abstract

Image Editing Evaluation

Explore similar work

PaintBench: Deterministic Evaluation of Precise Visual Editing

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets