V-CAPE: Visual Consistency Assessment for Product Image Editing

Iaroslav Melekhov; Roberto Alcover Couso; Francesco Taioli; Daniel Coelho; Artur Bekasov; Virginia Fernandez Arguedas; Jose Miguel Grande Saiz

cs.CVOct 28, 2026

V-CAPE: Visual Consistency Assessment for Product Image Editing

Authors: Iaroslav Melekhov, Roberto Alcover Couso, Francesco Taioli, Daniel Coelho, Artur Bekasov, Virginia Fernandez Arguedas, Jose Miguel Grande Saiz

Organizations: Amazon · Faculty

Abstract

Recent advances in generative image editing enable complex transformations such as background replacement, object manipulation, and pose adjustment from natural language instructions. However, verifying that these transformations preserve visual consistency, defined as the preservation of an object's intrinsic visual attributes, including geometry, texture, color, and fine-grained details, remains a largely unsolved problem. This challenge is particularly pronounced at scale, where manual inspection is infeasible and automated quality assessment (AutoQA) could become essential. In this work, we introduce Visual Consistency Assessment for Product Image Editing (V-CAPE), a large-scale dataset for evaluating visual consistency in generative image editing. The dataset consists of 40K unique source-target product image pairs, each annotated with binary accept/reject labels and structured failure modes capturing fine-grained inconsistencies such as geometric distortions, pose errors, and identity mismatches. V-CAPE combines real-world edited images with a synthetic subset constructed from 3D assets, enabling both realistic evaluation and controlled analysis. Vision-language models (VLMs) across all evaluated size categories on AutoQA achieve accuracy below 65%, indicating that the dataset poses a substantial challenge regardless of model scale. Our results highlight fundamental limitations of current models in verifying visual consistency under transformation, positioning V-CAPE as a benchmark for advancing fine-grained image editing evaluation.

Figures & tables

Figure 1: V-CAPE is a large-scale dataset for automated quality assessment in generative image editing, targeting fine-grained visual fidelity. The dataset combines real-world generative outputs (V-CAPE-R) with controlled synthetic samples (V-CAPE-S), comprising $27$ K unique images ( $40$ K source–target pairs). Each sample is annotated by multiple trained annotators with a structured taxonomy of rejection reasons, going beyond binary labels to capture diverse failure modes.

Panel of Figure 2 — Figure 2: V-CAPE overview. Examples from the two complementary subsets: (a) real-world generative outputs reflecting practical artifacts, and (b) synthetic data with controlled transformations and precise supervision. Together, they enable evaluation of fine-grained visual consistency under both realistic and controlled conditions.

Figure 3: V-CAPE-R generation pipeline. For a given product type, the system aggregates all product images

\mathcal{I}

and then ranks candidate images by compatibility with visual consistency requirements, and a single source image

I^{\star}

is selected. The Image Generation and Editing module removes background clutter via a text-conditioned editing prompt and conditionally applies pose alignment based on predicted pose and symmetry. The edited image

\hat{I}

is then validated by the QA pipeline.

Panel of Figure 4 — Figure 4: AutoQA performance vs. model scale. Accuracy (left) and F1 (right) on V-CAPE-R and V-CAPE-S as a function of model size. Even the largest models remain below 70% accuracy, indicating that visual consistency assessment is far from solved by current VLMs.

Panel of Figure 5 — Figure 5: AutoQA performance per-rejection reason. Distribution of per-category rejection recall across all evaluated VLMs on V-CAPE-R (left) and V-CAPE-S (right). Boxes: median and IQR; dots: individual models. Recall drops sharply from coarse violations (geometry or ca) to fine-grained mismatches (texture, lighting, background, colour/material), revealing a clear difficulty gradient that none of the evaluated models closes.

Panel of Figure 6 — Figure 6: Aggregated zero-shot orientation classification results. Comparison of model families across V-CAPE-S (3-class) and V-CAPE-R (2-class). Bars represent the mean performance within each technique family, while the error bars visualize the standard deviation (std) across the family’s individual models.

	V-CAPE-R		V-CAPE-S
Model	Acc.	F1	Acc.	F1
Qwen 3 8B (baseline)	.515 $\pm$ .003	.067 $\pm$ .003	.484 $\pm$ .003	.079 $\pm$ .002
+ N1	.521 $\pm$ .002	.601 $\pm$ .004	.532 $\pm$ .002	.624 $\pm$ .003
+ N5	.529 $\pm$ .003	.628 $\pm$ .003	.551 $\pm$ .004	.683 $\pm$ .003
+ N10	.527 $\pm$ .003	.617 $\pm$ .002	.543 $\pm$ .002	.681 $\pm$ .003

Table 1: Performance on V-CAPE: fine-tuning. We report accuracy and F1-score as bootstrap mean

\pm

standard deviation (

B{=}1{,}000

) for the baseline and its fine-tuned variants trained with different numbers of samples N .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.2 : Image candidate selection and generation modules. a) Given a set of catalog images $\mathcal{I}$ for a given product, each image is independently processed by pose and symmetry classifiers to infer object orientation and symmetry attributes (cf. Section A.1.1 ). These attributes, together with predefined visual consistency constraints, are used by a ranking and selection algorithm to deterministically select a single source image $I^{\star}$ for downstream processing. b) Image generation and editing: Starting from $I^{\star}$ , background clutter is first removed using Gemini 2.5 Flash [ 57 ] while preserving the main object. The QEM model [ 4 ] is then applied to perform controlled geometric edits, aligning the object pose with the target visual consistency requirements.

Category Samples Accepted 10 000 Geometry Artifacts 6 000 Wrong Orientation 1 804 Irrelevant Objects 989 Texture / Lighting / Color 802 Background Issues 306 Color / Material / Pattern Mismatch 99 Total 20 000 (a) V-CAPE-R

Category Samples Accepted 10 843 Wrong Orientation 4 303 Incorrect instance but same category 4 096 Category mismatch 758 Total 20 000 (b) V-CAPE-S

Appendix

Table A.1 : V-CAPE label taxonomy statistics. The number of pairs per each class of the proposed label taxonomy.

Panel of Figure 10 — Figure A.3 : V-CAPE-S product type distribution.

Model	Acc.	Prec.	Recall	F1
VQA-based baseline [ 58 , 59 ]	.481 $\pm$ .004	.540 $\pm$ .007	.516 $\pm$ .002	.394 $\pm$ .004

Appendix

Table D.1: VQA-based models performance on V-CAPE-S (binary accept/reject, strict conjunction aggregation, bootstrap mean

\pm

std,

B{=}1{,}000

Question	VQA-based baseline [ 58 , 59 ]
Source–target comparison
Geometry	.644 $\pm$ .003
Material	.472 $\pm$ .004
Color	.395 $\pm$ .003
Component count	.259 $\pm$ .003
Object identity	.305 $\pm$ .003

Appendix

Table D.2: Per-question accuracy of VQA-based models on V-CAPE-S (bootstrap mean

\pm

std,

B{=}1{,}000

). Questions are grouped by whether they require source–target comparison or target-only inspection.

Figure D.1 : V-CAPE-R rejected pairs for the Irrelevant Objects category (cf. Table A.1 ). Unlike prior datasets, our annotations provide explicit rejection reasons for each sample.

Figure E.1 : Visual overview of the V-CAPE-R rejection taxonomy . We showcase representative failure cases, including Wrong Pose and Geometry Artifacts , highlighting the fine-grained error categories (cf. Table A.1 ) captured by our annotations.

Figure E.2 : Visual overview of the V-CAPE-R rejection taxonomy . We showcase representative failure cases, including Texture / Lighting / Color Issues and Background Issues , highlighting the fine-grained error categories (cf. Table A.1 ) captured by our annotations.

Figure E.3 : V-CAPE fine-tuning: training dynamics

Figure F.1 : V-CAPE-S rejected pairs . For each rejection category in our taxonomy, we show the source–target image pair alongside the corresponding editing prompt.

Figure H.1 : Annotation template. We employ this template to collect labels from three annotators per image triplet (source, additional view, and target), enabling the assessment of generated image quality according to the proposed label taxonomy (Table A.1 ).

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 94 papers

GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
ProductConsistency: Improving Product Identity Preservation in Instruction-Based Image Editing via SFT and RL
Jun 17, 2026Mukund Khanna, Raj Singh Yadav, Kunal SinghImage Editing Evaluation Text-Guided Image Editing
Fractal Analytics
DiTailed: Ensuring Visual Object Consistency in Text-Image-to-Image Flow Matching Models
Jul 14, 2026Francesco Taioli, Daniel Coelho, Iaroslav Melekhov +4Flow Matching Text-Guided Image Editing
Amazon · Faculty

Mar 30, 2026J/K move · Enter open · S save

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang+6

Nanyang Technological University · StepFun · Southeast University

Image Editing Evaluation Text-Guided Image Editing Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.

Abstract

Image Editing Evaluation

Explore similar work

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

ProductConsistency: Improving Product Identity Preservation in Instruction-Based Image Editing via SFT and RL

DiTailed: Ensuring Visual Object Consistency in Text-Image-to-Image Flow Matching Models