V-CAPE: Visual Consistency Assessment for Product Image Editing
Authors: Iaroslav Melekhov, Roberto Alcover Couso, Francesco Taioli, Daniel Coelho, Artur Bekasov, Virginia Fernandez Arguedas, Jose Miguel Grande Saiz
Recent advances in generative image editing enable complex transformations such as background replacement, object manipulation, and pose adjustment from natural language instructions. However, verifying that these transformations preserve visual consistency, defined as the preservation of an object's intrinsic visual attributes, including geometry, texture, color, and fine-grained details, remains a largely unsolved problem. This challenge is particularly pronounced at scale, where manual inspection is infeasible and automated quality assessment (AutoQA) could become essential. In this work, we introduce Visual Consistency Assessment for Product Image Editing (V-CAPE), a large-scale dataset for evaluating visual consistency in generative image editing. The dataset consists of 40K unique source-target product image pairs, each annotated with binary accept/reject labels and structured failure modes capturing fine-grained inconsistencies such as geometric distortions, pose errors, and identity mismatches. V-CAPE combines real-world edited images with a synthetic subset constructed from 3D assets, enabling both realistic evaluation and controlled analysis. Vision-language models (VLMs) across all evaluated size categories on AutoQA achieve accuracy below 65%, indicating that the dataset poses a substantial challenge regardless of model scale. Our results highlight fundamental limitations of current models in verifying visual consistency under transformation, positioning V-CAPE as a benchmark for advancing fine-grained image editing evaluation.
Figures & tables
Figure 1: V-CAPE is a large-scale dataset for automated quality assessment in generative image editing, targeting fine-grained visual fidelity. The dataset combines real-world generative outputs (V-CAPE-R) with controlled synthetic samples (V-CAPE-S), comprising 27 K unique images ( 40 K source–target pairs). Each sample is annotated by multiple trained annotators with a structured taxonomy of rejection reasons, going beyond binary labels to capture diverse failure modes.
Figure 2: V-CAPE overview. Examples from the two complementary subsets: (a) real-world generative outputs reflecting practical artifacts, and (b) synthetic data with controlled transformations and precise supervision. Together, they enable evaluation of fine-grained visual consistency under both realistic and controlled conditions.
Figure 3: V-CAPE-R generation pipeline. For a given product type, the system aggregates all product images I and then ranks candidate images by compatibility with visual consistency requirements, and a single source image I⋆ is selected. The Image Generation and Editing module removes background clutter via a text-conditioned editing prompt and conditionally applies pose alignment based on predicted pose and symmetry. The edited image I^ is then validated by the QA pipeline.
Figure 4: AutoQA performance vs. model scale. Accuracy (left) and F1 (right) on V-CAPE-R and V-CAPE-S as a function of model size. Even the largest models remain below 70% accuracy, indicating that visual consistency assessment is far from solved by current VLMs.
Figure 5: AutoQA performance per-rejection reason. Distribution of per-category rejection recall across all evaluated VLMs on V-CAPE-R (left) and V-CAPE-S (right). Boxes: median and IQR; dots: individual models. Recall drops sharply from coarse violations (geometry or ca) to fine-grained mismatches (texture, lighting, background, colour/material), revealing a clear difficulty gradient that none of the evaluated models closes.
Figure 6: Aggregated zero-shot orientation classification results. Comparison of model families across V-CAPE-S (3-class) and V-CAPE-R (2-class). Bars represent the mean performance within each technique family, while the error bars visualize the standard deviation (std) across the family’s individual models.
V-CAPE-R
V-CAPE-S
Model
Acc.
F1
Acc.
F1
Qwen 3 8B (baseline)
.515 ± .003
.067 ± .003
.484 ± .003
.079 ± .002
+ N1
.521 ± .002
.601 ± .004
.532 ± .002
.624 ± .003
+ N5
.529 ± .003
.628 ± .003
.551 ± .004
.683 ± .003
+ N10
.527 ± .003
.617 ± .002
.543 ± .002
.681 ± .003
Table 1: Performance on V-CAPE: fine-tuning. We report accuracy and F1-score as bootstrap mean ± standard deviation ( B=1,000 ) for the baseline and its fine-tuned variants trained with different numbers of samples N .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.2 : Image candidate selection and generation modules. a) Given a set of catalog images I for a given product, each image is independently processed by pose and symmetry classifiers to infer object orientation and symmetry attributes (cf. Section A.1.1 ). These attributes, together with predefined visual consistency constraints, are used by a ranking and selection algorithm to deterministically select a single source image I⋆ for downstream processing. b) Image generation and editing: Starting from I⋆ , background clutter is first removed using Gemini 2.5 Flash [ 57 ] while preserving the main object. The QEM model [ 4 ] is then applied to perform controlled geometric edits, aligning the object pose with the target visual consistency requirements.
Category Samples Accepted 10 000 Geometry Artifacts 6 000 Wrong Orientation 1 804 Irrelevant Objects 989 Texture / Lighting / Color 802 Background Issues 306 Color / Material / Pattern Mismatch 99 Total 20 000 (a) V-CAPE-R
Category Samples Accepted 10 843 Wrong Orientation 4 303 Incorrect instance but same category 4 096 Category mismatch 758 Total 20 000 (b) V-CAPE-S
Appendix
Table A.1 : V-CAPE label taxonomy statistics. The number of pairs per each class of the proposed label taxonomy.
Figure A.3 : V-CAPE-S product type distribution.
Model
Acc.
Prec.
Recall
F1
VQA-based baseline [ 58 , 59 ]
.481 ± .004
.540 ± .007
.516 ± .002
.394 ± .004
Appendix
Table D.1: VQA-based models performance on V-CAPE-S (binary accept/reject, strict conjunction aggregation, bootstrap mean ± std, B=1,000 ).
Question
VQA-based baseline [ 58 , 59 ]
Source–target comparison
Geometry
.644 ± .003
Material
.472 ± .004
Color
.395 ± .003
Component count
.259 ± .003
Object identity
.305 ± .003
Appendix
Table D.2: Per-question accuracy of VQA-based models on V-CAPE-S (bootstrap mean ± std, B=1,000 ). Questions are grouped by whether they require source–target comparison or target-only inspection.
Figure D.1 : V-CAPE-R rejected pairs for the Irrelevant Objects category (cf. Table A.1 ). Unlike prior datasets, our annotations provide explicit rejection reasons for each sample.
Figure E.1 : Visual overview of the V-CAPE-R rejection taxonomy . We showcase representative failure cases, including Wrong Pose and Geometry Artifacts , highlighting the fine-grained error categories (cf. Table A.1 ) captured by our annotations.
Figure E.2 : Visual overview of the V-CAPE-R rejection taxonomy . We showcase representative failure cases, including Texture / Lighting / Color Issues and Background Issues , highlighting the fine-grained error categories (cf. Table A.1 ) captured by our annotations.
Figure E.3 : V-CAPE fine-tuning: training dynamics
Figure F.1 : V-CAPE-S rejected pairs . For each rejection category in our taxonomy, we show the source–target image pair alongside the corresponding editing prompt.
Figure H.1 : Annotation template. We employ this template to collect labels from three annotators per image triplet (source, additional view, and target), enabling the assessment of generated image quality according to the proposed label taxonomy (Table A.1 ).
Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7
Nanyang Technological University · StepFun · Southeast University
Recent advances in instruction-based image editing have enabled models to perform complex visual edits from natural language instructions. However, in product-centric scenarios where preserving product features, branding, and textual elements are critical, current open and closed source models often struggle to maintain this fine-grained object identity. This issue is further compounded by the lack of datasets for instruction-based product image editing with text fidelity constraints, leaving it largely treated as an implicit capability of instruction-based image editing models. In this work, we introduce the ProductConsistency dataset which is designed to improve product-centric image editing. Our approach includes a supervised fine-tuning (SFT) dataset of 87k samples for product editing, a reinforcement learning (RL) dataset with 869 unique product images, and a new benchmark dataset, the ProductConsistency Benchmark, to allow rigorous and standardized evaluation of editing models. To guide RL training, we propose a Cyclic Consistency reward that enforces semantic preservation of product identity by using caption similarity between the original product description and captions generated from the edited image. We fine-tune both Qwen-Image-Edit-2511 and Flux.1-Kontext-dev using our dataset and demonstrate consistent improvements over baseline models in OCR and Perceptual metrics, and MLLM-based evaluations as well, indicating stronger product consistency, text rendering, and overall visual quality; with the Qwen-Image-Edit-2511 model achieving a 5x reduction in the character error rate. The code and pipeline is available at https://anonymous.4open.science/r/ProductConsistency-6FCC/README.md
Despite remarkable progress in text-guided image editing, generative models frequently fail to preserve visual object consistency, defined as the preservation of a subject's key attributes throughout the editing process. We address this limitation through three contributions. First, we introduce ABO-Edit, a dataset specifically designed to study object consistency, comprising over 12,000 triplets of source images, editing prompts, and high-quality target images rendered from artist-designed 3D assets, with multi-view coverage and human-verified quality control. Second, we uncover an overlooked property of image-editing rectified flow models: the conditioning embedding space, not directly supervised during training, encodes a prediction of the final generated image even at high noise levels. Third, exploiting this finding, we propose FlowMirror, a parameter-free auxiliary loss that supervises this conditioning embedding space. Without architectural changes, our method improves generation quality across several metrics over baselines.
Francesco Taioli, Daniel Coelho, Iaroslav Melekhov +4