Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
Figures & tables
Figure 2: Comparison with previous benchmark. The CREval-Bench dataset extends existing instruction-based editing benchmarks by incorporating more complex, creative, and semantically rich instructions. Such design facilitates a comprehensive evaluation of model performance in handling imaginative and complex instruction editing tasks. In (b), the edited image examples on the right correspond one-to-one with the image-instruction pairs on the left.
Dataset
Size
Scoring
Creative
Fully-Automatic
ImgEdit-Bench [ 55 ]
791
MLLMs scoring
✗
✓
KRIS-Bench [ 50 ]
1267
MLLMs scoring
✗
✓
RISE-Bench [ 62 ]
360
MLLMs scoring
✗
✓
GEdit-Bench [ 27 ]
606
MLLMs scoring
✗
✓
CREval-Bench
874
VQA scoring
✓
✓
Table 1: Comparison to other existing benchmark. Our benchmark provides a comprehensive evaluation of creative image manipulation by leveraging VQA-based scoring.
Figure 3: Distribution of creative editing types. Creative types are organized into 3 primary categories and 9 dimensions, with balanced sample counts to ensure comprehensive and consistent evaluation.
Figure 4: Overview of CREval. (1) In stage 1, we manually select high-quality images. We then construct several editing instruction examples and utilize the GPT-4o model for few-shot learning across 9 predefined dimensions, generating dimension consistent editing instructions and producing image–instruction pairs. (2) In stage 2, we use these image–instruction pairs to construct evaluation tasks. To reduce bias, we use different MLLMs such as Qwen2.5-VL-72B, to generate evaluation questions for 3 metrics using the Chain-of-Thought (CoT) method. Each metric contains at least 5 questions, with a total of no fewer than 15 questions per pair, completing the construction of the CREval-Bench. (3)In Stage 3, we evaluate mainstream image manipulation models using CREval method. A MLLM model is employed as the evaluator to score each edited image based on evaluation questions. The final performance metric is obtained by computing a weighted average score across all evaluation metrics.
Figure 5: Performance comparison across all creative dimensions under different metrics. Top row: closed-source models; bottom row: open-source models.
Methods
Customization
Contextualization
Stylization
Overall
IF
VC
VQ
avg
IF
VC
VQ
avg
IF
VC
VQ
avg
IF
VC
VQ
avg
Open-Source
OmniGen2 [ 49 ]
68.49
52.84
83.64
65.26
68.42
56.57
81.40
66.28
78.59
61.48
85.93
73.21
71.58
57.20
83.28
68.17
ICEdit [ 59 ]
40.90
61.75
68.11
54.68
42.17
44.34
65.92
47.79
54.31
62.81
72.16
61.28
45.33
55.25
67.72
53.78
UniWorld-V1 [ 23 ]
45.30
81.15
73.04
65.19
41.74
80.31
65.33
61.89
59.89
76.86
77.26
70.15
48.29
79.74
70.77
65.37
Table 2: Evaluation results of mainstream image generation and editing models on CREval-Bench using GPT-4o as the evaluator. The scores for the three editing types and the overall average are reported across three evaluation metrics: Instruction Following (IF), Visual Consistency (VC), and Visual Quality (VQ). The best performance among closed-source models is highlighted in red , and the second best in blue . For open-source models, the top result is shown in bold , and the second best is underlined .
Methods
Aesthetic Score
VIEScore
EditScore
CREvalScoreQwen3−VL
CREvalScoreGPT4o
HumanScore
Bagel [ 7 ]
5.56
6.02
7.28
72.59
68.99
49.98
FLUX.1-Kontext [dev] [ 2 ]
5.81
7.17
7.36
80.38
75.05
51.77
Qwen-Image-Edit [ 48 ]
5.82
6.83
7.97
83.02
79.18
63.49
GPT-Image-1 [ 30 ]
6.05
6.73
8.21
83.15
78.01
63.21
Gemini 2.5 Flash Image [ 10 ]
5.77
7.39
7.92
87.14
81.78
66.14
Table 3: Human preference verification. Aesthetic Score, VIEScore, and EditScore serve as baselines to evaluate six representative models (three open-source and three closed-source). CREvalScore Qwen3-VL and CREvalScore GPT4o use Qwen3-VL and GPT-4o as evaluators. Bold denotes the highest score, and underlining indicates the second-highest score.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8
Category
Dimension
Metric
Open-source Models
Closed-Source Models
OmniGen2 [ 49 ]
Bagel [ 7 ]
Bagel (think) [ 7 ]
UniWorld-V1 [ 23 ]
ICEdit [ 59 ]
Qwen-Image-Edit [ 48 ]
FLUX.1 Kontext[dev] [ 2 ]
Step1X-Edit-v1p2 [ 27 ]
Step1X-Edit-v1p2 (think) [ 27 ]
GPT-Image-1 [ 53 ]
Seedream4.0 [ 38 ]
Gemini 2.5 Flash Image [ 10 ]
FLUX.1 Kontext[pro] [ 2 ]
Customization
Derivative Character
IF
79.36
80.76
74.45
61.40
51.11
85.91
76.07
78.03
76.82
88.37
90.21
82.68
75.40
VC
61.23
61.78
69.01
79.58
51.41
73.83
74.44
62.49
62.92
74.15
75.64
76.87
75.58
VQ
85.59
84.02
80.64
74.57
64.88
89.33
90.91
85.03
80.07
89.15
90.07
89.12
91.27
avg
73.35
73.82
73.51
71.31
53.98
81.76
78.39
73.21
71.91
82.84
84.35
81.64
78.65
Appendix
Table S1 : More quantitative comparisons on CREval-Bench by GPT-4o. The best performance among closed-source models is highlighted in red , and the second best in blue . For open-source models, the top result is shown in bold , and the second best is underlined .
Algorithm
Ii denotes the original input image, P denotes the prompt
used for generating questions, and Model represents the
image generation model.
Benchmark Construction
instruction=MLLM1(Ii)
inputi={(Ii,instruction),Pi},i=IF,VC,VQ
Appendix
Table S2 : Our Pipeline Algorithm
Instruction: Transform the figure into a chibi-style decorative resin bust for a tabletop display. Feature an embroidered brown robe with exaggerated patterns and a colorful headdress with vivid stripes. Accentuate the pomegranate by making it shine with gloss, and use minimalistic facial lines, ensuring the bust captures charm and elegance, sitting atop a smooth wooden base.
Questions-IF
Q1: Is the figure now a chibi-style bust with an oversized head and simplified body proportions, suitable for tabletop display?
Q2: Does the figure wear a brown robe with visibly exaggerated and stylized embroidery patterns?
Q3: Does the figure wear a headdress featuring vividly colored, clearly defined stripes?
Q4: Is the pomegranate rendered with a glossy, reflective surface that highlights its shine?
Appendix
Table S3 : Examples of question-answer pairs, where the ideal answer in the table showing cases is ‘Yes’.
Figure S3 : Prompt for generating instructions.
Figure S4 : Prompt for generating IF Questions.
Figure S5 : Prompt for generating VC questions.
Figure S6 : Prompt for generating VQ questions.
Figure S7 : Prompt for evaluation.
Figure S8 : More visual comparison.
Figure S9 : Visual comparison of open-source models in Customization.
Figure S10 : Visual comparison of closed-source models in Customization.
Figure S11 : Visual comparison of open-source models in Contextualization.
Figure S12 : Visual comparison of closed-source models in Contextualization.
Figure S13 : Visual comparison of open-source models in Stylization.
Figure S14 : Visual comparison of closed-source models in Stylization.
Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7
Nanyang Technological University · StepFun · Southeast University
Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, existing benchmarks often fail to faithfully reflect human judgment, especially for strong frontier models, due to limited task difficulty and coarse-grained evaluation protocols. In parallel, reward models have become increasingly important for RL-based image editing optimization, yet existing reward model benchmarks still rely on unrealistic evaluation settings that deviate from practical RL scenarios. These limitations hinder reliable assessment of both image editing models and reward models. To address these challenges, we introduce Edit-Compass and EditReward-Compass, a unified evaluation suite for image editing and reward modeling. Edit-Compass contains 2,388 carefully annotated instances spanning six progressively challenging task categories, covering capabilities such as world knowledge reasoning, visual reasoning, and multi-image editing. Beyond broad task coverage, Edit-Compass adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed scoring rubrics. In parallel, EditReward-Compass contains 2,251 preference pairs that simulate realistic reward modeling scenarios during RL optimization.
With the rapid advancement of image editing models and their widespread application across various domains, there is an increasingly urgent need to deploy these model capabilities directly into real-world scenarios. However, existing benchmarks remain confined to simple single-image tasks, suffering from limited coverage dimensions and an inability to effectively differentiate performance among diverse models. Consequently, they fail to reliably evaluate model performance in complex multi-image editing, highly demanding reasoning instructions, and practical deployment settings. To address these limitations, we propose CPI-Bench, a Comprehensive, Practical and Intelligent benchmark for real-world image editing. CPI-Bench comprises three core subsets: CPI-General-Bench, which comprehensively covers diverse editing tasks and introduces multi-image editing evaluation; CPI-Practical-Bench, which focuses on high-frequency real-user application scenarios; and CPI-Intelligent-Bench, which is dedicated to evaluating capabilities in highly demanding reasoning-based editing. Evaluation results of mainstream image editing models based on CPI-Bench demonstrate that CPI-Bench enhances performance differentiation among models. It provides a comprehensive and reliable quantification of gaps in general editing capabilities, practical deployment efficacy, and advanced reasoning-based editing, offering invaluable guidance for the future optimization of image editing models. Crucially, our ranking analysis reveals that CPI-Bench achieves the highest alignment with the Arena Image Edit Leaderboard, indicating stronger consistency with public human preference rankings, serving as an effective proxy for public human evaluations.