CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions

Chonghuinan Wang; Zihan Chen; Yuxiang Wei; Tianyi Jiang; Xiaohe Wu; Fan Li; Wangmeng Zuo; Hongxun Yao

cs.CVMar 27, 2026

CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions

Authors: Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao

Organizations: Harbin Institute of Technology · Huawei Noah’s Ark Lab · Pengcheng Lab, Guangzhou

Abstract

Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.

Figures & tables

Figure 2: Comparison with previous benchmark. The CREval-Bench dataset extends existing instruction-based editing benchmarks by incorporating more complex, creative, and semantically rich instructions. Such design facilitates a comprehensive evaluation of model performance in handling imaginative and complex instruction editing tasks. In (b), the edited image examples on the right correspond one-to-one with the image-instruction pairs on the left.


Dataset	Size	Scoring	Creative	Fully-Automatic
ImgEdit-Bench [ 55 ]	791	MLLMs scoring	✗	✓
KRIS-Bench [ 50 ]	1267	MLLMs scoring	✗	✓
RISE-Bench [ 62 ]	360	MLLMs scoring	✗	✓
GEdit-Bench [ 27 ]	606	MLLMs scoring	✗	✓
CREval-Bench	874	VQA scoring	✓	✓

Table 1: Comparison to other existing benchmark. Our benchmark provides a comprehensive evaluation of creative image manipulation by leveraging VQA-based scoring.

Figure 3: Distribution of creative editing types. Creative types are organized into 3 primary categories and 9 dimensions, with balanced sample counts to ensure comprehensive and consistent evaluation.

Figure 4: Overview of CREval. (1) In stage 1, we manually select high-quality images. We then construct several editing instruction examples and utilize the GPT-4o model for few-shot learning across 9 predefined dimensions, generating dimension consistent editing instructions and producing image–instruction pairs. (2) In stage 2, we use these image–instruction pairs to construct evaluation tasks. To reduce bias, we use different MLLMs such as Qwen2.5-VL-72B, to generate evaluation questions for 3 metrics using the Chain-of-Thought (CoT) method. Each metric contains at least 5 questions, with a total of no fewer than 15 questions per pair, completing the construction of the CREval-Bench. (3)In Stage 3, we evaluate mainstream image manipulation models using CREval method. A MLLM model is employed as the evaluator to score each edited image based on evaluation questions. The final performance metric is obtained by computing a weighted average score across all evaluation metrics.

Figure 5: Performance comparison across all creative dimensions under different metrics. Top row: closed-source models; bottom row: open-source models.


	Methods	Customization				Contextualization				Stylization				Overall
	Methods	IF	VC	VQ	avg	IF	VC	VQ	avg	IF	VC	VQ	avg	IF	VC	VQ	avg

Open-Source	OmniGen2 [ 49 ]	68.49	52.84	83.64	65.26	68.42	56.57	81.40	66.28	78.59	61.48	85.93	73.21	71.58	57.20	83.28	68.17
	ICEdit [ 59 ]	40.90	61.75	68.11	54.68	42.17	44.34	65.92	47.79	54.31	62.81	72.16	61.28	45.33	55.25	67.72	53.78
	UniWorld-V1 [ 23 ]	45.30	81.15	73.04	65.19	41.74	80.31	65.33	61.89	59.89	76.86	77.26	70.15	48.29	79.74	70.77	65.37

Table 2: Evaluation results of mainstream image generation and editing models on CREval-Bench using GPT-4o as the evaluator. The scores for the three editing types and the overall average are reported across three evaluation metrics: Instruction Following (IF), Visual Consistency (VC), and Visual Quality (VQ). The best performance among closed-source models is highlighted in red , and the second best in blue . For open-source models, the top result is shown in bold , and the second best is underlined .


Methods	Aesthetic Score	VIEScore	EditScore	$CREvalScore_{Qwen3-VL}$	$CREvalScore_{GPT4o}$	HumanScore
Bagel [ 7 ]	5.56	6.02	7.28	72.59	68.99	49.98
FLUX.1-Kontext [dev] [ 2 ]	5.81	7.17	7.36	80.38	75.05	51.77
Qwen-Image-Edit [ 48 ]	5.82	6.83	7.97	83.02	79.18	63.49
GPT-Image-1 [ 30 ]	6.05	6.73	8.21	83.15	78.01	63.21
Gemini 2.5 Flash Image [ 10 ]	5.77	7.39	7.92	87.14	81.78	66.14

Table 3: Human preference verification. Aesthetic Score, VIEScore, and EditScore serve as baselines to evaluate six representative models (three open-source and three closed-source). CREvalScore Qwen3-VL and CREvalScore GPT4o use Qwen3-VL and GPT-4o as evaluators. Bold denotes the highest score, and underlining indicates the second-highest score.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix


Category	Dimension	Metric	Open-source Models									Closed-Source Models
Category	Dimension	Metric	OmniGen2 [ 49 ]	Bagel [ 7 ]	Bagel (think) [ 7 ]	UniWorld-V1 [ 23 ]	ICEdit [ 59 ]	Qwen-Image-Edit [ 48 ]	FLUX.1 Kontext[dev] [ 2 ]	Step1X-Edit-v1p2 [ 27 ]	Step1X-Edit-v1p2 (think) [ 27 ]	GPT-Image-1 [ 53 ]	Seedream4.0 [ 38 ]	Gemini 2.5 Flash Image [ 10 ]	FLUX.1 Kontext[pro] [ 2 ]
Customization	Derivative Character	IF	79.36	80.76	74.45	61.40	51.11	85.91	76.07	78.03	76.82	88.37	90.21	82.68	75.40
		VC	61.23	61.78	69.01	79.58	51.41	73.83	74.44	62.49	62.92	74.15	75.64	76.87	75.58
		VQ	85.59	84.02	80.64	74.57	64.88	89.33	90.91	85.03	80.07	89.15	90.07	89.12	91.27
		avg	73.35	73.82	73.51	71.31	53.98	81.76	78.39	73.21	71.91	82.84	84.35	81.64	78.65

Appendix

Table S1 : More quantitative comparisons on CREval-Bench by GPT-4o. The best performance among closed-source models is highlighted in red , and the second best in blue . For open-source models, the top result is shown in bold , and the second best is underlined .

Algorithm

I_{i}

denotes the original input image,

P

denotes the prompt

used for generating questions, and

Model

represents the

image generation model.

Benchmark Construction

instruction=MLLM_{1}(I_{i})

input_{i}=\{(I_{i},instruction),P_{i}\},i=IF,VC,VQ

Appendix

Table S2 : Our Pipeline Algorithm


	Instruction: Transform the figure into a chibi-style decorative resin bust for a tabletop display. Feature an embroidered brown robe with exaggerated patterns and a colorful headdress with vivid stripes. Accentuate the pomegranate by making it shine with gloss, and use minimalistic facial lines, ensuring the bust captures charm and elegance, sitting atop a smooth wooden base.

Questions-IF	Q1: Is the figure now a chibi-style bust with an oversized head and simplified body proportions, suitable for tabletop display?
	Q2: Does the figure wear a brown robe with visibly exaggerated and stylized embroidery patterns?
	Q3: Does the figure wear a headdress featuring vividly colored, clearly defined stripes?
	Q4: Is the pomegranate rendered with a glossy, reflective surface that highlights its shine?

Appendix

Table S3 : Examples of question-answer pairs, where the ideal answer in the table showing cases is ‘Yes’.

Figure S3 : Prompt for generating instructions.

Figure S4 : Prompt for generating IF Questions.

Figure S5 : Prompt for generating VC questions.

Figure S6 : Prompt for generating VQ questions.

Figure S7 : Prompt for evaluation.

Figure S9 : Visual comparison of open-source models in Customization.

Figure S10 : Visual comparison of closed-source models in Customization.

Figure S11 : Visual comparison of open-source models in Contextualization.

Figure S12 : Visual comparison of closed-source models in Contextualization.

Figure S13 : Visual comparison of open-source models in Stylization.

Figure S14 : Visual comparison of closed-source models in Stylization.

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 92 papers

GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling
May 13, 2026Xuehai Bai, Yang Shi, Yi-Fan Zhang +7Reward Modeling Image Editing Evaluation
HDU · PKU · Kling Team +1
CPI-Bench: A Comprehensive, Practical and Intelligent Benchmark for Real-World Image Editing
Aug 14, 2026Qinye Zhou, Jun Zheng, Yongchao Du +17Image Editing Evaluation Image Editing
Alibaba Group

Mar 30, 2026J/K move · Enter open · S save

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Zhangqi Jiang, Zheng Sun, Xianfang Zeng, Yufeng Yang+6

Nanyang Technological University · StepFun · Southeast University

Image Editing Evaluation Text-Guided Image Editing Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.

Abstract

Image Editing Evaluation

Explore similar work

GEditBench v2: A Human-Aligned Benchmark for General Image Editing

Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

CPI-Bench: A Comprehensive, Practical and Intelligent Benchmark for Real-World Image Editing