Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
Figures & tables
Figure 1: GEditBench v2 spans 23 diverse image editing tasks, ranging from predefined edits to complex open-set real-world instructions, offering a comprehensive testbed for evaluating instruction-based image editing models. The central rose diagram visualizes the corresponding count distribution.
Benchmark
Size
Tasks
Complex Edit
Open-set
Evaluation
Support
Instruction
Unify Metrics
Human-Aligned
KontextBench Labs et al. (2025)
1,026
5
✗
✗
✗
−
AnyEdit-Bench Yu et al. (2025)
1,250
25
✗
✗
✗
Low
ImgEdit-Bench Ye et al. (2025c)
811
14
✗
✗
✓
Mid
GEdit-Bench Liu et al. (2025)
606
11
✗
✗
✓
Mid
ICE-Bench Pan et al. (2025)
6,538
31
✗
✗
✗
Low
Table 1: Comparison of general image editing benchmarks. GEditBench v2 pioneers a necessary transition to complex open-set scenarios, spanning 23 diverse tasks curated from real-world queries to establish a truly comprehensive evaluation standard.
Figure 2: Human preference agreement of pointwise and pairwise evaluation paradigms across four VLMs in (a) instruction following, (b) visual quality, and (c) visual consistency dimensions. Pairwise evaluation consistently achieves higher agreement with human judgments, suggesting its superior human alignment over prior absolute scoring. NC and SC prompts are adopted from ( Ye et al., 2025b ) and ( Luo et al., 2025 ) , respectively.
Figure 3: Two-stage candidates curation pipeline with prompt filtering.
Figure 4: Average accuracy of different image-instruction pairs per task on six representative tasks for visual consistency. Performance improves steadily and saturates around 1,500.
Figure 5: Preference data construction pipelines. (A) Object-centric pipeline: instructions are parsed to localize edited entities, partitioning the image into edit and non-edit regions. Region-specific metrics are then applied to enforce background fidelity while evaluating identity consistency within edited areas. (B) Human-centric pipeline: extends the object-centric pipeline by decomposing human attributes (face identity, body appearance, hair appearance) and dynamically excluding the edited attribute, enabling fine-grained consistency evaluation using specialized expert models.
Figure 6: Task-wise primary score distributions for preference pair synthesis. (a-b) Shaded regions indicate the top/bottom 30% Z-score threshold used to identify Winners (green) and Losers (red) with a clear optimization margin for preference learning. (c-d) Combined with Pareto filtering, candidate pairs that only exhibit score separation across two primary metrics are retained.
Figure 7: Average accuracy on three tasks of different in-group pairs ( P ) for visual consistency. P =6, chosen as the optimal balance between supervision density and computation cost.
Benchmark
Size
Tasks
EditScore ( Luo et al., 2025 )
3K
11
EditReward ( Wu et al., 2025c )
1.5K
-
VCReward-Bench
3.5K
21
Table 2: Comparison of VCReward-Bench with existing reward benchmarks.
Figure 8: Human alignment of assessment models for visual consistency on (a) EditReward-Bench ( Luo et al., 2025 ) and (b) VCReward-Bench. Notably, our model achieves state-of-the-art performance among open-source evaluators across nearly all tasks, performing on par with or even surpassing the proprietary GPT-5.1-2026-02-28.
Model
Samples
Instruction Following
Visual Quality
Visual Consistency
Overall
Arena ⋆
ELO ↑
95% CI
ELO ↑
95% CI
ELO ↑
95% CI
ELO ↑
95% CI
ELO ↑
Rank
∙ Nano Banana Pro (26-03-04)
1,156
1,126
-13/+15
1,066
-9/+10
1,108
-11/+11
1,096
-6/+6
1,251
2
∙ Seedream 4.5 (26-03-11)
1,190
1,111
-12/+12
1,142
-11/+11
1,030
-11/+12
1,089
-7/+7
1,196
3
∙ GPT Image 1.5 (26-03-04)
1,081
1,260
-13/+15
1,149
-12/+12
846
-13/+13
1,071
-7/+6
1,270
1
∙ FLUX.2 [klein] 9B
1,200
1,083
-13/+12
1,025
-11/+10
1,019
-10/+9
1,039
-6/+6
1,166
4
∙ Qwen-Image-Edit-2511
1,200
1,095
-10/+10
1,060
-11/+11
972
-9/+10
1,038
-6/+6
1,164
5
Table 3: OpenEdit leaderboard. Models are ranked by Elo ratings derived from a pairwise comparison paradigm. Instruction Following and Visual Quality are assessed by GPT-4o (26-03-24), while Visual Consistency is exclusively evaluated by our proposed PVC-Judge. Overall Elo scores and their 95% Confidence Intervals (CI) are computed via 1,000 bootstrap iterations. ∙ means closed-source, ∙ means open-source. ⋆ The Elo scores of Arena were recorded on March 26, 2026.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: (Input Image, Instruction) examples for the Relation Change task.
Figure 10: (Input Image, Instruction) examples for the Text Editing task. Our benchmark comprehensively evaluates fundamental semantic modifications alongside highly demanding typographic layout changes and precise font alterations.
Figure 11: (Input Image, Instruction) examples for the Chart Editing task.
Figure 12: (Input Image, Instruction) examples for the Camera Motion task.
Figure 13: (Input Image, Instruction) examples for the Object Reference task.
Figure 14: (Input Image, Instruction) examples for the Open-Set task.
Figure 15: Comparison results of three different prompting strategies for visual consistency evaluation on EditReward-Bench ( Luo et al., 2025 ) . (a) and (b) show the results of Gemini 3 Pro and Qwen3-VL-8B-Instruct, respectively. Pairwise statistical significance is evaluated using the two–sided Mann–Whitney U test ( ∗∗p<0.01 , ∗p<0.05 , ns: p≥0.05 ). (c) reports the average evaluation time over 890 testing pairs measured with Qwen3-VL-8B-Instruct.
Table 4: (Sub-Task, Pipeline, Region-Specific Metrics) mapping for preference pair automated annotation. Primary metrics are marked in bold .
Figure 16: Distribution of the preference dataset. The donut chart shows the detailed allocation across 16 editing tasks, providing reliable signals for training the PVC-Judge model.
Hyper-parameter
Value
optimizer
AdamW
num_train_epochs
3
per_device_train_batch_size
2
gradient_accumulation_steps
1
learning_rate
2.0e-6
warmup_ratio
0.05
Appendix
Table 5: LoRA Training Parameters.
Figure 17: Screenshots of our custom-built annotation interface for VCReward-Bench.
Figure 18: Distribution of VCReward-Bench instances by tasks.
Sub-Tasks
Gemini 3 Pro
GPT 5.1
EditReward
EditScore Avg@4
Qwen3-VL-8B-Instruct
PVC-Judge
( Wu et al., 2025c )
( Luo et al., 2025 )
Subject Addition
80.88
80.88
74.38
57.65
72.06
77.94
Subject Removal
83.56
63.01
89.90
51.91
61.64
67.12
Subject Replace
90.57
77.36
81.55
54.85
73.58
84.91
Color Alteration
91.30
86.96
83.49
77.83
72.46
94.20
Material Modification
90.16
88.52
67.00
51.23
75.41
83.61
Appendix
Table 6: Numerical results of assessment models for visual consistency on EditReward-Bench ( Luo et al., 2025 ) . Bold marks the best-performing open-source model.
Sub-Tasks
Gemini 3 Pro
GPT 5.1
EditReward Wu et al. (2025c)
EditScore Avg@4 Luo et al. (2025)
Qwen3-VL-8B-Instruct
PVC-Judge
Subject Addition
93.59
80.68
67.39
41.30
72.53
86.96
Subject Removal
86.32
68.49
70.97
35.48
69.01
74.19
Subject Replace
85.88
67.96
65.14
36.70
74.00
82.57
Size Adjustment
83.52
69.45
57.34
34.81
68.22
72.35
Color Alteration
92.90
75.96
75.46
53.24
67.01
87.04
Material Modification
86.91
80.09
57.02
50.88
72.89
86.84
Appendix
Table 7: Numerical results of assessment models for visual consistency on our VCReward-Bench. Bold marks the best-performing open-source model.
Figure 19: Open-set editing examples across six representative models. Many open-source models fail to fully capture implicit user instructions, leading to partial or inconsistent edits. This highlights the need for open-set evaluation to reveal limitations of current generative editing models.
Figure 20: Visualization results of relation change task. Open-source models often struggle to capture complex spatial relationships, leading to partial or simplified relational edits, while closed-source models better preserve inter-object dependencies.
Figure 21: Fine-grained detail generation across multiple tasks. Open-source models frequently distort small entities and background subjects, whereas closed-source models better maintain structure, with some difficulty preserving identity on small faces.
Figure 22: The pairwise evaluation prompt for the Instruction Following dimension.
Figure 23: The pairwise evaluation prompt for the Visual Quality dimension.
Figure 24: The pairwise evaluation prompt for the Visual Consistency dimension.
Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, existing benchmarks often fail to faithfully reflect human judgment, especially for strong frontier models, due to limited task difficulty and coarse-grained evaluation protocols. In parallel, reward models have become increasingly important for RL-based image editing optimization, yet existing reward model benchmarks still rely on unrealistic evaluation settings that deviate from practical RL scenarios. These limitations hinder reliable assessment of both image editing models and reward models. To address these challenges, we introduce Edit-Compass and EditReward-Compass, a unified evaluation suite for image editing and reward modeling. Edit-Compass contains 2,388 carefully annotated instances spanning six progressively challenging task categories, covering capabilities such as world knowledge reasoning, visual reasoning, and multi-image editing. Beyond broad task coverage, Edit-Compass adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed scoring rubrics. In parallel, EditReward-Compass contains 2,251 preference pairs that simulate realistic reward modeling scenarios during RL optimization.
The evaluation of visual editing models remains fragmented across methods and modalities. Existing benchmarks are often tailored to specific paradigms, making fair cross-paradigm comparisons difficult, while video editing lacks reliable evaluation benchmarks. Furthermore, common automatic metrics often misalign with human preference, yet directly deploying large multimodal models (MLLMs) as evaluators incurs prohibitive computational and financial costs. We present UniEditBench, a unified benchmark for image and video editing that supports reconstruction-based and instruction-driven methods under a shared protocol. UniEditBench includes a structured taxonomy of nine image operations (Add, Remove, Replace, Change, Stroke-based, Extract, Adjust, Count, Reorder) and eight video operations, with coverage of challenging compositional tasks such as counting and spatial reordering. To enable scalable evaluation, we distill a high-capacity MLLM judge (Qwen3-VL-235B-A22B Instruct) into lightweight 4B/8B evaluators that provide multi-dimensional scoring over structural fidelity, text alignment, background consistency, naturalness, and temporal-spatial consistency (for videos). Experiments show that the distilled evaluators maintain strong agreement with human judgments and substantially reduce deployment cost relative to the teacher model. UniEditBench provides a practical and reproducible protocol for benchmarking modern visual editing methods. Our benchmark and the associated reward models are publicly available at https://github.com/wesar1/UniEditBench.
While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (R2=0.91, p<0.001). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.