cs.CVMar 30, 2026
SaveGEditBench v2: A Human-Aligned Benchmark for General Image Editing
Organizations: Nanyang Technological University · StepFun · Southeast University
Abstract
Recent advances in image editing have enabled models to handle complex instructions with impressive realism. However, existing evaluation frameworks lag behind: current benchmarks suffer from narrow task coverage, while standard metrics fail to adequately capture visual consistency, i.e., the preservation of identity, structure and semantic coherence between edited and original images. To address these limitations, we introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks, including a dedicated open-set category for unconstrained, out-of-distribution editing instructions beyond predefined tasks. Furthermore, we propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines. Besides, we construct VCReward-Bench using expert-annotated preference pairs to assess the alignment of PVC-Judge with human judgments on visual consistency evaluation. Experiments show that our PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average. Finally, by benchmarking 16 frontier editing models, we show that GEditBench v2 enables more human-aligned evaluation, revealing critical limitations of current models, and providing a reliable foundation for advancing precise image editing.
Figures & tables
Figure 1: GEditBench v2 spans 23 diverse image editing tasks, ranging from predefined edits to complex open-set real-world instructions, offering a comprehensive testbed for evaluating instruction-based image editing models. The central rose diagram visualizes the corresponding count distribution.
| Benchmark | Size | Tasks | Complex Edit | Open-set | Evaluation | |
| Support | Instruction | Unify Metrics | Human-Aligned | |||
| KontextBench Labs et al. (2025) | 1,026 | 5 | ✗ | ✗ | ✗ | |
| AnyEdit-Bench Yu et al. (2025) | 1,250 | 25 | ✗ | ✗ | ✗ | Low |
| ImgEdit-Bench Ye et al. (2025c) | 811 | 14 | ✗ | ✗ | ✓ | Mid |
| GEdit-Bench Liu et al. (2025) | 606 | 11 | ✗ | ✗ | ✓ | Mid |
| ICE-Bench Pan et al. (2025) | 6,538 | 31 | ✗ | ✗ | ✗ | Low |
Table 1: Comparison of general image editing benchmarks. GEditBench v2 pioneers a necessary transition to complex open-set scenarios, spanning 23 diverse tasks curated from real-world queries to establish a truly comprehensive evaluation standard.
Figure 2: Human preference agreement of pointwise and pairwise evaluation paradigms across four VLMs in (a) instruction following, (b) visual quality, and (c) visual consistency dimensions. Pairwise evaluation consistently achieves higher agreement with human judgments, suggesting its superior human alignment over prior absolute scoring. NC and SC prompts are adopted from ( Ye et al., 2025b ) and ( Luo et al., 2025 ) , respectively.
Figure 3: Two-stage candidates curation pipeline with prompt filtering.
Figure 4: Average accuracy of different image-instruction pairs per task on six representative tasks for visual consistency. Performance improves steadily and saturates around 1,500.
Figure 5: Preference data construction pipelines. (A) Object-centric pipeline: instructions are parsed to localize edited entities, partitioning the image into edit and non-edit regions. Region-specific metrics are then applied to enforce background fidelity while evaluating identity consistency within edited areas. (B) Human-centric pipeline: extends the object-centric pipeline by decomposing human attributes (face identity, body appearance, hair appearance) and dynamically excluding the edited attribute, enabling fine-grained consistency evaluation using specialized expert models.
Figure 6: Task-wise primary score distributions for preference pair synthesis. (a-b) Shaded regions indicate the top/bottom 30% Z-score threshold used to identify Winners (green) and Losers (red) with a clear optimization margin for preference learning. (c-d) Combined with Pareto filtering, candidate pairs that only exhibit score separation across two primary metrics are retained.
Figure 7: Average accuracy on three tasks of different in-group pairs ( ) for visual consistency. =6, chosen as the optimal balance between supervision density and computation cost.
| Benchmark | Size | Tasks |
| EditScore ( Luo et al., 2025 ) | 3K | 11 |
| EditReward ( Wu et al., 2025c ) | 1.5K | - |
| VCReward-Bench | 3.5K | 21 |
Table 2: Comparison of VCReward-Bench with existing reward benchmarks.
Figure 8: Human alignment of assessment models for visual consistency on (a) EditReward-Bench ( Luo et al., 2025 ) and (b) VCReward-Bench. Notably, our model achieves state-of-the-art performance among open-source evaluators across nearly all tasks, performing on par with or even surpassing the proprietary GPT-5.1-2026-02-28.
| Model | Samples | Instruction Following | Visual Quality | Visual Consistency | Overall | Arena ⋆ | |||||
| ELO | 95% CI | ELO | 95% CI | ELO | 95% CI | ELO | 95% CI | ELO | Rank | ||
| Nano Banana Pro (26-03-04) | 1,156 | 1,126 | -13/+15 | 1,066 | -9/+10 | 1,108 | -11/+11 | 1,096 | -6/+6 | 1,251 | 2 |
| Seedream 4.5 (26-03-11) | 1,190 | 1,111 | -12/+12 | 1,142 | -11/+11 | 1,030 | -11/+12 | 1,089 | -7/+7 | 1,196 | 3 |
| GPT Image 1.5 (26-03-04) | 1,081 | 1,260 | -13/+15 | 1,149 | -12/+12 | 846 | -13/+13 | 1,071 | -7/+6 | 1,270 | 1 |
| FLUX.2 [klein] 9B | 1,200 | 1,083 | -13/+12 | 1,025 | -11/+10 | 1,019 | -10/+9 | 1,039 | -6/+6 | 1,166 | 4 |
| Qwen-Image-Edit-2511 | 1,200 | 1,095 | -10/+10 | 1,060 | -11/+11 | 972 | -9/+10 | 1,038 | -6/+6 | 1,164 | 5 |
Table 3: OpenEdit leaderboard. Models are ranked by Elo ratings derived from a pairwise comparison paradigm. Instruction Following and Visual Quality are assessed by GPT-4o (26-03-24), while Visual Consistency is exclusively evaluated by our proposed PVC-Judge. Overall Elo scores and their 95% Confidence Intervals (CI) are computed via 1,000 bootstrap iterations. means closed-source, means open-source. ⋆ The Elo scores of Arena were recorded on March 26, 2026.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: (Input Image, Instruction) examples for the Relation Change task.
Figure 10: (Input Image, Instruction) examples for the Text Editing task. Our benchmark comprehensively evaluates fundamental semantic modifications alongside highly demanding typographic layout changes and precise font alterations.
Figure 11: (Input Image, Instruction) examples for the Chart Editing task.
Figure 12: (Input Image, Instruction) examples for the Camera Motion task.
Figure 13: (Input Image, Instruction) examples for the Object Reference task.
Figure 14: (Input Image, Instruction) examples for the Open-Set task.
Figure 15: Comparison results of three different prompting strategies for visual consistency evaluation on EditReward-Bench ( Luo et al., 2025 ) . (a) and (b) show the results of Gemini 3 Pro and Qwen3-VL-8B-Instruct, respectively. Pairwise statistical significance is evaluated using the two–sided Mann–Whitney U test ( , , ns: ). (c) reports the average evaluation time over 890 testing pairs measured with Qwen3-VL-8B-Instruct.
| Sub-Task | Pipeline Type | Grounding Image | Region-Specific Metrics | |
| Edit Region ( ) | Non-edit Region ( ) | |||
| subject addition | object-centric | output | SSIM, LPIPS , EMD | |
| subject removal | object-centric | input | SSIM, LPIPS , EMD | |
| subject replace | object-centric | input,output | SSIM, LPIPS , EMD | |
| size adjustment | object-centric | input,output | SAM-based CLIP [CLS] similarity, SAM-based DINO [CLS] similarity | LPIPS , EMD |
| color alteration | object-centric | input,output | L-channel SSIM , DINO structure similarity | LPIPS , EMD |
Table 4: (Sub-Task, Pipeline, Region-Specific Metrics) mapping for preference pair automated annotation. Primary metrics are marked in bold .
Figure 16: Distribution of the preference dataset. The donut chart shows the detailed allocation across 16 editing tasks, providing reliable signals for training the PVC-Judge model.
| Hyper-parameter | Value |
| optimizer | AdamW |
| num_train_epochs | 3 |
| per_device_train_batch_size | 2 |
| gradient_accumulation_steps | 1 |
| learning_rate | 2.0e-6 |
| warmup_ratio | 0.05 |
Table 5: LoRA Training Parameters.
Figure 17: Screenshots of our custom-built annotation interface for VCReward-Bench.
Figure 18: Distribution of VCReward-Bench instances by tasks.
| Sub-Tasks | Gemini 3 Pro | GPT 5.1 | EditReward | EditScore Avg@4 | Qwen3-VL-8B-Instruct | PVC-Judge |
| ( Wu et al., 2025c ) | ( Luo et al., 2025 ) | |||||
| Subject Addition | 80.88 | 80.88 | 74.38 | 57.65 | 72.06 | 77.94 |
| Subject Removal | 83.56 | 63.01 | 89.90 | 51.91 | 61.64 | 67.12 |
| Subject Replace | 90.57 | 77.36 | 81.55 | 54.85 | 73.58 | 84.91 |
| Color Alteration | 91.30 | 86.96 | 83.49 | 77.83 | 72.46 | 94.20 |
| Material Modification | 90.16 | 88.52 | 67.00 | 51.23 | 75.41 | 83.61 |
Table 6: Numerical results of assessment models for visual consistency on EditReward-Bench ( Luo et al., 2025 ) . Bold marks the best-performing open-source model.
| Sub-Tasks | Gemini 3 Pro | GPT 5.1 | EditReward Wu et al. (2025c) | EditScore Avg@4 Luo et al. (2025) | Qwen3-VL-8B-Instruct | PVC-Judge |
| Subject Addition | 93.59 | 80.68 | 67.39 | 41.30 | 72.53 | 86.96 |
| Subject Removal | 86.32 | 68.49 | 70.97 | 35.48 | 69.01 | 74.19 |
| Subject Replace | 85.88 | 67.96 | 65.14 | 36.70 | 74.00 | 82.57 |
| Size Adjustment | 83.52 | 69.45 | 57.34 | 34.81 | 68.22 | 72.35 |
| Color Alteration | 92.90 | 75.96 | 75.46 | 53.24 | 67.01 | 87.04 |
| Material Modification | 86.91 | 80.09 | 57.02 | 50.88 | 72.89 | 86.84 |
Table 7: Numerical results of assessment models for visual consistency on our VCReward-Bench. Bold marks the best-performing open-source model.
Figure 19: Open-set editing examples across six representative models. Many open-source models fail to fully capture implicit user instructions, leading to partial or inconsistent edits. This highlights the need for open-set evaluation to reveal limitations of current generative editing models.
Figure 20: Visualization results of relation change task. Open-source models often struggle to capture complex spatial relationships, leading to partial or simplified relational edits, while closed-source models better preserve inter-object dependencies.
Figure 21: Fine-grained detail generation across multiple tasks. Open-source models frequently distort small entities and background subjects, whereas closed-source models better maintain structure, with some difficulty preserving identity on small faces.
Figure 22: The pairwise evaluation prompt for the Instruction Following dimension.
Figure 23: The pairwise evaluation prompt for the Visual Quality dimension.
Figure 24: The pairwise evaluation prompt for the Visual Consistency dimension.