From Evaluation to Evolution: Benchmarking Image Editing and Evolving Image Models
Organizations: Sun Yat-sen University · WeChat Vision, Tencent Inc. · Xidian University · Fudan University ·
Abstract
Real-world image editing requires coordinating multiple visual transformations with content preservation and reference constraints, yet aggregate scores can readily obscure unmet local requirements. Effective evaluation must therefore identify local failures and translate them into priorities for subsequent model improvement. We introduce WeGenEditBench, comprising 2,740 evaluation instances and 148 task types across single-image, text-intensive, and multi-reference editing. Pre-generated checklists shared across models decompose each request into atomic requirements; item-level deductions and visual evidence are established first and then summarized into scores for edit execution, preservation, visual quality, and task-specific correctness. To support the experience-acquisition component of recursive self-improvement (RSI), we develop WeGenEvoStudio, a unified multi-agent harness for RSI-oriented learning that integrates model-conditioned failure diagnosis, retrieval-grounded data construction, supervised adaptation, and re-evaluation. Among 15 evaluated configurations, GPT-Image-2 leads single-image editing with an Overall score of 4.076, while Seedream-5.0-Pro leads text-intensive editing at 4.291. Evolution experiments across four generation and editing benchmarks demonstrate substantial targeted gains: on their report-selected Top-5 single-image targets, FLUX.2 [dev] improves Overall from 1.083 to 3.517, while Qwen-Image-Edit-2511 gains 13.2%. Together, our benchmark and harness provide a foundation for developing more reliable image models through interpretable evaluation and feedback-driven learning.
Figures & tables
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.298 | 4.049 | 4.463 | 4.182 | 4.076 |
| Nano-Banana-Pro | 4.159 | 3.752 | 4.370 | 4.050 | 3.882 |
| Qwen-Image-3.0 | 4.120 | 3.469 | 4.331 | 4.126 | 3.813 |
| Seedream-5.0-Pro | 4.207 | 4.084 | 4.434 | 4.206 | 4.029 |
| Open-source Models | |||||
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.289 | 4.031 | 4.329 | 4.399 | 4.604 | 3.919 | 4.421 | 4.452 | 4.132 |
| Nano-Banana-Pro | 4.295 | 4.092 | 4.310 | 4.322 | 4.396 | 4.208 | 4.232 | 4.306 | 4.116 |
| Qwen-Image-3.0 | 4.367 | 3.732 | 4.329 | 4.385 | 4.509 | 3.869 | 4.406 | 4.414 | 4.109 |
| Seedream-5.0-Pro | 4.415 | 4.327 | 4.416 | 4.504 | 4.362 | 4.207 | 4.455 | 4.429 | 4.291 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.358 | 4.134 | 3.707 | 4.138 |
| Nano-Banana-Pro | 4.031 | 3.993 | 3.340 | 3.877 |
| Seedream-5.0-Pro | 4.270 | 4.093 | 3.750 | 4.095 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.279 | 3.493 | 3.091 | 3.327 |
| Model | T2I | Image editing | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WeGen Bench | WeGenEditBench-SingleImageEdit | GEdit | ImgEdit | ||||||||||
| F | EG | PR | VQ | DS | F | SC | PQ | O | I | Q | D | F | |
| Qwen-Image | 7.700 | 3.244 | 2.646 | 3.500 | 3.565 | 2.834 | 8.515 | 7.131 | 7.511 | 4.621 | 4.374 | 4.400 | 4.465 |
| + WeGenEvoStudio | 8.668 | 3.705 | 2.650 | 4.105 | 3.437 | 3.208 | 8.172 | 7.241 | 7.476 | 4.621 | 4.416 | 4.450 | 4.496 |
| +0.968 | +0.461 | +0.003 | +0.605 | -0.128 | +0.373 | -0.343 | +0.109 | -0.035 | +0.000 | +0.042 | +0.051 | +0.031 | |
| FLUX.2 [dev] | 7.767 | 1.550 | 3.178 | 2.780 | 0.809 | 1.083 | 3.425 | 7.592 | 3.465 | 1.915 | 2.626 | 2.873 | 2.471 |
Appendix figures & tables67 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Single-image | Text-intensive | Multi-reference |
|---|---|---|---|
| Instances | 1,008 | 1,000 | 732 |
| Task families | 8 | 5 | 6 |
| Task types | 65 | 35 | 48 |
| Mean instruction length | 79.2 | 124.6 | 146.3 |
| Median instruction length | 60.5 | 126 | 116.5 |
| Instruction length range | 12–255 | 10–344 | 30–439 |
| Benchmark | Taxonomy | Tokens | Task emphasis | Inputs |
|---|---|---|---|---|
| GEdit-Bench [ 22 ] | 11 categories | 2–38 | General editing | 1 |
| ImgEdit [ 46 ] | 9 Basic types | 4–60 | Single/multi-turn | 1 |
| REDEdit [ 33 ] | 15 categories | — | General/text | 1 |
| RISEBench [ 56 ] | 4 families / 16 types | — | Implicit reasoning | 1 |
| UnicBench [ 45 ] | 4 families / 22 types | 6–42 | General/reasoning | 1 |
| UniREditBench [ 12 ] | 8 dimensions / 18 subtypes | — | Real/game reasoning | 1 |
| Benchmark | Multi-dim. | Verification criteria | Retained judgment unit |
|---|---|---|---|
| GEdit-Bench | Yes | General rubric | SC/PQ scores; explanations in VIEScore |
| ImgEdit | Yes | Task-specific rubrics | Dimension scores + brief reason |
| REDEdit | Yes | Task/text criteria; OCR | Dimension scores; text-error metrics |
| RISEBench | Yes | Reasoning guidelines | Reasoning, consistency, plausibility |
| UnicBench | Yes | Instance reasoning points | Dimension scores + reason |
| UniREditBench | Yes | Text + GT image references | Reference-aided dimension scores |
| Subset | Overall | Count | Rate | 95% CI |
|---|---|---|---|---|
| Single-image | 315/644 | 48.9% | [45.1, 52.8]% | |
| Text-intensive | 171/666 | 25.7% | [22.4, 29.0]% | |
| Multi-reference | 119/503 | 23.7% | [19.9, 27.2]% | |
| Single-image | 101/323 | 31.3% | [26.5, 36.6]% |
| Model | Both sides | Text only | Image only | Neither |
|---|---|---|---|---|
| GPT-Image-2 | 70 (25.0%) | 29 | 77 | 104 |
| Seedream-5.0-Pro | 76 (27.1%) | 49 | 66 | 89 |
| Model | WeGenBench | GEdit | ImgEdit | |||||
|---|---|---|---|---|---|---|---|---|
| F | SC | PQ | O | I | Q | D | F | |
| Qwen-Image | 8.170 | 8.774 | 7.676 | 7.962 | 4.704 | 4.438 | 4.490 | 4.544 |
| + WeGenEvoStudio | 8.840 | 8.338 | 7.597 | 7.772 | 4.688 | 4.474 | 4.476 | 4.546 |
| +0.670 | -0.436 | -0.079 | -0.190 | -0.016 | +0.035 | -0.014 | +0.002 | |
| FLUX.2 [dev] | 8.352 | 4.388 | 7.621 | 4.395 | 2.323 | 3.124 | 3.207 | 2.885 |
| + WeGenEvoStudio | 8.475 | 7.785 | 8.132 | 7.404 | 4.170 | 4.099 | 4.190 | 4.153 |
| Category | SC | PQ | O | SC | PQ | O |
|---|---|---|---|---|---|---|
| motion_change | 8.114 | 8.430 | 7.907 | +5.747 | +0.848 | +5.295 |
| subject-remove | 7.272 | 8.316 | 6.911 | +3.991 | +0.351 | +3.706 |
| color_alter | 9.275 | 7.812 | 8.308 | +5.463 | +0.050 | +4.371 |
| material_alter | 7.625 | 7.600 | 7.002 | +4.438 | +0.575 | +3.678 |
| subject-replace | 7.817 | 7.933 | 7.226 | +3.658 | +0.425 | +3.173 |
| Scope | Pairs | Before | After | |
|---|---|---|---|---|
| Report Top-5 | 326 | 7.767 | 7.997 | +0.230 |
| Full benchmark | 995 | 8.352 | 8.475 | +0.124 |
| Setting | Counter- intuitive | Icons / symbols | Animals | IP | Top-5 | Full |
|---|---|---|---|---|---|---|
| Baseline | 7.150 | 7.540 | 8.000 | 7.810 | 7.700 | 8.170 |
| Without web search | 8.450 | 8.440 | 8.903 | 8.506 | 8.650 | 8.819 |
| With web search | 8.700 | 8.760 | 8.611 | 8.671 | 8.668 | 8.840 |
| +0.250 | +0.320 | -0.292 | +0.165 | +0.018 | +0.021 |
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.424 | 3.966 | 4.445 | 4.225 | 4.143 |
| Nano-Banana-Pro | 4.317 | 3.640 | 4.402 | 4.056 | 3.963 |
| Qwen-Image-3.0 | 4.228 | 3.517 | 4.359 | 4.125 | 3.893 |
| Seedream-5.0-Pro | 4.328 | 3.985 | 4.424 | 4.251 | 4.091 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.096 | 4.068 | 4.494 | 4.215 | 3.974 |
| Nano-Banana-Pro | 3.983 | 3.880 | 4.424 | 4.010 | 3.803 |
| Qwen-Image-3.0 | 3.916 | 3.485 | 4.324 | 4.149 | 3.699 |
| Seedream-5.0-Pro | 4.035 | 4.163 | 4.441 | 4.189 | 3.939 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.443 | 4.156 | 4.464 | 4.328 | 4.222 |
| Nano-Banana-Pro | 4.390 | 3.696 | 4.350 | 4.226 | 4.050 |
| Qwen-Image-3.0 | 4.332 | 3.451 | 4.337 | 4.279 | 3.977 |
| Seedream-5.0-Pro | 4.376 | 4.125 | 4.402 | 4.325 | 4.162 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.228 | 4.134 | 4.433 | 4.143 | 4.036 |
| Nano-Banana-Pro | 4.138 | 3.745 | 4.423 | 3.992 | 3.864 |
| Qwen-Image-3.0 | 4.153 | 3.627 | 4.385 | 3.923 | 3.825 |
| Seedream-5.0-Pro | 4.155 | 4.074 | 4.429 | 4.184 | 3.990 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.117 | 3.372 | 4.314 | 4.125 | 3.789 |
| Nano-Banana-Pro | 4.223 | 3.269 | 4.234 | 4.088 | 3.806 |
| Qwen-Image-3.0 | 4.190 | 3.216 | 4.230 | 4.153 | 3.792 |
| Seedream-5.0-Pro | 4.003 | 3.806 | 4.471 | 4.264 | 3.874 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.347 | 4.425 | 4.565 | 4.462 | 4.272 |
| Nano-Banana-Pro | 3.612 | 4.003 | 4.550 | 4.392 | 3.726 |
| Qwen-Image-3.0 | 3.672 | 3.950 | 4.313 | 4.290 | 3.678 |
| Seedream-5.0-Pro | 4.117 | 4.202 | 4.465 | 4.362 | 4.044 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.376 | 3.662 | 4.364 | 4.009 | 3.983 |
| Nano-Banana-Pro | 4.353 | 3.317 | 4.190 | 3.762 | 3.804 |
| Qwen-Image-3.0 | 4.190 | 3.047 | 4.198 | 3.910 | 3.691 |
| Seedream-5.0-Pro | 4.225 | 3.613 | 4.312 | 4.047 | 3.882 |
| Open-source Models | |||||
| Model Name | EG | PR | VQ | DS | Overall |
|---|---|---|---|---|---|
| Closed-source Models | |||||
| GPT-Image-2 | 4.450 | 4.105 | 3.802 | 3.671 | 3.919 |
| Nano-Banana-Pro | 4.566 | 3.703 | 3.960 | 3.774 | 3.965 |
| Qwen-Image-3.0 | 4.496 | 3.264 | 3.850 | 3.880 | 3.840 |
| Seedream-5.0-Pro | 4.410 | 4.340 | 4.097 | 4.082 | 4.104 |
| Open-source Models | |||||
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.249 | 4.050 | 4.308 | 4.356 | 4.594 | 3.924 | 4.409 | 4.446 | 4.109 |
| Nano-Banana-Pro | 4.317 | 4.129 | 4.303 | 4.305 | 4.392 | 4.220 | 4.227 | 4.295 | 4.141 |
| Qwen-Image-3.0 | 4.350 | 3.692 | 4.289 | 4.395 | 4.507 | 3.887 | 4.408 | 4.410 | 4.096 |
| Seedream-5.0-Pro | 4.366 | 4.327 | 4.412 | 4.475 | 4.352 | 4.218 | 4.447 | 4.428 | 4.260 |
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.144 | 3.817 | 4.372 | 4.439 | 4.567 | 3.908 | 4.483 | 4.471 | 4.038 |
| Nano-Banana-Pro | 4.032 | 4.032 | 4.326 | 4.177 | 4.304 | 4.208 | 4.216 | 4.268 | 3.913 |
| Qwen-Image-3.0 | 4.257 | 3.509 | 4.477 | 4.289 | 4.528 | 3.854 | 4.463 | 4.390 | 4.015 |
| Seedream-5.0-Pro | 4.288 | 4.152 | 4.316 | 4.496 | 4.260 | 4.187 | 4.528 | 4.407 | 4.152 |
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.341 | 4.120 | 4.329 | 4.423 | 4.639 | 4.139 | 4.500 | 4.583 | 4.187 |
| Nano-Banana-Pro | 4.260 | 3.949 | 4.333 | 4.434 | 4.595 | 4.351 | 4.351 | 4.351 | 4.103 |
| Qwen-Image-3.0 | 4.385 | 3.856 | 4.299 | 4.297 | 4.639 | 3.833 | 4.167 | 4.500 | 4.106 |
| Seedream-5.0-Pro | 4.535 | 4.371 | 4.435 | 4.576 | 4.541 | 4.027 | 4.486 | 4.568 | 4.392 |
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.448 | 3.676 | 4.295 | 4.423 | 4.782 | 3.818 | 4.473 | 4.518 | 4.160 |
| Nano-Banana-Pro | 4.382 | 4.027 | 4.282 | 4.408 | 4.508 | 3.847 | 4.220 | 4.432 | 4.139 |
| Qwen-Image-3.0 | 4.418 | 3.455 | 4.391 | 4.515 | 4.508 | 3.763 | 4.322 | 4.356 | 4.139 |
| Seedream-5.0-Pro | 4.472 | 4.241 | 4.380 | 4.578 | 4.474 | 3.895 | 4.351 | 4.368 | 4.296 |
| Model Name | Text-side | Image-side | Overall | ||||||
| EG | PR | VQ | DS | EG | PR | VQ | DS | ||
| Closed-source Models | |||||||||
| GPT-Image-2 | 4.457 | 3.837 | 4.511 | 4.454 | 4.571 | 3.839 | 4.304 | 4.348 | 4.227 |
| Nano-Banana-Pro | 4.274 | 4.105 | 4.274 | 4.541 | 4.228 | 4.404 | 4.018 | 4.351 | 4.127 |
| Qwen-Image-3.0 | 4.448 | 3.615 | 4.521 | 4.549 | 4.534 | 3.810 | 4.483 | 4.466 | 4.209 |
| Seedream-5.0-Pro | 4.333 | 4.312 | 4.387 | 4.449 | 4.175 | 4.404 | 4.368 | 4.395 | 4.195 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.315 | 4.107 | 3.658 | 4.101 |
| Nano-Banana-Pro | 4.284 | 4.122 | 3.236 | 4.009 |
| Seedream-5.0-Pro | 4.374 | 4.094 | 3.835 | 4.154 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.496 | 3.656 | 3.216 | 3.504 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.220 | 4.068 | 3.339 | 3.983 |
| Nano-Banana-Pro | 3.795 | 3.889 | 3.043 | 3.682 |
| Seedream-5.0-Pro | 4.250 | 4.056 | 3.361 | 3.994 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.198 | 3.089 | 2.861 | 3.087 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.147 | 3.983 | 3.690 | 3.990 |
| Nano-Banana-Pro | 3.664 | 3.779 | 3.319 | 3.641 |
| Seedream-5.0-Pro | 4.018 | 3.936 | 3.697 | 3.921 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 2.700 | 3.130 | 2.730 | 2.878 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.453 | 4.222 | 3.795 | 4.229 |
| Nano-Banana-Pro | 4.060 | 3.966 | 3.368 | 3.884 |
| Seedream-5.0-Pro | 4.272 | 4.123 | 3.658 | 4.089 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.209 | 3.549 | 2.978 | 3.299 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.579 | 4.211 | 3.833 | 4.282 |
| Nano-Banana-Pro | 4.357 | 4.018 | 3.384 | 4.027 |
| Seedream-5.0-Pro | 4.463 | 4.176 | 3.926 | 4.241 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.787 | 3.734 | 3.468 | 3.702 |
| Model Name | EG | PR | VQ | Overall |
|---|---|---|---|---|
| Closed-source Models | ||||
| GPT-Image-2 | 4.455 | 4.223 | 3.955 | 4.263 |
| Nano-Banana-Pro | 3.955 | 4.153 | 3.739 | 3.991 |
| Seedream-5.0-Pro | 4.216 | 4.171 | 4.000 | 4.155 |
| Open-source Models | ||||
| HunyuanImage-3.0-Instruct-Distil | 3.255 | 3.798 | 3.287 | 3.479 |
| \TableCaptionLabel Table 33 Single-Image Task Taxonomy. Family headings give unique covered instances; rows define the second-level tasks. The subset has 1,008 instances and 65 task labels. | |
|---|---|
| Second-level task | Operational definition |
| Basic local editing 631 unique instances | |
| Object addition | Insert a specified object with correct location, scale, and occlusion. |
| Object removal | Remove a target, reconstruct its background, and preserve surrounding content. |
| Object replacement | Replace a target while respecting its scene role. |
| Object count change | Change the specified objects to the requested count. |
| \TableCaptionLabel Table 34 Text-Intensive Task Taxonomy. Family headings give unique covered instances; rows define the second-level tasks. The subset has 1,000 instances and 35 task labels. | |
|---|---|
| Second-level task | Operational definition |
| Basic text editing 678 unique instances | |
| Text-content replacement | Replace specified wording while preserving placement and surrounding content. |
| Text insertion | Insert wording without obscuring protected visual information. |
| Text removal | Remove text, reconstruct its carrier, and preserve non-target content. |
| Typography preservation | Change wording while preserving specified typography and arrangement. |
| \TableCaptionLabel Table 35 Multi-Reference Task Taxonomy. Family headings give unique covered instances; rows define the second-level tasks. The subset has 732 instances and 48 task labels. | |
|---|---|
| Second-level task | Operational definition |
| Portrait editing 150 unique instances | |
| Face transfer | Transfer facial identity while preserving required body, pose, and scene. |
| Reference-based clothing transfer | Transfer reference clothing while preserving the target person’s identity. |
| Hairstyle transfer | Transfer hairstyle without substituting unrelated identity attributes. |
| Expression transfer | Transfer expression while preserving the target person’s identity. |
Image Editing Evaluation
Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.
See all 99 papers