High-quality source-target image pairs with precise editing instructions are essential for instruction-guided image editing, yet constructing such training triplets at scale remains costly. Recent pipelines often rely on vision-language models to synthesize editing instructions automatically, but we find that strong VLMs still struggle to describe visual transformations between image pairs. In particular, they exhibit three recurring failure modes: orientation inconsistency, viewpoint ambiguity, and missing fine-grained attributes. In a human evaluation on 400 image pairs, several open-source VLM baselines produce critical-error rates above 47%, making many synthesized instructions unsuitable for downstream training. To address this, we propose EditCaption, a two-stage post-training pipeline for image editing instruction synthesis. First, we construct a 100K supervised fine-tuning dataset through GLM-based auto-captioning, EditScore filtering, and human refinement. Second, we collect 10K human-annotated preference pairs, where each rejected instruction is labeled with its primary error type and severity. Based on this dataset, we propose Hardness-Adaptive Error-Aware DPO (HAE-DPO), a task-adapted DPO objective that introduces an adaptive margin based on human-labeled severity, failure-mode type, and reference-model hardness. Experiments across three benchmarks demonstrate that our 235B model with SFT+HAE-DPO achieves state-of-the-art performance among open-source and closed models, scoring 4.720 on Eval-400, 4.672 on HQ-Edit, and 4.651 on ByteMorph-Bench -- surpassing Gemini-3-Pro on all three. Human evaluation confirms critical error rates drop from 47.75% to 17.50%, with correct rates improving from 41.75% to 70.25%, surpassing Gemini-3-Pro (66.00%).
Figures & tables
Figure 1: Overview of our approach. (a) Task & Failure Modes: Given a source-target image pair, VLMs must generate accurate editing instructions but suffer from three systematic failures: directional confusion, viewpoint ambiguity, and missing fine-grained detail. (b) HAE-DPO: SFT model outputs serve as rejected samples; human-corrected instructions as chosen samples. Each rejection is annotated with an error type and severity score, enabling HAE-DPO to modulate the preference margin beyond uniform DPO.
Figure 2: Overview of Data Distribution. Our training dataset is divided into three categories: Semantic Editing (content-based modifications), Stylistic Editing (aesthetic adjustments), and Structural Editing(spatial arrangement and composition). Our data collection strategy ensures a balance of diversity and quality throughout the training process, providing comprehensive coverage and precise annotations to foster robust model training.
Figure 3: The three-step SFT data construction pipeline.
Dataset
#Samples
Characteristics
In-house Test Set
400
Diverse editing operations, covering spatial, viewpoint, and attribute changes
ByteMorph-Bench
600
Non-rigid motion transformations with spatial and compositional challenges
HQ-Edit
500
High-resolution images with rich details and comprehensive editing instructions
Table 1: Summary of evaluation datasets
Model
Correct ↑
P0 ↓
P1
P2
Closed-source models
Gemini-3-Pro
66.00%
21.00%
12.00%
1.00%
GPT-4.1
48.75%
42.25%
8.25%
0.75%
Open-source baseline models
GLM-4.5V (106B-A12B)
38.60%
51.20%
9.20%
1.00%
Qwen3-VL-235B-A22B (base)
41.75%
47.75%
8.50%
2.00%
Table 2: Human evaluation results on 400 image pairs. Each model generates one instruction per pair. P0: critical errors (instruction unusable); P1: moderate errors; P2: minor issues. Higher Correct and lower P0 error rates indicate better performance.
Eval-400 (In-house)
HQ-Edit
ByteMorph-Bench
Model
S
Acc
Comp
Clar
S
Acc
Comp
Clar
S
Acc
Comp
Clar
Closed-source models
Gemini-3-Pro
4.706
4.70
4.85
4.43
4.658
4.54
4.89
4.43
4.522
4.55
4.70
4.11
GPT-4.1
4.220
4.03
4.60
3.84
4.507
4.57
4.86
3.68
3.412
3.27
3.68
3.14
Open-source baseline models
Qwen3.5-397B-A17B
4.380
4.30
4.62
4.06
4.383
4.07
4.85
4.07
3.867
3.71
4.16
3.61
Table 3: Objective evaluation results on three benchmarks. Scores are reported as weighted composites ( S=0.4×Acc+0.4×Comp+0.2×Clar , scale 0–5). Best and second-best scores are highlighted in bold and underline, respectively. † marks our fine-tuned models.
Eval-400
HQ-Edit
ByteMorph
Components
S
Acc
Comp
Clar
S
Acc
Comp
Clar
S
Acc
Comp
Clar
None (Standard DPO)
4.662
4.63
4.85
4.34
4.598
4.46
4.89
4.29
4.564
4.68
4.72
4.02
Only sic(ti) (severity-type)
4.698
4.70
4.86
4.37
4.648
4.55
4.90
4.34
4.612
4.73
4.75
4.10
Only hi (hardness)
4.710
4.71
4.88
4.37
4.664
4.58
4.89
4.38
4.630
4.74
4.77
4.13
HAE-DPO
4.720
4.72
4.89
4.38
4.672
4.61
4.87
4.40
4.651
4.75
4.79
4.17
Table 4: Ablation study on HAE-DPO margin components using Qwen3-VL-235B-A22B with the SFT stage fixed. The term sic(ti) denotes the severity-type product, where si is the error severity and c(ti) is the failure-type weight. The term hi denotes model-perceived hardness. Rows “Only sic(ti) ” and “Only hi ” add the corresponding margin component alone, while HAE-DPO uses both. S=0.4× Acc +0.4× Comp +0.2× Clar.
Model
ImgEdit ↑
GEdit-EN ↑
GEdit-CN ↑
Step1X-Edit
3.062
6.444
6.658
+ Base VLM captioner
3.358
6.583
6.774
+ EditCaption (ours)
3.882
7.158
7.312
Table 5: Downstream image editing evaluation on ImgEdit-Bench and GEdit-Bench (EN and CN). All models are fine-tuned from Step1X-Edit on 1M triplets annotated by different captioning sources. Scores are overall ratings evaluated by GPT-4.1.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Severity
Category
Problem Description
Examples / Notes
1
P0
(Subject) Text Recognition / Editing Error
Instruction conflicts with text in image; text unchanged, incorrectly modified, garbled, or misspelled.
Instruction: “change sign to ‘OPEN’ ”, but target still shows “SHOP”; or text becomes distorted/garbled.
P1
(Background) Text Recognition / Editing Error
Same as above.
Same as above.
2
P0
(Subject) Instruction or Attribute Error
Primary task instruction or attribute is incorrect; or affected region exceeds 1/10 of total image area. Covers: color, shape, material, style, quantity, etc.
“Change black skirt to blue” but result is grey; “Change two cats to three” but target still has two; “Enlarge to 15cm” or “Rotate 37 degrees” (unverifiable from image).
P1
(Non-subject) Instruction or Attribute Error
Non-primary task; affected area below 1/10 of image. Same instruction types as above.
Same as above.
3
P0
(Subject) Instruction or Content Omission
Primary task instruction or attribute is missing. Includes: missing add/remove action; missing specific attributes (color, position) when adding/deleting; ambiguous editing target.
Original has no hat → target has hat, but “add hat” not mentioned; “Add flower” without specifying color; “Delete person” when multiple people are present; “Remove old flower + add new flower” should use replace ; “Put object somewhere” should use move not add ; missing description of border/frame edits.
P1
(Non-subject) Instruction or Content Omission
Non-primary task; affected area below 1/10 of image. Same instruction types as above.
Same as above.
Appendix
Table 6: Annotation Quality Checklist for Image Editing Instructions
Figure 4: Qualitative examples of generated editing instructions. Given the same source-target image pairs, our model produces instructions that are more spatially precise and complete compared to Gemini-3-Pro and GLM4.5V. Green denotes key attributes where our model outperforms baselines; Red denotes errors or omissions.
Images
Annotation
[Primary Changes] 1. The dress color changes from grey to pink. 2. The shoulder strap style changes from wide straps to thin spaghetti straps. 3. The dress hem length is shortened from midi-length (approximately mid-calf) to a mini skirt (approximately mid-thigh). [Secondary Changes] 1. The gold necklace around the neck is replaced with a thin silver necklace. [Overall Description] The woman in the image, originally wearing a grey wide-strap midi dress, has been edited to wear a pink spaghetti-strap mini dress. Her gold necklace has been replaced with a thin silver one. Her pose and white handbag remain unchanged.
Appendix
Table 7: An annotation example of image editing instructions.
Score
Criteria
Case
Reasoning
5
Perfect accuracy • All facts correct • Coverage ≥ 50%
“Change dress to pink spaghetti-strap; necklace to silver.”
All facts correct: pink ✓, spaghetti-strap ✓, silver necklace ✓. Coverage 2/3 (67%) ≥ 50%. Note: dress length not mentioned, but accuracy is perfect and coverage is sufficient.
4
Core correct, minor flaw • Major facts correct • Coverage ≥ 30% • Error on unchanged elements
“Change dress to pink; bag to red.”
Dress pink ✓. Bag unchanged but turned red × . Non-core error: − 1.
3
Single error or conservative penalty • 1 major factual error • Or coverage < 30%
A: “Dress to blue.” B: “Necklace to silver.”
A: GT is pink; blue × . B: Dress coverage =0× . Prevents skipping primary edits.
2
Multiple errors or hallucination • ≥ 2 major errors • Or fabricated operations
A: “Long blue dress.” B: “Pink dress; add hat.”
A: Color × + length × . B: “Add hat” not in GT × . Severely misleads.
1
Completely wrong • Wrong object • Unrelated to image
“Change car to red.”
No car in image × .
Appendix
Table 8: Representative scoring examples for the Accuracy dimension, with corresponding instructions and reasoning.
Hyperparameter
SFT
HAE-DPO
Learning rate
1×10−5
5×10−6
Global batch size
512
128
Training epochs
6
4
Max sequence length
8192
8192
Learning-rate schedule
Cosine decay
Cosine decay
Warmup ratio
0.03
0.03
Appendix
Table 11: Training hyperparameters for the SFT and HAE-DPO stages. Batch size denotes the global effective batch size after gradient accumulation.
Instruction guided image editing has advanced substantially with recent generative models, yet it still fails to produce reliable results across many seemingly simple cases. We observe that a large portion of these failures stem not from insufficient model capacity, but from poorly formulated editing tasks, such as those involving small targets, implicit spatial relations, or under-specified instructions. In this work, we frame image editing failures as a task formulation problem and propose an adaptive task reformulation framework that improves editing performance without modifying the underlying model. Our key idea is to transform the original image-instruction pair into a sequence of operations that are dynamically determined and executed by a MLLM agent through analysis, routing, reformulation, and feedback-driven refinement. Experiments on multiple benchmarks, including ImgEdit, PICA, and RePlan, across diverse editing backbones such as Qwen Image Edit and Nano Banana, show consistent improvements, with especially large gains on challenging cases. These results suggest that task reformulation is a critical but underexplored factor, and that substantial gains can be achieved by better matching editing tasks to the effective operating regime of existing models.
Bo Zhao, Kairui Guo, Runnan Du +6
Nanjing University China · Beijing Institute of Technology China · College of Artificial Intelligence, China University of Petroleum (Beijing) China +3
Instruction-guided image editing has a training-time blind spot. Generative editors are never required to semantically verify whether their outputs actually satisfy the instruction. Supervision stops at reconstruction and input textual-level conditioning. This produces incomplete edits, spatial spillover, and poor localization. We present IABEdit, a model-agnostic framework that embeds differentiable semantic verification into training. A frozen vision-language model extracts spatially-aware descriptors from the ground-truth edit. A trainable aligner then reproduces them from the generated output. The residual between the two becomes a gradient that teaches the generator both what to edit and where, with no inference-time VLM cost. IABEdit is compatible with diverse backbones, including U-Net (Stable Diffusion) and MMDiT (FLUX), without altering their inference pipelines. On MagicBrush, it improves structural fidelity by +3.49 DINO-I over the best diffusion baseline and +1.26 over the best overall baseline, while remaining competitive on instruction alignment. It also achieves state-of-the-art instruction adherence performance on RealEdit and EMU Edit benchmarks based on embedding-based metrics. Most consequentially, on the D-LORD surveillance benchmark, it surpasses the proprietary Gemini agent by +5.13 DINO-P under heavy occlusion, where preserving identity is hardest. This shows that gradient-aligned VLM distillation holds up under real-world-like surveillance and occlusion conditions. Human and GPT-4o evaluations confirm perceptually precise, well-localized edits.
In text-guided image editing, describing the desired change is often straightforward, but identifying the intended object or region can be cumbersome, especially when several objects look alike. We introduce a new image editing interface that lets users place spatial marks and optional short notes directly on the image. Together, these annotations form a canvas instruction that specifies where to edit and what to change. Our editor, VibeEdit, follows these instructions to perform object addition, removal, replacement, attribute modification, and movement without a separate text prompt. We construct 1.55 million source-target edit pairs with object masks and structured edit descriptions, from which we render canvas instructions during training. We adapt Qwen-Image-Edit with layer-decoupled conditioning that separately encodes source images and canvas instructions for image editing. We train the model with region-weighted supervised fine-tuning, followed by rubric-guided reinforcement learning to improve edit completion, local edit quality, and preservation of unedited regions. We evaluate VibeEdit on an independently constructed, human-curated benchmark of 419 cases emphasizing target selection among similar objects. VibeEdit achieves a VLM rubric score of 79.9 and an outside-region PSNR of 32.8 dB, compared with 67.4 and 24.0 dB for FireRed, the highest-scoring text-instructed baseline in our evaluation.
Jinjing Zhao, Fangyun Wei, Yitong Wang +9
University of Sydney · Microsoft Research · Fudan University +5