cs.CVFeb 27, 2026
SaveDLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model
Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Meituan Group
Abstract
Significant progress has been made in the field of Instruction-based Image Editing Models (IIEMs). However, while these models demonstrate plausible adherence to instructions and strong reasoning ability on current benchmarks, their ability to edit small objects remains underexplored, despite its importance for precise local editing and refining details in both real and generated images. In this paper, we introduce DeepLookEditBench (DLEBench), the first benchmark dedicated to assessing the abilities of IIEMs in editing small-scale objects. Specifically, we construct a challenging testbed comprising 1889 samples across seven instruction types. In these samples, target objects occupy only 1%-10% of the image area, covering complex scenarios such as partial occlusion and multi-object editing. To ensure robust evaluation on this benchmark, we propose an evaluation protocol with refined score rubrics to minimize subjectivity and ambiguity in two criteria: Instruction Following and Visual Consistency. This protocol also introduces a dual-mode evaluation framework (Tool-driven and Oracle-guided Modes) addressing the misalignment between LMM-as-a-Judge and human judgements on DLEBench. Empirical results on 10 IIEMs reveal significant performance gaps in small-scale object editing, highlighting the need for specialized benchmarks to advance this ability.
Figures & tables
Figure 1: An example illustrating the challenge of small-scale editing. Given the instruction to edit the green scarf, Gemini-3-Pro misidentifies the target and modifies the foreground object instead.
Figure 2: Overview of the three-stage data transformation pipeline. We begin by selecting a raw visual reasoning sample from V*-Bench, specifically one that inquires about the color of a woman’s scarf. In Stage 1, we employ a counterfactual synthesis strategy via GPT-4.1 to generate the image-edit metadata (as shown in the bottom red box). Subsequently, Stage 2 utilizes a crop-and-edit strategy with Gemini-3-Pro to generate the reference image. Finally, all data is submitted to Stage 3 for rigorous human verification.
Figure 3: Comparison of the Cumulative Distribution Function (CDF) of target object area ratios across different instruction-based image editing benchmarks.
Figure 4: Distribution of target object area ratios in DLEBench. The bars represent the number of samples in different area intervals, with specific counts annotated above each bar.
Figure 5: Overview of the input and output data for the tools employed in our Tool-driven Mode. These tools are also utilized when evaluating VC in the Oracle-guided Mode.
| Instruction Types | Closed-Source Models | Open-Source Models | ||||||||
| Gemini-3-Pro | GPT-Image-1 | OmniGen2 | Bagel-Think | UniREdit-Bagel | MagicBrush | Qwen-Edit | UniWorld-V1 | UniWorld-V2 | Step1X-Edit | |
| Change Material | 74.01 | 44.31 | 47.03 | 71.03 | 41.66 | 24.24 | 48.91 | 40.37 | 57.74 | 62.50 |
| Change Color | 66.99 | 37.87 | 33.58 | 62.24 | 39.47 | 24.32 | 44.52 | 30.69 | 55.21 | 57.93 |
| Change OCR | 66.54 | 37.28 | 38.93 | 49.46 | 29.69 | 18.39 | 36.79 | 28.65 | 47.79 | 59.39 |
| Change Shape | 68.84 | 46.38 | 41.31 | 67.75 | 43.02 | 28.99 | 33.34 | 35.51 | 53.63 | 60.87 |
| Removal Object | 73.76 | 50.52 | 52.67 | 67.29 | 48.65 | 28.41 | 55.72 | 35.50 | 52.53 | 63.90 |
Table 1: Performance of different IIEMs across various instruction types on DLEBench. We use the Oracle-guided Mode to evaluate each output on two criteria: IF and VC. The final scores are then obtained by averaging the results of these two criteria (see Table 4 for the specific scores of each criterion). The best performance is highlighted in bold , and the second best is underlined .
Figure 6: Impact of object scale on performance. The central scatter plots illustrate the relationship between the target object’s relative scale (log scale) and IF score. The top-left inset displays the trend of average scores as the bbox area increases, along with the Pearson correlation coefficient ( ) and statistical significance ( ). The bottom-right inset presents the overall distribution of scores.
Figure 7: Performance on DLEBench across different instruction types and two different criteria. Top: Closed-source models. Bottom: Open-source models.
| Instrustion Types | Numbers | IF ( ) | VC ( ) |
| Change Material | 84 | 84.43 | 85.65 |
| Change Color | 699 | 85.41 | 92.12 |
| Change OCR | 455 | 83.80 | 89.62 |
| Change Shape | 23 | 81.77 | 95.41 |
| Removal Object | 577 | 93.32 | 94.60 |
| Replace Object | 47 | 87.64 | 94.49 |
Table 2: IAA on DLEBench. We report Krippendorff’s Alpha ( ) for both IF and VC across different instruction types.
Figure 8: Correlation coefficients and MAE between human judgments and different evaluation methods across IF and VC.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
| Label | Description |
| Perfect Consistency | The highest standard. The background environment and all non-target objects remain visually identical to the Original Image. The edit integrates seamlessly without disturbing the surrounding pixels. The scene looks like the exact same photo, just with the specific target modified. |
| Single Anomaly | The general background environment remains consistent, but exactly ONE specific non-target object or detail has been altered, removed, or distorted. For example, everything is perfect except for one cup on the table that changed color, or the addition of one person who did not exist in the original image. Changes that only affect the overall image texture or global visual effects (e.g., lighting filters or grain) are not considered anomalies. |
| Multiple Anomalies | The general background environment remains consistent, but TWO OR MORE distinct non-target objects or details have been altered, removed, or distorted. For example, a painting on the wall changed content AND a chair in the corner disappeared. There are multiple scattered errors in the scene. Similarly, changes that only affect the overall image texture or global visual effects are not counted as anomalies. |
| Scene Collapse | The high-level semantic category of the environment or the artistic style has fundamentally changed. The Edited Image depicts a completely different type of location (e.g., a city turning into a forest) or implies a complete shift in medium (e.g., from photorealistic to an oil painting). Crucial: You must IGNORE changes to specific background objects or the layout of the scene; this label applies strictly when the general setting category or artistic style is destroyed. |
Table 3: Scoring rubrics for the Visual Consistency.
Figure 9: Annotation examples for the two evaluation dimensions.
| Instruction Types | Closed-Source Models | Open-Source Models | |||||||||
| Gemini-3-Pro | GPT-Image-1 | OmniGen2 | Bagel-Think | UniREdit-Bagel | MagicBrush | Qwen-Edit | UniWorld-V1 | UniWorld-V2 | Step1X-Edit | ||
| Instruction Following (IF) | Change Material | 59.92 | 54.07 | 36.11 | 51.19 | 51.59 | 12.70 | 24.80 | 16.06 | 51.98 | 52.78 |
| Change Color | 49.21 | 40.17 | 22.37 | 39.82 | 39.97 | 14.62 | 18.84 | 10.70 | 45.27 | 42.74 | |
| Change OCR | 54.84 | 39.73 | 14.73 | 22.34 | 22.56 | 6.04 | 8.09 | 15.45 | 40.95 | 47.69 | |
| Change Shape | 55.07 | 53.62 | 21.74 | 49.28 | 40.58 | 20.29 | 7.25 | 10.14 | 46.38 | 44.93 | |
| Removal Object | 60.26 | 65.68 | 37.82 | 43.09 | 60.01 | 26.80 | 40.76 | 21.20 | 56.32 | 56.00 | |
Table 4: Performance on DLEBench across different instruction types and two different criteria.
| Label | Description |
| Flawless Execution | The model demonstrates flawless instruction following. It correctly identifies the target region or object, executes the specific modification accurately, and strictly preserves the object’s original identity (including shape, texture, and structure) without introducing any unintended changes. |
| Over Modification | This label applies when the model correctly locates the target and executes the requested edit, but fails to preserve the object’s original details that should have remained unchanged. Except for the specific changes explicitly requested by the instruction, the target object must remain visually identical to the Original Image; therefore, any unrequested alterations to the object’s structure, or fine details—such as changing a T-shirt into a hoodie when only a color shift was requested—are classified as Over Modification. Notably, for object removal instructions, the accidental deletion of non-target objects is NOT penalized under this category. For object replacement instructions, as long as the target is successfully replaced with the requested type of object and the replaced object itself is clearly recognizable as that object, alterations to surrounding objects are NOT classified as Over Modification. However, if the replaced object is not clearly recognizable, it should be classified as Over Modification. Additionally, if the new object is inconsistent with the original image’s style (e.g., lighting, rendering style, or realism level), it should also be classified as Over Modification. |
| Wrong Action | The model successfully locates the target object, but executes a modification that mismatches the requested action category. You must strictly verify against these specific categories: Change Color, Change Material, Change Text, Change Shape, Object Count (Reduction, Addition), and Object Manipulation (Remove, Replace). If the model performs an action from a different category than requested (e.g., executing a ”Remove Object” operation when a ”Change Color” was requested, or ”Replace Object“ when only a ”Change Material” was asked), it must be labeled as Wrong Action. Notably, for object count reduction instructions, if the target objects are not reduced to the exact specified quantity, this is classified as Wrong Action; similarly, for object addition instructions, if the target objects are not increased to the exact specified quantity, this is also classified as Wrong Action. For the object removal instructions, filling in the area where the object was located after removal shall not be considered a Wrong Action, unless it is clearly evident that a replacement operation has been performed. |
| Localization Failure | Localization Failure occurs when the model fails to execute the specified modification on the intended target, leaving the object effectively unchanged. This score encompasses scenarios where the target exhibits minor, unintentional artifacts—such as subtle color shifts or slight geometric distortions—that do not constitute the requested edit, as well as cases where severe blurriness or distortion renders visual verification of the modification impossible. Furthermore, this score also includes mis-localization within the target object, where the model modifies an incorrect sub-component of the target object (e.g., when the target object is a wheel and the instruction is to change the hub of the wheel to red, but the model instead changes the tire to red), because the specific attribute defined by the instruction remains unmodified. |
Table 5: Score rubrics for the Instruction Following.
Figure 10: Instruction Statistics of DLEBench. (a) Distribution of attribute-level modifications, dominated by color and OCR. (b) Distribution of object-level modifications, where removal is the most frequent operation.
Table 6: The prompt used by GPT-4.1 to extract target objects.
Figure 11: The user interface for annotating IF and VC for human judgements.
Figure 13: Visualization results for Change Color.
Figure 14: Visualization results for Change OCR.
Figure 15: Visualization results for Change Material.
Figure 16: Visualization results for Replace Object.
Figure 17: Visualization results for Change Count.
Figure 18: Visualization results for Removal Object.
Figure 19: Visualization results for Change Shape.
| Example 1. Using Oracle-guide Mode to Evaluate an Edited Image Generated by UniWorld-V2 | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image Reference Image | |
| Gemini-3-Pro | <Start Thinking> |
| 1. Verify Modification Occurrence: I compare the Original Image with the Edited Image. The Original Image displays a blue dustpan with a tall black handle standing next to a broom. The Edited Image shows a large red shovel-like object with a red handle. There is a distinct visual difference, so a modification has definitely occurred. 2. Verify Action Alignment: The Editing Instruction is “Changed attribute of blue dustpan to red”. This is a request to change the color of the dustpan. The Edited Image shows an object that is now red. Thus, the model attempted the correct action category (changing color) and achieved the target color. 3. Verify Visual Preservation: I assess whether the model preserved the object’s identity and details that were not supposed to change. * Shape and Structure: The Original Image shows a specific type of dustpan (flat bottom, upright) with a distinct thin black handle. The Edited Image shows a bulky, deep scoop that resembles a snow shovel more than the original dustpan. The shape and silhouette are completely altered. * Unrequested Changes: The handle in the Original Image is black. In the Edited Image, the handle has been changed to red, which was not requested (the instruction targets the “blue dustpan”, implying the blue part should change, or at least the handle material/color should be preserved if not blue). Furthermore, the broom standing next to the dustpan in the Original Image has been completely removed or covered by the new, larger red object. * Comparison with Reference: The Reference Image shows the ideal outcome: the blue plastic pan becomes red, but the black handle remains black, the shape remains identical, and the broom remains visible next to it. * Conclusion: While the model successfully applied the color “red”, it failed to preserve the structural identity of the dustpan (changing it to a shovel-like shape), incorrectly changed the handle color, and removed the adjacent broom. This aligns with the definition of Over Modification, where the structure and silhouette are altered despite only a color change being requested. | |
| </Start Thinking> | |
| <Start Final Answer> | |
Table 7: An example of the feedback generated by Oracle-guide Mode in Instruction Following.
| Example 1. Using Tool-driven Mode to Evaluate an Edited Image Generated by UniWorld-V2 | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image | |
| Gemini-3-Pro | <Start Thinking> The user wants to modify the color of a specific object (“blue dustpan”) to red. 1. Localization: I need to locate the blue dustpan in the Original Image. Scanning the image, I see a blue object leaning against the wall on the right side of the storefront, next to a grey trash bin structure. In the Edited Image, this object appears to be red. 2. Action Execution: The instruction is to change the color to red. Visually, the object has changed from blue to red. 3. Visual Preservation: I need to confirm if the object’s geometry and fine details (handle, pan shape) are preserved, or if the object was regenerated/replaced (Over Modification). Since the object is small in the overall scene, I will use ‘localize_differences’ to get a zoomed-in comparison of the modified region. This will allow me to check for structural consistency. </Start Thinking> <tool_call> { “name”: “localize_differences”, “parameters”: { “comparison_image_1”: “Original Image”, “comparison_image_2”: “Edited Image” } } </tool_call> |
| Observation | [Response] : From provided the first image to the third image show specific difference regions. For each of these images, the layout is a side-by-side comparison: the Left side is the original crop, and the Right side is the edited crop, clearly separated by a vertical red line. Please note that these detections are based on strict pixel-level comparison and might include negligible variations imperceptible to humans. You should disregard insignificant fluctuations and only focus on the crops showing significant, visually obvious changes. Think first, if necessary, choose the appropriate tool to call, then answer. Format strictly as: <Start Thinking>…</Start Thinking>followed by <tool_call>…</tool_call>(if tools are needed), <Start Final Answer>…</Start Final Answer>(if the final evaluation step is reached, output final results). |
| Gemini-3-Pro | <Start Thinking> Based on the analysis of Crop 1 returned by localize_differences, the model demonstrated precise localization by correctly targeting the blue dustpan while leaving the adjacent blue broom untouched, ensuring the edit did not bleed into surrounding similar objects. The primary instruction to change the color to red was executed; however, the edit exhibits a specific failure in visual preservation regarding the object’s details. The original dustpan possesses a distinct black handle, which contrasts with the blue scoop. In the edited version, this handle has been incorrectly recolored to red along with the body. Since the instruction was to change the ”blue dustpan” (using the color to identify the object or the specific part), altering the originally black handle—a non-blue component—disrupts the object’s original dual-tone identity. This conversion into a monochromatic red object represents an unrequested alteration of a structural detail that should have remained unchanged, thereby classifying the result as Over Modification. </Start Thinking> |
Table 8: An example of the feedback generated by Tool-driven Mode in Instruction Following.
| Example 1. Using LMM-as-a-Judge to Evaluate an Edited Image Generated by UniWorld-V2 | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image | |
| Gemini-3-Pro | <Start Thinking> First, I compare the Original Image and the Edited Image to identify the modification. The instruction is to change the “attribute of blue dustpan to red”. In the Original Image, located on the right side of the flower shop steps, leaning against the wall next to a trash can pole, there is a blue dustpan. In the Edited Image, this specific object has been changed to a bright red color. This confirms a modification occurred at the correct location. Next, I verify the alignment of the action with the instruction. The instruction explicitly asks to change the color to red. The model has successfully performed this color swap on the target object. The new color is clearly red, matching the request. Finally, I assess the visual preservation of the target object. I compare the structure and details of the red dustpan in the Edited Image with the blue dustpan in the Original Image. The shape of the scoop and the long handle is perfectly preserved. The object’s orientation, leaning angle, and overall geometry remain identical. The texture appears consistent with the material, and shadows/lighting are handled well. No other parts of the image (like the wall or the adjacent trash can pole) have been altered. The edit is clean and precise. </Start Thinking> |
| <Start Final Answer> Flawless Execution </Start Final Answer> | |
| Human | [Result] : Over Modification |
Table 9: An example of the feedback generated by LMM-as-a-Judge (Gemini-3-Pro) in Instruction Following.
| Example 1. Using LMM-as-a-Judge to Evaluate an Edited Image Generated by UniWorld-V2 | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image | |
| GPT-4.1 | <Start Thinking> First, I compare the Original Image (first) and the Edited Image (second), focusing solely on the blue dustpan as specified by the instruction. In the Original Image, the blue dustpan is located on the right next to the flower shop entrance. In the Edited Image, the dustpan at the same location has been changed to a bright red color. This confirms that a modification occurred at the correct target location. Next, I verify action alignment. The instruction was to “change attribute of blue dustpan to red.” The blue dustpan has indeed become red in the edited image, matching the requested color change action. Finally, for visual preservation, I compare the shape, texture, and general identity of the dustpan in both images. Aside from the color change, the dustpan’s form, handle, size, and presence next to the same wall remain consistent and visually accurate. There are no significant distortions, blur, or unnecessary changes to the dustpan’s design or context. All other characteristics of the dustpan appear accurately preserved except for the intended color modification. Therefore, the edit fulfills all three criteria: correct localization, correct action execution, and visual preservation. </Start Thinking> |
| <Start Final Answer> Flawless Execution </Start Final Answer> | |
| Human | [Result] : Over Modification |
Table 10: An example of the feedback generated by LMM-as-a-Judge (GPT-4.1) in Instruction Following.
| Example 1. Using Oracle-guide Mode to Evaluate an Edited Image Generated by Gemini-3-Pro. | |
| Editing Instruction: Change the color of the bicycle from yellow to black. | |
| Source Image Edited Image Reference Image | |
| Gemini-3-Pro | <Start Thinking> The user wants me to evaluate the visual consistency between the Source and Edited images, ignoring the masked target object (the bicycle specified in the text). The instruction is to change the color of a bicycle from yellow to black. The target bicycle is masked out with a white box near the building on the right. Step 1: Check Global Scene Stability I am comparing the general environment of the Edited Image to the Source Image. - The scene is a lakeside view with swan boats, mountains, and a cloudy sky. - The Edited Image retains the exact same scene composition, lighting, and style (photorealistic). - There is no scene collapse. Step 2: Scan and Count Local Anomalies I need to look for unintended changes in the non-masked areas. - Sky/Clouds: The cloud patterns match perfectly. - Mountains: No changes. - Water/Boats: The swan boats and the water ripples look identical.- Foreground Right (Grass/Path): - Source Image: On the grassy bank near the blue rowboats, there is just grass and some vegetation. - Reference Image: Same as Source, just grass. - Edited Image: There is a distinct blue bicycle lying on the grass next to the blue rowboats. This object is completely new and was not present in the Source or Reference images.- Other areas: The buildings, the people, and the ticket booth (outside the mask) look consistent. The addition of the blue bicycle in the foreground is a significant error. The model likely hallucinated an extra bicycle because the prompt mentioned “bicycle”, even though it was supposed to modify the specific one (which is masked). To confirm there are no other subtle changes (like distortions in the boats or background structures) and to verify the extent of the bicycle addition, I will use the ’localize_differences‘ tool. </Start Thinking> <tool_call> { “name”: “localize_differences”, “parameters”: { “comparison_image_1”: “Original Image”, “comparison_image_2”: “Edited Image” } } { “name”: “detect_object”, “parameters”: { “target_image”: “Original Image”, “detect_object_name”: “yellow bicycle” } } </tool_call> |
| Observation | [Response] : From provided the first image to the third image show specific difference regions. For each of these images, the layout is a side-by-side comparison: the Left side is the original crop, and the Right side is the edited crop, clearly separated by a vertical red line. Please note that these detections are based on strict pixel-level comparison and might include negligible variations imperceptible to humans. You should disregard insignificant fluctuations and only focus on the crops showing significant, visually obvious changes. No yellow bicycle detected in the evaluated ‘Original Image’. Think first, if necessary, choose the appropriate tool to call, then answer. Format strictly as: <Start Thinking>…</Start Thinking>followed by <tool_call>…</tool_call>(if tools are needed), <Start Final Answer>…</Start Final Answer>(if the final evaluation step is reached, output final results). |
| Gemini-3-Pro | <Start Thinking> Based on the visual evidence provided by the tool’s crops and the full image comparison, I can now determine the final evaluation. 1. Global Scene Stability: The overall scene context (lakeside, mountains, cloudy sky, swan boats) and the photorealistic style are well-preserved. There is no scene collapse. 2. Local Anomaly Detection: I have identified multiple distinct errors in the non-masked background areas: * Error 1 (Object Addition): As seen clearly in Crop 1, a blue bicycle has appeared on the grassy bank near the blue rowboats. This object does not exist in the Source or Reference images. The model likely hallucinated this object based on the text prompt b̈icycle,̈ failing to restrict the change to the specific masked target. * Error 2 (Object Distortion): As seen in Crop 2, the people in the background have suffered significant degradation. The man in the red shirt (Crop 2) has become extremely blurry and ghost-like, losing the structural integrity seen in the source. * Error 3 (Text/Detail Degradation): Crops 3 show that the people have become noticeably blurrier and less defined compared to the source. 3. Final Verdict: Since I have identified at least two distinct and significant errors (the hallucinated bicycle on the left and the distorted people on the right), the this falls under the category of Multiple Anomalies. The changes are not limited to a single isolated object but affect different semantic elements (objects and people) in spatially separated areas of the image. </Start Thinking> |
Table 11: An example of the feedback generated by Oracle-guide Mode in Visual Consistency.
| Example 1. Using Tool-driven Mode to Evaluate an Edited Image Generated by Gemini-3-Pro. | |
| Editing Instruction: Change the color of the bicycle from yellow to black. | |
| Source Image Edited Image | |
| Gemini-3-Pro | <Start Thinking> My primary task is to evaluate the visual consistency of the background and non-target objects. 1. Analyze the Request: The user wants to change a “yellow bicycle” to “black”. 2. Initial Scan of Original Image: I am looking for a yellow bicycle. Scanning the right foreground path and the area near the building. I see a blue boat on the grass, a path with people, and a building with a green/yellow sign. I do not see a prominent yellow bicycle. 3. Initial Scan of Edited Image: I see a blue/black bicycle leaning against the blue boat in the foreground on the right. This bicycle was not present in the Original Image. 4. Hypothesis: The model has added a bicycle where there was none, likely hallucinating the object to fulfill the prompt (or I am missing a very small detail). 5. Consistency Check Strategy: I need to determine if there are any other changes in the image besides this added bicycle. If the model only added the bicycle (the target object, effectively), and the rest of the scene (sky, mountains, water, other boats, building, grass texture) remains pixel-perfect or semantically identical, then the visual consistency is high. If the model hallucinated other things or degraded the quality, the score drops. 6. Tool Usage: I will use ‘localize_differences’ to pinpoint exactly which pixels changed. This will confirm if the changes are isolated to the bicycle area or if there are global shifts (e.g., lighting, cloud shifts, compression artifacts). I will also use ‘detect_object’ on the Original Image with the text ÿellow bicycleẗo confirm its absence, which clarifies if the edit is an insertion or a modification. Let’s gather the data. </Start Thinking> <tool_call> { “name”: “localize_differences”, “parameters”: { “comparison_image_1”: “Original Image”, “comparison_image_2”: “Edited Image” } } { “name”: “detect_object”, “parameters”: { “target_image”: “Original Image”, “detect_object_name”: “yellow bicycle” } } </tool_call> |
| Observation | [Response] : From provided the first image to the third image show specific difference regions. For each of these images, the layout is a side-by-side comparison: the Left side is the original crop, and the Right side is the edited crop, clearly separated by a vertical red line. Please note that these detections are based on strict pixel-level comparison and might include negligible variations imperceptible to humans. You should disregard insignificant fluctuations and only focus on the crops showing significant, visually obvious changes. No yellow bicycle detected in the evaluated ‘Original Image’. Think first, if necessary, choose the appropriate tool to call, then answer. Format strictly as: <Start Thinking>…</Start Thinking>followed by <tool_call>…</tool_call>(if tools are needed), <Start Final Answer>…</Start Final Answer>(if the final evaluation step is reached, output final results). |
| Gemini-3-Pro | <Start Thinking> 1. Analyze Object Detection: The ‘detect_object’ tool returned ”No yellow bicycle detected” in the Original Image. This confirms that the editing instruction ”Changed the color of the bicycle from yellow to black” was based on a non-existent object (or the user mistook the blue boat or another element for a bicycle). Consequently, the model likely generated (hallucinated/inserted) a bicycle to fulfill the request. 2. Analyze Difference Localization (‘localize_differences’): The tool returned 3 specific crop regions showing changes. I need to categorize these changes into ”Target Edit” vs. ”Non-target Anomalies”. * Crop 1 (Boat area): This crop shows the insertion of a black bicycle leaning against the blue boat. This is a Anomaly (Anomaly #1). * Crop 2 (Man in red shirt): The Left side (Original) shows a man in a red shirt and dark pants standing clearly. The Right side (Edited) shows this figure is significantly blurred and distorted, losing structural integrity. This is a major alteration to a non-target object (Anomaly #2). * Crop 3 (Person): The person details are sharp in the Original but appear blurry in the Edited version. This represents a degradation of background details (Anomaly #3). 3. Evaluate Consistency: * Global Scene Stability: The general scene (park, boat, building) is preserved. No scene collapse. * Local Anomaly Detection: I found specific, distinct errors in non-target areas. The man in the red shirt was distorted/blurred. A black bicycle has been added to the scene. The person details were degraded. 4. Conclusion: The model failed to preserve image integrity by making three key errors. Since multiple non-target elements were incorrectly altered, this case is classified as Multiple Anomalies. This qualifies as Multiple Anomalies. </Start Thinking> |
Table 12: An example of the feedback generated by Tool-driven Mode in Visual Consistency.
| Example 1. Using LMM-as-a-Judge (Gemini-3-Pro) to Evaluate an Edited Image Generated by Gemini-3-Pro | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image | |
| Gemini-3-Pro | <Start Thinking> 1. Check Global Scene Stability: - I compare the Original Image and the Edited Image to assess the high-level semantic category and artistic style. - Both images depict a scenic lakeside view with swan boats, a dock, mountains in the background, and a cloudy sky. - The photorealistic style is perfectly preserved. The location has not shifted (e.g., it is still the same lake). - Conclusion: Global Scene Stability is maintained. No Scene Collapse. 2. Scan and Count Local Anomalies: - I meticulously scan the image for changes in the non-target areas. The instruction is “Changed the color of the bicycle from yellow to black.̈ - Target Identification: I look for the target (bicycle) in the Original Image. Scanning the foreground, dock, and booth area, there is no visible yellow bicycle in the Original Image. There is a blue boat in the foreground, but no bicycle. - Comparison: In the Edited Image, a blue bicycle has been inserted into the foreground, leaning against the blue boat on the grass. - Anomaly Analysis: - Since the bicycle did not exist in the Original Image at that location, the model has added a new object rather than modifying an existing one. - According to the evaluation rules, the “addition of one person [or object] who did not exist in the original image” is classified as an anomaly in the scene content. The model hallucinated an object to fulfill the prompt, thereby altering the background (the space previously occupied only by the boat and grass). - I check the rest of the image (sky, mountains, water, other boats, people in the distance). All other pixels appear identical to the Original Image. - Count: I found exactly 1 distinct error: the unwanted addition of the bicycle in the foreground. 3. Determine Final Verdict: - Distinct errors found: 1 (The added bicycle). - Based on the criteria, exactly one distinct error results in “Single Anomaly”. </Start Thinking> |
| <Start Final Answer> Single Anomaly </Start Final Answer> | |
| Human | [Result] : Multiple Anomalies |
Table 13: An example of the feedback generated by LMM-as-a-Judge (Gemini-3-Pro) in Visual Consistency.
| Example 1. Using LMM-as-a-Judge (GPT-4.1) to Evaluate an Edited Image Generated by Gemini-3-Pro | |
| Editing Instruction: Change attribute of blue dustpan to red. | |
| Source Image Edited Image | |
| GPT-4.1 | <Start Thinking> Step 1: Check Global Scene Stability. - The overall scene context in all images is a lakeside environment with greenery, boats, mountains, and scattered buildings. The sky remains blue with white clouds, and the artistic style is photorealistic in all cases. The Edited Image preserves the high-level environment type completely—remaining a lakeside with the same mountains and buildings. There is no evidence of the scene shifting to a different setting or medium. - No Scene Collapse. Step 2: Scan and Count Local Anomalies. - The editing instruction targets a non-background element: the color of the bicycle. I must ignore changes to this, as instructed. - I compare the immediate lakeside elements: boats, vegetation, dock, buildings, signage, people, water, sky, and the arrangement of the scene outside the bicycle. - The dock structure, boats (rowboats and swans), water, vegetation, hillside foliage, buildings, signage, and all visible non-target objects in the foreground and background are identical between the Original Image and the Edited Image. - No objects are missing, added, or visually altered aside from the specified change to the bicycle. - No color, structure, or placement differences are visible in the environment or non-target objects. Step 3: Determine Final Verdict. - After a step-by-step scan, I detect 0 errors or anomalies outside of the bicycle. </Start Thinking> |
| <Start Final Answer> Perfect Consistency </Start Final Answer> | |
| Human | [Result] : Multiple Anomalies |
Table 14: An example of the feedback generated by LMM-as-a-Judge (GPT-4.1) in Visual Consistency.
Table 15: Prompt used to transform visual reasoning tasks into instruction-based image editing tasks.
Table 16: The system prompt used by Tool-driven Mode.
Table 17: The prompt used by Tool-driven Mode to evaluate Instruction Following. (Part One)
Table 18: The prompt used by Tool-driven Mode to evaluate Instruction Following. (Part Two)
Table 19: The prompt used by Tool-driven Mode to evaluate Instruction Following. (Part Three)
Table 20: The prompt used by Tool-driven Mode to evaluate Visual Consistency. (Part One)
Table 21: The prompt used by Tool-driven Mode to evaluate Visual Consistency. (Part Two)
Table 22: The prompt used by Oracle-guide Mode to evaluate Instruction Following. (Part One)
Table 23: The prompt used by Oracle-guide Mode to evaluate Instruction Following. (Part Two)
Table 24: The prompt used by Oracle-guide Mode to evaluate Instruction Following. (Part Three)
Table 25: The prompt used by Oracle-guide Mode to evaluate Visual Consistency. (Part One)
Table 26: The prompt used by Oracle-guide Mode to evaluate Visual Consistency. (Part Two)
Table 27: The prompt used by LMM-as-a-Judge to evaluate Instruction Following. (Part One)
Table 28: The prompt used by LMM-as-a-Judge to evaluate Instruction Following. (Part Two)
Table 29: The prompt used by LMM-as-a-Judge to evaluate Instruction Following. (Part Three)
Table 30: The prompt used by LMM-as-a-Judge to evaluate Visual Consistency. (Part One)
Table 31: The prompt used by LMM-as-a-Judge to evaluate Visual Consistency. (Part Two)
More in this topic
Image Editing Evaluation
Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.
See all 92 papers