cs.CVFeb 24, 2026
SaveSee and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis
Organizations: KAIST · Seoul National University · KRAFTON
Abstract
Despite recent advances in diffusion models, AI generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, which makes artifact mitigation a highly crucial area of study. Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale, underscoring the need for an automated approach to reliably acquire artifact-annotated datasets. In this paper, we propose ArtiAgent, which efficiently creates pairs of real and artifact-injected images. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images, a synthesis agent that introduces artifacts via artifact injection tools through novel patch-wise embedding manipulation within a diffusion transformer, and a curation agent that filters the synthesized artifacts and generates both local and global explanations for each instance. Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications. Code is available at link.
Figures & tables
Figure 1: Overview of our challenges and approach. The red boxes indicate the regions with visual artifacts. (a) Examples of structural visual artifacts in state-of-the-art diffusion models and the inability of VLMs to recognize or explain them. (b) Overview of ArtiAgent, a novel agentic framework that synthesizes artifacts for arbitrary visual contexts without human intervention. (c) Example of VLM-based artifact comprehension via detection, explanation, and localization. (d) Application to reward-guided text-to-image generation. (e) Application to image correction, where artifact-aware VLM-guided inpainting removes the flawed regions.
Figure 3: ArtiAgent consists of three coordinated agents: (1) the perception agent detects entities and subentities using Grounded-SAM; (2) the synthesis agent injects artifacts through patch mapping tool and the inversion-injection paradigm; and (3) the curation agent filters low-quality results and generates localized and global textual explanations.
Figure 4: Visualization of each target-reference patch mapping and its resulting artifact-injected image.
Figure 5: Inversion-injection module. In this example, the right arm from the reference patches is added to target patches below it.
| Benchmark | Source Models | Sample | Bin. | Loc. | Exp. | |
| Oldest | Newest | |||||
| RichHF | SD2.1 | Dreamlike | 955 | ✓ | ||
| Photoreal | ||||||
| LOKI | pix2pix | FLUX | 229 | ✓ | ✓ | |
| SynthScars | Midjourney | DALL E3 | 1K | ✓ | ✓ | |
| ArtiBench | SD3.5 | Nano-Banana | 1K | ✓ | ✓ | ✓ |
Table 2: Comparison of artifact benchmark datasets. If a benchmark reused another dataset as a source, we describe the generative models used in that source. The table with the full citations is provided in Appendix B.1 .
| Methods | (a) Binary detection | (b) Localization | (c) Explanation | |||||||||||||
| ArtiBench | RichHF | LOKI | SynthScars | ArtiBench | LOKI | SynthScars | ArtiBench | |||||||||
| Acc | F1 | mIoU | F1 | mIoU | F1 | mIoU | F1 | mIoU | F1 | ROUGE | CSS | ROUGE | CSS | ROUGE | CSS | |
| PAL | - | - | 0.079 | 0.028 | 0.021 | 0.037 | 0.035 | 0.053 | 0.040 | 0.066 | - | - | - | - | - | - |
| DiffDoctor | - | - | 0.077 | 0.139 | 0.175 | 0.274 | 0.083 | 0.136 | 0.081 | 0.137 | - | - | - | - | - | - |
| LEGION | - | - | 0.067 | 0.112 | 0.100 | 0.158 | 0.106 † | 0.152 † | 0.062 | 0.099 | 0.133 | 0.314 | 0.247 † | 0.589 † | 0.143 | 0.332 |
| GPT-4o | 0.619 | 0.601 | 0.086 | 0.040 | 0.037 | 0.056 | 0.032 | 0.052 | 0.049 | 0.084 | 0.107 | 0.266 | 0.125 | 0.404 | 0.143 | 0.433 |
Table 3: Artifact understanding performance across (a) binary detection, (b) localization, and (c) explanation. The benchmarks are listed in the order in which they were published.
Figure 6: Scaling effect of data generated by ArtiAgent with Qwen2.5-VL-7B. We average the results of all the benchmarks.
| Methods | (a) Binary detection | (b) Localization | (c) Explanation | |||
| Acc | F1 | mIoU | F1 | ROUGE | CSS | |
| SynthScars | 0.555 | 0.548 | 0.094 | 0.147 | 0.156 | 0.521 |
| ArtiAgent | 0.555 | 0.550 | 0.074 | 0.121 | 0.222 | 0.606 |
Table 4: ArtiBench result of the VLM trained on a 1K dataset.
Figure 7: Reward-guided generation. ArtiAgent can train a reward model that guides diffusion to generate artifact-free images.
Figure 8: Image correction. The ArtiAgent-trained VLM can effectively guide image inpainting models to correct artifact regions.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Attention Visualization. We compare the attention maps of InternVL3.5-8B before (base) and after (fine-tuned) training on ArtiAgent. The fine-tuned model reliably focuses on genuine artifact features across both synthesized images and real-world images.
| Steps | Acc | F1 |
| 5/25 | 0.513 | 0.377 |
| 10/25 | 0.583 | 0.565 |
| 15/25 | 0.586 | 0.570 |
| 20/25 | 0.540 | 0.477 |
Table 5: Ablation study on injection steps. Performance of Qwen2.5-VL-7B on ArtiBench binary detection when trained with 1K ArtiAgent samples generated using varying numbers of injection steps out of 25 total steps.
Figure 10: Visualizations of artifacts injected with ArtiAgent. The first and third rows show the original images, whereas the second and fourth rows show the output images with artifacts injected.
Figure 11: An instance of ArtiAgent with the annotation.
Figure 12: Guidance and rules for generating entity-subentity vocabulary sets.
Figure 13: Full prompt for generating entity-subentity vocabulary sets.
Figure 14: Type descriptions and instructions by artifact type.
Figure 15: Full prompt and example for data filtering.
Figure 16: Full prompt and example for regional explanation generation.
Figure 17: Full prompt and example for global explanation generation.
Figure 18: Hyperparameter study on PE injection and value injection steps (1). The image in the red box shows our selected configuration.
| Benchmark | Generative Sources | Sample | Bin. | Loc. | Exp. |
| RichHF [ 27 ] | SD2.1 [ 42 ] , SDXL [ 38 ] , Dreamlike Photoreal [ 10 ] | 955 | ✓ | ||
| LOKI [ 51 ] | FLUX [ 5 ] , SD1.4–2.1 [ 42 ] , Midjourney [ 30 ] , StyleGAN [ 22 ] , pix2pix [ 18 ] , CUT [ 36 ] | 229 | ✓ | ✓ | |
| SynthScars [ 20 ] | RichHF [ 27 ] , Chameleon [ 50 ] , Midjourney [ 30 ] , DALL·E 3 [ 32 ] , SD1.x [ 42 ] | 1K | ✓ | ✓ | |
| ArtiBench | SD3.5 [ 11 ] , FLUX [ 5 ] , Qwen-Image [ 49 ] , Nano-Banana [ 13 ] | 1K | ✓ | ✓ | ✓ |
Table 6: Comparison of artifact benchmark datasets, their generative sources, and evaluation tasks. Highlighted entries denote dataset sources that were reused by subsequent benchmarks.
Figure 19: Artifact examples of the four benchmarks, listed in the order of publishment.
Figure 20: Evaluation prompts for VLMs.
| Task | Template |
| Bin. | Q: Does this image contain any visual artifacts? |
| A: No. | |
| Loc. 1 | Q: Locate the ⟨entity⟩’s ⟨subentity⟩. |
| A: ⟨bbox⟩ | |
| Loc. 2 | Q: Is there a ⟨entity⟩’s ⟨subentity⟩ in ⟨bbox⟩? |
| A: Yes / No |
Table 7: VQA templates for clean images.
| Task | Template |
| Bin. | Q: Does this image contain any visual artifacts? |
| A: Yes. | |
| Loc. 1 | Q: Provide bounding boxes for all artifact regions. |
| A: [⟨bbox⟩] | |
| Loc. 2 | Q: Explain why region ⟨bbox⟩ is an artifact. |
| A: ⟨label⟩ |
Table 8: VQA templates for artifact-injected images.
| Prompt | Instruction Text |
| Localization | “Provide the bounding box for the artifact region. Format as [x_min, y_min, x_max, y_max]. Only output the bounding box.” |
| Captioning | “Describe what the image would look like if it had no artifacts. Provide a short caption of the clean scene.” |
| Verifying | “Is there an artifact within ⟨B⟩? Answer Yes or No.” |
Table 9: Prompts used by the VLM in the artifact-correction loop.
More in this topic
Synthetic Data Augmentation
Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.
See all 453 papers