AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation

Xiangfei Sheng; Weidong Zou; Tianjiao Gu; Zhichao Yang; Pengfei Chen; Leida Li

cs.CVAug 21, 2026

AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation

Authors: Xiangfei Sheng, Weidong Zou, Tianjiao Gu, Zhichao Yang, Pengfei Chen, Leida Li

Organizations: Xidian University Xi’an, Shaanxi, China

Abstract

Generative AI can now produce highly realistic images, yet current models still exhibit subtle but critical defects that undermine their reliability. While existing AI-generated image (AGI) evaluation benchmarks have made notable progress, comprehensive AGI defect diagnosis remains underexplored. To bridge this gap, we introduce AGIDefect-4K, a richly annotated dataset of 4,000 images from 15 state-of-the-art generative models spanning both open-source and closed-source systems. AGIDefect-4K features hierarchical defect annotations: (1) detection labels identifying whether defects exist, (2) pixel-level segmentation masks localizing defective regions, and (3) detailed textual explanations characterizing defect types and their perceptual impact. Each image is further annotated with an overall quality score. Building on this, we present AGIDA (AGI Defect Assistant), a baseline framework leveraging Multimodal Large Language Models (MLLMs) for joint defect detection, localization, explanation, and quality prediction. Comprehensive benchmarking on AGIDefect-4K reveals that AGI defect understanding remains challenging, underscoring the value of this dataset. The dataset is publicly available at https://github.com/sxfly99/AGIDefect-4K.

Figures & tables

Figure 1. Overview of our approach. Left: Illustration of our motivation. Even state-of-the-art generative models exhibit notable defect rates. Middle: AGIDefect-4K dataset comprising images from 15 diverse generative models. Right: Our hierarchical annotation framework: detection (binary classification), localization (segmentation masks), and explanation (detailed textual descriptions), enabling comprehensive defect detection through multimodal large language models. Overview figure showing motivation, dataset, and annotation framework.

Figure 2. Overview of AGIDefect-4K dataset construction pipeline. The three-stage construction pipeline of AGIDefect-4K. First, image-prompt pairs are sampled from existing AI-generated image datasets, while additional images are generated from filtered DiffusionDB prompts using proprietary models. Second, multiple annotators independently classify each image as defective or flawless, and disputed cases undergo joint review. Third, defective images receive independently annotated segmentation masks, textual explanations, and quality scores. An expert panel then discards invalid annotations, merges valid masks and descriptions, and averages the quality scores.

Dataset	#Img	Pub	Det.	Loc.	Exp.	QS
AGIQA-3K ( Li et al., 2024b )	2,982	TCSVT’24	✗	✗	✗	✓
RichHF-18K ( Liang et al., 2024 )	18,000	CVPR’24	✓	H	✗	✓
A-Bench ( Zhang et al., 2025b )	2,864	ICLR’25	✗	✗	✓	✗
Q-Eval-100K ( Zhang et al., 2025a )	100,000	CVPR’25	✗	✗	✗	✓
AIGI-VC ( Tian et al., 2025 )	2,500	AAAI’25	✗	✗	✓	✓
EvalMuse-40K ( Han et al., 2026 )	40,000	AAAI’26	✓	✗	✗	✗

Table 1. Comparison with existing AGI evaluation datasets. Det. : defect detection. Loc. : defect localization (H: heatmap, M: mask). Exp. : textual explanation. QS : quality score.

Panel of Figure 3 — Figure 3. Statistical analysis of AGIDefect-4K. (a) Word cloud of textual descriptions. (b) Distribution of defect mask sizes as a percentage of total image area. (c) Token length distribution of defect descriptions. (d) Quality score distributions for defective and flawless images. Four statistical summaries of AGIDefect-4K. The word cloud highlights frequent concepts in the defect explanations, including overall quality, noticeable, unnatural, finger, hand, eye, and structural anomaly. The defect-mask area distribution is strongly right-skewed, showing that most defects occupy a small proportion of an image. Description lengths concentrate around 50 to 100 tokens with a long tail approaching 200 tokens. Quality scores for defective images are lower and more broadly distributed than those for flawless images, whose scores concentrate around 3.3.

Figure 4. Quality score distribution across defect categories. Box plots comparing image quality scores across five defect categories. The mean scores increase from face defects at 2.48 and body defects at 2.55 to animal defects at 2.75, logical defects at 3.10, and object defects at 3.29. Face and body defects therefore have the strongest negative effect on perceived image quality, while object and logical defects have relatively smaller effects.

Figure 5. Overview of AGIDA architecture. Architecture of AGIDA. An input image and text prompt are processed by a frozen multimodal large language model equipped with trainable LoRA modules. The SEG token guides a trainable SAM decoder, together with image features from a frozen SAM encoder, to produce a defect localization mask. The DET token is passed through a trainable detection head to predict whether the image is defective. The QUA token queries the defect representation through cross-attention and is mapped to a quality score. The multimodal language model also generates a textual explanation of the detected defect. Snowflake icons denote frozen components, and flame icons denote trainable components.

Closed-source Models
Model	Params	Detection	Explanation			Rank
Model	Params	AUC	Comp.	Prec.	Mean	Rank
Grok-4 ( xAI, 2025b )	NA	0.29	0.89	0.21	0.55	10/7
GPT-4o ( OpenAI, 2024 )	NA	0.31	0.54	0.38	0.46	6/8
Claude-Sonnet-4.6 ( Anthropic, 2026 )	NA	0.30	0.69	0.50	0.59	8/4
Qwen-VL-MAX ( Alibaba Cloud, 2025 )	NA	0.38	0.51	0.35	0.43	3/9

Table 2. Comparison of detection and explanation performance on AGIDefect-4K. Best are in bold , second best are underlined .

Figure 6. Case studies. Best viewed when zoomed in. Three qualitative examples comparing AGIDA with general-purpose multimodal models. In the first example, AGIDA correctly identifies a flawless portrait. In the second, the generated man has fused fingers: Claude-Sonnet-4.6 incorrectly declares the image flawless, and Qwen3.5-Plus focuses on the computer mouse, whereas AGIDA identifies the fused fingers and produces a mask aligned with the ground truth. In the third example, a bird has only one malformed leg: GLM-4.6V describes unrelated defects and Grok-4 misses the defect, while AGIDA correctly explains and localizes the malformed leg.

Table 9

Image Generation Evaluation

Image generation evaluation measures the quality, fidelity, diversity, and instruction-following ability of AI-generated images.

See all 113 papers

When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images
Aug 26, 2026Ruoqi Hu, Chulin Zhao, Jiashuo Chang +2T2I Generation Evaluation Compositional T2I Generation
Dundee International Institute of Central South University, Central South University, China · Faculty of Science, Engineering and Business, University of Dundee, United Kingdom
Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback
Jun 4, 2026Huaisong Zhang, Hao Yu, Yuxuan Zhang +7Diffusion Model Alignment T2I Generation Evaluation
Tsinghua University · University of British Columbia · Vector Institute +2
SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Data
Aug 13, 2026Yicheng Bao, Xiahui Guo, Xuhong Wang +1Adversarial Training Adversarial Examples
1East China Normal University · 2Shanghai Artificial Intelligence Laboratory

Aug 26, 2026J/K move · Enter open · S save

When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images

Ruoqi Hu, Chulin Zhao, Jiashuo Chang, Ramon Ruiz-Dolz+1

Dundee International Institute of Central South University, Central South University, China · Faculty of Science, Engineering and Business, University of Dundee, United Kingdom

T2I Generation Evaluation Compositional T2I Generation

Open paper AskPDF arXiv

*Chulin Zhao and Ruoqi Hu contributed equally to this work. State-of-the-art text-to-image (T2I) models exhibit pronounced and systematic defects when prompts involve intricate compositional factors such as multiple entities and multiple attributes. In this paper, we investigate how humans identify such defects. Specifically, we manually select 651 reference images from the four categories of people, hand, object, and scene that exhibit complex compositional characteristics, from which prompts emphasizing compositional factors are derived by manually editing ChatGPT-generated prompts. We then feed the prompts into three selected T2I models to generate AI images and conduct a comprehensive subjective study to identify their defects. For each image, 29 participants provide multi-label assessments specifying defect types and locations. The study yields the compositional AI-generated image defect (CO-AID) dataset, including reference images, prompts, AI-generated images, and information on defect locations and types. Experimental results show that training a deep model on CO-AID can both predict defects in AI-generated images and optimize AI image generation, demonstrating its usability and effectiveness. The database and supplementary materials are available at: https://github.com/Future-IQA/CO-AID .

Abstract

Image Generation Evaluation

Explore similar work

When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images

Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Data