When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images
Authors: Ruoqi Hu, Chulin Zhao, Jiashuo Chang, Ramon Ruiz-Dolz, Hanhe Lin
Organizations: Dundee International Institute of Central South University, Central South University, China · Faculty of Science, Engineering and Business, University of Dundee, United Kingdom
*Chulin Zhao and Ruoqi Hu contributed equally to this work. State-of-the-art text-to-image (T2I) models exhibit pronounced and systematic defects when prompts involve intricate compositional factors such as multiple entities and multiple attributes. In this paper, we investigate how humans identify such defects. Specifically, we manually select 651 reference images from the four categories of people, hand, object, and scene that exhibit complex compositional characteristics, from which prompts emphasizing compositional factors are derived by manually editing ChatGPT-generated prompts. We then feed the prompts into three selected T2I models to generate AI images and conduct a comprehensive subjective study to identify their defects. For each image, 29 participants provide multi-label assessments specifying defect types and locations. The study yields the compositional AI-generated image defect (CO-AID) dataset, including reference images, prompts, AI-generated images, and information on defect locations and types. Experimental results show that training a deep model on CO-AID can both predict defects in AI-generated images and optimize AI image generation, demonstrating its usability and effectiveness. The database and supplementary materials are available at: https://github.com/Future-IQA/CO-AID .
Figures & tables
Fig. 1: Example AI-generated images in CO-AID dataset. (a) Image with global defect. 19 participants report that it shows a CGI-like appearance. (b) Image with local defects marked by yellow dots. The three most densely clicked regions and their corresponding defect reasons are highlighted.
Fig. 2: Workflow for collecting the AI-generated compositional images. A total of 651 reference images that exhibit compositional characteristics across four categories of people, hand, object, and scene were manually selected and downloaded from Pexels and Unsplash, which were then used to generate compositional prompts by manually editing ChatGPT-generated prompts. The prompts were then fed into three T2I models Midjourney, Imagen, and Flux to produce AI-generated images for further subjective study.
Fig. 3: Overview of our proposed human perceptual defect identification study. The study procedure consists of three steps: Introduction, Training and Experiment. Before the main study, a small pilot study was conducted to improve experimental instruction, complete and polish optional defect reasons, and optimize the graphical user interface. In the experiment step, participants sequentially view AI-generated compositional images and identify their defects. For those who show noticeable defects, participants provided feedback at the global and/or local levels.
Fig. 4: Statistics of the experimental results.
Fig. 5: Visualization of the proof-of-concept experimental results. From left to right, the figure shows the input AI-generated images, human-annotated ground truth (GT) defect maps, defect detection results (bounding boxes) by GPT-Image-1, defect detection results by fine-tuned TranSalNet model [ 15 ] , and the corresponding defect-guided image repair results. Zoom in for more details.
Generative AI can now produce highly realistic images, yet current models still exhibit subtle but critical defects that undermine their reliability. While existing AI-generated image (AGI) evaluation benchmarks have made notable progress, comprehensive AGI defect diagnosis remains underexplored. To bridge this gap, we introduce AGIDefect-4K, a richly annotated dataset of 4,000 images from 15 state-of-the-art generative models spanning both open-source and closed-source systems. AGIDefect-4K features hierarchical defect annotations: (1) detection labels identifying whether defects exist, (2) pixel-level segmentation masks localizing defective regions, and (3) detailed textual explanations characterizing defect types and their perceptual impact. Each image is further annotated with an overall quality score. Building on this, we present AGIDA (AGI Defect Assistant), a baseline framework leveraging Multimodal Large Language Models (MLLMs) for joint defect detection, localization, explanation, and quality prediction. Comprehensive benchmarking on AGIDefect-4K reveals that AGI defect understanding remains challenging, underscoring the value of this dataset. The dataset is publicly available at https://github.com/sxfly99/AGIDefect-4K.
Text-to-Image (T2I) diffusion models have made rapid progress on semantic alignment (generating what is described in the prompt), yet users still lack reliable control over aesthetic composition (how visual elements are put together). Prior work often treats aesthetics as a single, preference-driven notion (e.g., "high quality", "detailed", "breathtaking"), which does not map cleanly to compositional intent. We propose Aesthetic Alignment: aligning generated images to explicit, user-specified compositional constraints. We operationalize these constraints using the Principles of Art (PoA)-e.g., Balance, Rhythm, and Emphasis-commonly used in art education to describe composition. To support this task, we introduce CompArt, a dataset of 80,032 WikiArt images augmented with captions and PoA analyses produced by a multimodal LLM under structured prompting. We further propose ArtDapter, a lightweight and disentangled adapter that enables steering a pretrained T2I model along 10 PoA dimensions while retaining the base model's semantic capability. Experiments on CompArt show improved adherence to PoA controls over strong baselines under a dual evaluation protocol.
Recent text-to-image (T2I) generators can synthesize realistic images, but still struggle with compositional prompts involving multiple objects, counts, attributes, and relations. We introduce EPIC (Efficient Predicate-Guided Inference-Time Control), a training-free inference-time refinement framework for compositional T2I generation. EPIC casts refinement as predicate-guided search: it parses the original prompt once into a fixed visual program of object variables and typed predicates, covering checkable conditions such as object presence, counts, attributes, and relations. Each generated or edited image is verified against this program using visual evidence extracted from that image. An image is judged to satisfy the prompt only when all predicates are satisfied; otherwise, failed predicates decide the next step, routing local failures to targeted editing and global failures to resampling while the fixed visual program remains unchanged. On GenEval2, EPIC improves prompt-level accuracy from 34.16% for single-pass generation with the base generator to 71.46%. Under the same generator/editor setting and maximum image-model execution budget, EPIC outperforms the strongest prior refinement baseline by 19.23 points while reducing realized cost by 31% in image-model executions, 72% in MLLM calls, and 81% in MLLM tokens per prompt.
Sunung Mun, Sunghyun Cho, Jungseul Ok
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science & Engineering, POSTECH