When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images
Authors: Ruoqi Hu, Chulin Zhao, Jiashuo Chang, Ramon Ruiz-Dolz, Hanhe Lin
Organizations: Dundee International Institute of Central South University, Central South University, China · Faculty of Science, Engineering and Business, University of Dundee, United Kingdom
*Chulin Zhao and Ruoqi Hu contributed equally to this work. State-of-the-art text-to-image (T2I) models exhibit pronounced and systematic defects when prompts involve intricate compositional factors such as multiple entities and multiple attributes. In this paper, we investigate how humans identify such defects. Specifically, we manually select 651 reference images from the four categories of people, hand, object, and scene that exhibit complex compositional characteristics, from which prompts emphasizing compositional factors are derived by manually editing ChatGPT-generated prompts. We then feed the prompts into three selected T2I models to generate AI images and conduct a comprehensive subjective study to identify their defects. For each image, 29 participants provide multi-label assessments specifying defect types and locations. The study yields the compositional AI-generated image defect (CO-AID) dataset, including reference images, prompts, AI-generated images, and information on defect locations and types. Experimental results show that training a deep model on CO-AID can both predict defects in AI-generated images and optimize AI image generation, demonstrating its usability and effectiveness. The database and supplementary materials are available at: https://github.com/Future-IQA/CO-AID .
Figures & tables
Fig. 1: Example AI-generated images in CO-AID dataset. (a) Image with global defect. 19 participants report that it shows a CGI-like appearance. (b) Image with local defects marked by yellow dots. The three most densely clicked regions and their corresponding defect reasons are highlighted.
Fig. 2: Workflow for collecting the AI-generated compositional images. A total of 651 reference images that exhibit compositional characteristics across four categories of people, hand, object, and scene were manually selected and downloaded from Pexels and Unsplash, which were then used to generate compositional prompts by manually editing ChatGPT-generated prompts. The prompts were then fed into three T2I models Midjourney, Imagen, and Flux to produce AI-generated images for further subjective study.
Fig. 3: Overview of our proposed human perceptual defect identification study. The study procedure consists of three steps: Introduction, Training and Experiment. Before the main study, a small pilot study was conducted to improve experimental instruction, complete and polish optional defect reasons, and optimize the graphical user interface. In the experiment step, participants sequentially view AI-generated compositional images and identify their defects. For those who show noticeable defects, participants provided feedback at the global and/or local levels.
Fig. 4: Statistics of the experimental results.
Fig. 5: Visualization of the proof-of-concept experimental results. From left to right, the figure shows the input AI-generated images, human-annotated ground truth (GT) defect maps, defect detection results (bounding boxes) by GPT-Image-1, defect detection results by fine-tuned TranSalNet model [ 15 ] , and the corresponding defect-guided image repair results. Zoom in for more details.