AI-Generated Images as Data Source: The Dawn of Synthetic Era
Organizations: the Nanyang Technological University, Singapore. · the Max Planck Institute for Informatics, Germany.
Abstract
The advancement of visual intelligence is intrinsically tethered to the availability of large-scale data. In parallel, generative Artificial Intelligence (AI) has unlocked the potential to create synthetic images that closely resemble real-world photographs. This prompts a compelling inquiry: how much visual intelligence could benefit from the advance of generative AI? This paper explores the innovative concept of harnessing these AI-generated images as new data sources, reshaping traditional modeling paradigms in visual intelligence. In contrast to real data, AI-generated data exhibit remarkable advantages, including unmatched abundance and scalability, the rapid generation of vast datasets, and the effortless simulation of edge cases. Built on the success of generative AI models, we examine the potential of their generated data in a range of applications, from training machine learning models to simulating scenarios for computational modeling, testing, and validation. We probe the technological foundations that support this groundbreaking use of generative AI, engaging in an in-depth discussion on the ethical, legal, and practical considerations that accompany this transformative paradigm shift. Through an exhaustive survey of current technologies and applications, this paper presents a comprehensive view of the synthetic era in visual intelligence. A project associated with this paper can be found at https://github.com/mwxely/AIGS .
Figures & tables
| Dataset | Application | Generator Type | Public Availability | Size ( real : fake ) | Year |
|---|---|---|---|---|---|
| DiffusionDB [ 252 ] | Prompt Inversion | DM | 14 million | 2022 | |
| JourneyDB [ 259 ] | Prompt Inversion, Style Retrieval, Image Captioning, Visual Question Answering | DM | 4,692,751 | 2023 | |
| Pick-a-Pic [ 260 ] | Human Preference Prediction | DM | 583,747 | 2023 | |
| ImageReward [ 261 ] | Human Preference Prediction | DM | 354,608 | 2023 | |
| HPD v2 [ 253 ] | Human Preference Prediction | DM | 867,520 | 2023 | |
| GRE [ 254 ] | Regional Editing Detection | GAN & DM | 228,650 | 2023 |
| Metric | Evaluation Type |
|---|---|
| R-Precision [ 273 ] | Text-image Alignment |
| Captioning Metrics [ 274 ] | Text-image Alignment |
| SOA [ 275 ] | Text-image Alignment |
| CLIP Score [ 145 ] | Text-image Alignment |
| IS [ 276 ] | Perceptual Quality |
| FID [ 277 ] | Perceptual Quality |
| \begin{array}[]{l|c|c|c|c|c|c}\hline\cr\hline\cr\text{ {Backbone} }&\text{ {Input Size} }&\text{ {Params (M)} }&\text{ {Real Only} }&\text{ {Generated Only} }&\text{ {Real + Generated} }&\text{ {Performance} }\Delta\\ \hline\cr\hline\cr\lx@intercol\hfil\text{ ConvNets }\hfil\lx@intercol\\ \hline\cr\text{ ResNet-50 }&224\times 224&36&76.39&69.24&78.17&\pagecolor{yellow}{+1.78}\\ \text{ ResNet-101 }&224\times 224&45&78.15&71.31&79.74&\pagecolor{yellow}{+1.59}\\ \text{ ResNet-152 }&224\times 224&64&78.59&72.38&80.15&\pagecolor{yellow}{+1.56}\\ \text{ ResNet-RS-50 }&160\times 160&36&79.10&70.72&79.97&\pagecolor{yellow}{+0.87}\\ \text{ ResNet-RS-101 }&160\times 160&64&80.11&72.73&80.89&\pagecolor{yellow}{+0.78}\\ \text{ ResNet-RS-101 }&190\times 190&64&81.29&73.63&81.80&\pagecolor{yellow}{+0.51}\\ \text{ ResNet-RS-152 }&224\times 224&87&82.81&74.46&83.10&\pagecolor{yellow}{+0.29}\\ \hline\cr\lx@intercol\hfil\text{ Transformers }\hfil\lx@intercol\\ \hline\cr\text{ ViT-S/16 }&224\times 224&22&79.89&71.88&81.00&\pagecolor{yellow}{+1.11}\\ \text{ DeiT-S }&224\times 224&22&78.97&72.26&80.49&\pagecolor{yellow}{+1.52}\\ \text{ DeiT-B }&224\times 224&87&81.79&74.55&82.84&\pagecolor{yellow}{+1.04}\\ \text{ DeiT-B }&384\times 384&87&83.16&75.45&83.75&\pagecolor{yellow}{+0.59}\\ \text{ DeiT-L }&224\times 224&307&82.22&74.60&83.05&\pagecolor{yellow}{+0.83}\\ \hline\cr\hline\cr\end{array} |
| \begin{array}[]{l|c|c|c|c}\hline\cr\hline\cr\text{ {Backbone} }&\text{ {Real Only} }&\text{ {Generated Only} }&\text{ {Real + Generated} }&\text{ {Performance} }\Delta\\ \hline\cr\hline\cr\lx@intercol\hfil\text{ Semantic Segmentation on PASCAL VOC \cite[cite]{[\@@bibref{}{everingham2010pascal}{}{}]} }\hfil\lx@intercol\\ \hline\cr\text{ ResNet-50 }&43.4&60.3&66.1&\pagecolor{yellow}{+22.7}\\ \text{ Swin-B }&65.2&73.7&78.5&\pagecolor{yellow}{+13.3}\\ \hline\cr\lx@intercol\hfil\text{ Instance Segmentation on COCO \cite[cite]{[\@@bibref{}{lin2014microsoft}{}{}]} }\hfil\lx@intercol\\ \hline\cr\text{ ResNet-50 }&4.4&12.2&14.8&\pagecolor{yellow}{+10.4}\\ \text{ Swin-B }&11.3&17.6&23.3&\pagecolor{yellow}{+12}\\ \hline\cr\hline\cr\end{array} |
| \begin{array}[]{l|c|c|c|c|c|c}\hline\cr\hline\cr\text{ {Metric} }&\text{ {Real Only} }&\text{ {Real Only w/ Cut Paste} }&\text{ {Generated Foreground} }&\text{ {Generated Only} }&\text{ {Real + Generated} }&\text{ {Performance} }\Delta\\ \hline\cr\hline\cr\lx@intercol\hfil\text{ Object Detection on PASCAL VOC \cite[cite]{[\@@bibref{}{everingham2010pascal}{}{}]} }\hfil\lx@intercol\\ \hline\cr\text{ mAP@50 }&9.12&29.60&48.14&44.59&51.82&\pagecolor{yellow}{+42.7}\\ \text{ mAP }&2.35&10.82&21.62&20.72&25.87&\pagecolor{yellow}{+23.52}\\ \hline\cr\lx@intercol\hfil\text{ Object Detection on COCO \cite[cite]{[\@@bibref{}{lin2014microsoft}{}{}]} }\hfil\lx@intercol\\ \hline\cr\text{ mAP@50 }&1.47&2.89&17.87&16.22&20.82&\pagecolor{yellow}{+19.35}\\ \text{ mAP }&0.92&1.23&8.64&7.66&10.63&\pagecolor{yellow}{+9.71}\\ \hline\cr\hline\cr\end{array} |
| Zuhao Yang is currently a Ph.D. candidate with the Visual Intelligence Lab at Nanyang Technological University, Singapore, under the supervision of Shijian Lu. He received the B.S. degree in computing science from the University of Alberta. He previously worked on native multimodal foundation models at Kimi (Moonshot AI) and collaborated with researchers at MiroMind and ByteDance. His research focuses on video-centric multimodal intelligence, including temporal grounding, agentic reasoning, and long-horizon tool use. |
| Yu Wei is a Ph.D. student at the College of Computing and Data Science, Nanyang Technological University, Singapore. He is with the Visual Intelligence Lab. His research interests include 3D vision and deep learning, particularly 3D scene reconstruction and understanding. |
| Fangneng Zhan is an Assistant Professor at The Hong Kong University of Science and Technology, where he leads the World Mind Lab. He is also Chief Scientist of Simate AI. He received the Ph.D. in computer science and engineering from Nanyang Technological University in 2021. He previously held postdoctoral positions at Harvard University and the Max Planck Institute for Informatics and was a researcher at MIT. His research covers world models, robotics, and physical agents, including 3D and multimodal modeling, robot learning, and autonomous perception. |
| Kunhao Liu received the bachelor’s degree in software engineering from Beihang University, Beijing, China, in 2022. He received the Ph.D. degree in computer science from Nanyang Technological University, Singapore, in 2026, under the supervision of Shijian Lu. His research focuses on generative vision, particularly video foundation models and world models. He was a visiting student at the Max Planck Institute for Informatics and a research intern at Tencent ARC Lab and morpheus.ai. |
| Muyu Xu is currently a Ph.D. candidate at Nanyang Technological University, Singapore. His research interests include 3D vision and deep learning. His work spans neural radiance fields, 3D Gaussian splatting, and novel view synthesis. |
| Xiaoqin Zhang received the bachelor’s degree from Central South University, China, in 2005, and the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2010. He is currently a Full Professor at Zhejiang University of Technology, China. He previously held faculty positions at Wenzhou University. His research interests include artificial intelligence, computer vision, machine learning, and image processing. He was a recipient of China’s State Natural Science Award (Second Class) for 2020 and is an IET Fellow. |
| Ling Shao (Fellow, IEEE) received the B.Eng. degree from the University of Science and Technology of China and the M.Sc. and D.Phil. degrees from the University of Oxford, U.K. He is the Global President and Chief AI Officer of Terminus Group. He was the founding CEO of the Inception Institute of Artificial Intelligence and the founding Provost and Executive Vice President of Mohamed bin Zayed University of Artificial Intelligence. His research interests include computer vision, generative AI, multimodal AI, and AI for healthcare. He is a Fellow of the IAPR and received the Mohammed bin Rashid Medal for Scientific Distinction. |
| Shijian Lu is an Associate Professor with the College of Computing and Data Science, Nanyang Technological University, Singapore. He received the Ph.D. degree in electrical and computer engineering from the National University of Singapore. His research interests include computer vision, deep learning, visual intelligence, and image and video analytics. Before joining NTU, he held research and leadership positions at the Institute for Infocomm Research, A*STAR, Singapore. He has held associate editorships for Pattern Recognition and Neurocomputing and area chair or senior program committee roles at computer vision and AI conferences. |
Synthetic Data Augmentation
Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.
See all 474 papers