Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains

Edward Zhang; Marcel Hussing; Tanay Tandon; Shenbagaraj Kannapiran; Jason Hughes; Youkang Wang; Joshua Caswell; Agelos Kratimenos

cs.CVAug 13, 2026

Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains

Authors: Edward Zhang, Marcel Hussing, Tanay Tandon, Shenbagaraj Kannapiran, Jason Hughes, Youkang Wang, Joshua Caswell, Agelos Kratimenos,

Organizations: University of Pennsylvania, Philadelphia, PA, USA · The Hong Kong Polytechnic University (PolyU), Hung Hom, Hong Kong

Abstract

Advances in diffusion-based generative models have motivated the use of synthetic image generation to alleviate data scarcity in vision tasks. While this strategy has shown promise in natural image benchmarks such as ImageNet, its effectiveness in sparse, high-variance real-world domains remains unclear. In this work, we focus on domains where images differ substantially from common image datasets and additional data are expensive to obtain. Against non-generative data augmentation baselines, we evaluate the downstream classifier performance improvements yielded by two schools of generative sparse data extension: distribution modeling and sample perturbation. Across five trauma classification tasks using subject-wise train--validation splits, no generative approach consistently outperforms a strong non-generative baseline. Feature-space analysis reveals recurring failure modes: memorization or collapse, distributional drift, and generation of visually plausible but simplified canonical instances that are easier to classify than real data.

Figures & tables

Panel of Figure 1 — Figure 1: Distribution of trauma labels across classification tasks. Torso trauma, head trauma, and severe hemorrhage are binary presence–absence tasks, while extremity trauma additionally distinguishes wounds and amputations.

Figure 2: Example images from realistic medical trauma simulations, including medical manikins (left images) and de-identified actors in moulage (right images) - (UPenn IRB #856195, USAISR Protocol #M-11034).

Figure 3: Generative Model Outputs. Compared to the real images in Figure 2 , many of the distribution model images are unrealistically clean regarding lack of adverse lighting, dirt and/or grime obscuring the wound, and potentially suboptimal camera angles, especially from GPT.

Method	Sev. Hem.	Head	Torso	Upper Ext.	Lower Ext.
Real Data Only	50.00 $\pm$ 0.00	50.00 $\pm$ 0.00	50.00 $\pm$ 0.00	33.33 $\pm$ 0.00	33.33 $\pm$ 0.00
Image Dup	63.11 $\pm$ 0.00	44.15 $\pm$ 0.00	67.73 $\pm$ 0.00	39.37 $\pm$ 0.00	60.81 $\pm$ 0.00
AugMix (Baseline)	63.38 $\pm$ 1.62	48.61 $\pm$ 0.45	62.45 $\pm$ 0.30	37.78 $\pm$ 0.76	66.56 $\pm$ 1.26
IP-Adapter	56.00 $\pm$ 2.14	49.37 $\pm$ 0.79	55.36 $\pm$ 0.97	42.87 $\pm$ 0.67	46.13 $\pm$ 2.44
Img2Img	49.75 $\pm$ 0.59	54.04 $\pm$ 0.74	60.55 $\pm$ 1.01	40.32 $\pm$ 1.09	47.16 $\pm$ 0.49
DB Img2Img	57.76 $\pm$ 0.71	51.36 $\pm$ 1.24	63.69 $\pm$ 0.19	41.11 $\pm$ 0.67	57.06 $\pm$ 1.51

Table 1: Macro-recall (%) with standard error.

Figure 4: MMD² between each synthetic dataset and the real data versus the average percent gain in classifier recall over the AugMix baseline. Whiskers denote standard error. Sample-perturbation methods tend to remain closer to the real data distribution and yield larger gains, while distribution-modeling approaches lie farther away. Overall, greater distributional similarity correlates with larger performance improvements

Figure 5: t-SNE plots of Generated Images Relative to Real Data - Sample Perturbation Methods tend to have lower $MMD^{2}$ and fill in gaps between existing real data while the Distribution Modeling Methods tend to spawn their own separate cluster, resulting in higher $MMD^{2}$

Method	Hemorrhage	Head	Torso	Upper Ext.	Lower Ext.
Diffusion	-4.6	+0.1	-4.6	-5.1	-9.5
DreamBooth	-8.2	+0.1	-2.8	-2.6	+2.3
GPT	-15.2	-7.9	-23.1	-26.9	+4.7
IP-Adapter	+2.1	+32.6	+6.8	-3.6	-4.2
SD Img2Img	-9.0	+8.0	+7.8	-9.7	+21.8
SD Img2Img + DB	+1.2	+9.3	+8.9	-0.9	+2.7

Table 2: Absolute percentage-point change in macro-recall relative to the AugMix baseline for end-to-end ResNet classifiers trained with different synthetic augmentation pipelines.

Method	1:1	1:2	1:5	1:10
GPT	56.6	59.3	55.5	51.2
IP-Adapter	62.1	61.8	61.2	58.0
SD Img2Img	59.6	57.6	60.6	58.2

Table 3: Mean macro-recall (%) across the five trauma classification tasks under different real-to-synthetic training ratios. Bold indicates the best ratio for each generation method.

Figure 6: DINOv2 embeddings, projected onto the top two principal components of the real training data, show that synthetic exemplars are more easily separable than real images, requiring simpler classification boundaries, as illustrated by a SVM with an RBF kernel

Augmentation Method	Macro-recall (%)
Real Data Only	77.48
AugMix (baseline)	80.46
DreamBooth (Direct Gen)	77.20
Stable Diffusion (FT)	75.41
Img2Img	77.79
Img2Img + DreamBooth	76.62

Table 4: Macro-recall on weld classification with synthetic augmentation.

Data Generation Method	Hemo	Head	Torso	Upper Ext	Lower Ext
DreamBooth	-7.02%	-2.25%	-9.44%	+17.57%	-15.94%
Img2Img	+5.31%	+3.94%	+1.12%	+23.89%	-9.37%
Img2Img + DreamBooth	+1.63%	+9.06%	-4.41%	-1.61%	-4.41%
IP-Adapter	+4.93%	+4.37%	-4.33%	+17.69%	-4.22%
GPT	-6.25%	+4.65%	-7.36%	-14.17%	-5.41%

Table 5: Performance gain relative to AugMix when exemplar synthetic data are mixed with real training data across injury categories. Positive values indicate improvement over AugMix; negative values (red) show that adding synthetic data hurts performance.

Figure 7: Examples of the Synthetic Generation Outputs for the Welding Data

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 453 papers

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks
May 12, 2026Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev +2Class-Imbalanced Learning Synthetic Data Augmentation
National University of Science and Technology MISIS, Moscow, Russia
Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting
Jul 2, 2026Disheng Liu, Tuo Liang, Chaoda Song +1Synthetic Data Augmentation Diversity-Aware Sampling
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
Cross-Domain Adversarial Augmentation: Stabilizing GANs for Medical and Handwriting Data Scarcity
May 3, 2026Md. Sohanuzzaman Soad, Mahady Al Hady, S M Rafiuddin Rifat +1Synthetic Data Augmentation Medical Image Synthesis
Department of Computer Science and Engineering University of Asia Pacific Dhaka, Bangladesh

May 12, 2026J/K move · Enter open · S save

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

Daniil Dushenev, Nazariy Karpov, Daniil Zinovjev, Alexander Gorin+1

National University of Science and Technology MISIS, Moscow, Russia

Class-Imbalanced Learning Synthetic Data Augmentation Diffusion Model Fine-Tuning Diffusion-Based Data Augmentation

Open paper AskPDF arXiv

Class imbalance is a persistent challenge in visual recognition, particularly in safety-critical domains where collecting positive examples is expensive and rare events are inherently underrepresented. We propose a lightweight synthetic data augmentation pipeline that fine-tunes a LoRA adapter on as few as 20-50 real images of a rare class and uses a pretrained diffusion model to generate synthetic samples for training. We systematically vary the synthetic-to-real ratio and evaluate the approach across two structurally different domains: chest X-ray pathology classification (NIH ChestX-ray14) and industrial surface crack detection (Magnetic Tile Defect dataset). All evaluations are performed on held-out sets of real images only. Across both domains, synthetic augmentation consistently improves rare-class recall and F1 compared to training with real data alone. Performance improves with moderate synthetic augmentation and shows diminishing returns as the synthetic ratio increases. These results suggest that LoRA-adapted diffusion models provide a simple and scalable mechanism for augmenting rare classes, enabling effective learning in data-scarce scenarios across heterogeneous visual domains.

Abstract

Synthetic Data Augmentation

Explore similar work

Few-Shot Synthetic Data Generation with Diffusion Models for Downstream Vision Tasks

Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting

Cross-Domain Adversarial Augmentation: Stabilizing GANs for Medical and Handwriting Data Scarcity