Diffusion-Based Data Augmentation for Image Recognition: A Systematic Analysis and Evaluation

Zekun Li; Yinghuan Shi; Yang Gao; Dong Xu

cs.CVMar 9, 2026

Diffusion-Based Data Augmentation for Image Recognition: A Systematic Analysis and Evaluation

Authors: Zekun Li, Yinghuan Shi, Yang Gao, Dong Xu

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China. · School of Computing and Data Science, The University of Hong Kong, Hong Kong SAR, China.

Abstract

Diffusion-based data augmentation (DiffDA) has emerged as a promising approach to improving classification performance under data scarcity. However, existing works vary significantly in task configurations, model choices, and experimental pipelines, making it difficult to fairly compare methods or assess their effectiveness across different scenarios. Moreover, there remains a lack of systematic understanding of the full DiffDA workflow. In this work, we introduce UniDiffDA, a unified analytical framework that decomposes DiffDA methods into three core components: model fine-tuning, sample generation, and sample utilization. This perspective enables us to identify key differences among existing methods and clarify the overall design space. Building on this framework, we develop a comprehensive and fair evaluation protocol, benchmarking representative DiffDA methods across diverse low-data classification tasks. Extensive experiments reveal the relative strengths and limitations of different DiffDA strategies and offer practical insights into method design and deployment. All methods are re-implemented within a unified codebase, with full release of code and configurations to ensure reproducibility and to facilitate future research.

Figures & tables

Figure 1: Illustration of three diffusion-based image-to-image transition techniques used in representative DiffDA methods: (a) SDEdit, (b) InstructPix2Pix, and (c) DDIM Inversion and Interpolation.

Figure 2: Illustration of the fine-tuning pipeline combining Textual Inversion and DreamBooth-LoRA. A pseudo-token (e.g., <Sage_Thrasher> ) is learned and inserted into the text prompt to synthesize class-specific images. The corresponding embedding is optimized while keeping the rest of the text encoder frozen. Meanwhile, LoRA modules are inserted into the UNet and fine-tuned to better capture target-domain visual features. The generated image is trained to match the real target image via reconstruction loss. Tunable parameters are highlighted in orange.

Figure 3: Illustration of the UniDiffDA framework, which decomposes diffusion-based data augmentation (DiffDA) workflow into three core components: (1) Model Fine-tuning , where the diffusion model is optionally adapted to the target domain using real images; (2) Sample Generation , where the model synthesizes new samples guided by real data; and (3) Sample Utilization , where synthetic samples are either concatenated with or used to replace real samples for classifier training.

Method	Venue	Key Technical Designs
		✛No fine-tuning
Real Guidance ( He et al., 2023 )	ICLR 2023	✦SDEdit, single simple prompt
		✶Full Concatenation
		✛No fine-tuning
GIF ( Zhang et al., 2023b )	NeurIPS 2023	✦SDEdit, prompt pooling, noise optimization
		✶Full Concatenation

Table 1: Summary of representative DiffDA methods and their key technical designs under the UniDiffDA framework. 4 4 footnotetext: We use ✛, ✦, and ✶ to denote technical details related to model fine-tuning, sample generation, and sample utilization, respectively. 1 Prompt pooling refers to randomly selecting a prompt from a curated prompt set.

Methods	Datasets
Methods	Caltech-101	CIFAR-100	ImageNet-100	ImageNet-1K
Baseline	49.95 ± 3.94	55.22 ± 2.90	65.28 ± 0.65	22.37 ± 0.45
Mixup	51.75 ± 1.79	58.06 ± 2.15	67.45 ± 1.20	23.54 ± 1.20
CutMix	47.65 ± 1.39	56.08 ± 1.67	68.40 ± 1.81	24.08 ± 0.97
Real Guidance	67.20 ± 1.57	66.69 ± 0.53	76.83 ± 0.36	39.07 ± 0.22
GIF	76.11 ± 0.33	68.23 ± 0.51	78.63 ± 0.49	39.52 ± 0.35

Table 2: Top-1 accuracy (%) of ResNet-50 trained from scratch on conventional classification tasks.

Figure 4: Examples of synthetic “Beaver” images from CIFAR100 and “Wild Boar” images from ImageNet-100.

Methods	1-shot	5-shot	10-shot	Average
Baseline	16.20 ± 0.90	52.24 ± 0.85	69.94 ± 0.20	46.13
Mixup	13.91 ± 1.38	48.51 ± 0.98	67.40 ± 0.05	43.27
CutMix	12.45 ± 0.91	35.47 ± 1.26	59.49 ± 0.74	35.80
Real Guidance	17.29 ± 0.69	52.78 ± 0.32	69.94 ± 0.45	46.67
GIF	17.09 ± 1.23	52.32 ± 0.41	67.76 ± 0.35	45.72
DiffuseMix	16.34 ± 1.21	54.48 ± 1.32	68.44 ± 1.51	46.42

Table 3: Top-1 accuracy (%) of ImageNet-pre-trained ResNet-50 on few-shot fine-grained classification on the Birds dataset.

Methods	1-shot	5-shot	10-shot	Average
Baseline	10.21 ± 1.41	32.26 ± 1.16	52.64 ± 0.90	31.70
Mixup	7.57 ± 1.13	31.45 ± 0.14	54.12 ± 0.14	31.05
CutMix	6.62 ± 0.74	24.00 ± 0.77	43.70 ± 0.77	24.77
Real Guidance	11.23 ± 0.36	32.93 ± 0.43	53.72 ± 0.86	32.62
GIF	11.79 ± 0.48	35.12 ± 0.89	56.26 ± 0.43	34.39
DiffuseMix	13.03 ± 1.32	37.04 ± 1.38	57.89 ± 0.72	35.99

Table 4: Top-1 accuracy (%) of ImageNet-pre-trained ResNet-50 on few-shot fine-grained classification on the Aircraft dataset.

Figure 5: Examples of fine-grained concept generation with untuned diffusion models.

Figure 6: Although diffusion models fine-tuned on a limited set of real images can generate synthetic samples with overall structural resemblance, distinguishing between different types of blood cells relies on subtle morphological cues, such as nuclear shape and cytoplasmic staining, that are too intricate for the model to capture accurately.

Dataset	Blood		Skin
Number of Samples 1 1 1 Denoting the number of labeled samples per class.	5	25	10	50
Baseline	60.00 ± 1.56	77.67 ± 1.42	39.62 ± 1.75	50.48 ± 1.17
Mixup	60.33 ± 7.46	62.17 ± 8.32	40.67 ± 1.41	47.14 ± 2.54
CutMix	60.92 ± 6.42	64.58 ± 10.52	36.86 ± 0.29	49.05 ± 2.31
Real Guidance	70.67 ± 1.76	86.67 ± 1.53	42.29 ± 1.31	51.81 ± 0.72
GIF 2 2 2 For medical images, GIF adopts fine-tuned diffusion models.	66.58 ± 3.30	87.25 ± 1.98	40.95 ± 0.16	51.91 ± 1.08

Table 5: Top-1 accuracy (%) of ResNet-50 trained from scratch on medical classification tasks.

Figure 7: Illustration of hierarchical taxonomic annotations of the Semi-iNat dataset.

Method	Species		Family		Order
Method	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
Baseline	41.88	64.15	47.85	70.79	54.96	79.43
Real Guidance	38.96	61.89	44.78	66.63	53.85	78.27
GIF	37.83	60.21	43.91	66.26	53.16	77.83
Diff-Aug	32.91	55.73	43.96	66.37	54.58	79.81
Diff-Mix	37.04	60.86	45.06	67.20	55.42	81.94

Table 6: Top-1 and Top-5 accuracy (%) of different DiffDA methods on the Semi-iNat dataset under varying semantic granularities.

Figure 8: Examples of order-level categories from the Semi-iNat dataset that show increased or decreased classification accuracy after applying DiffDA.

Figure 9: Visualization of multi-domain samples in the DomainNet dataset.

Method	real	clipart	infograph	painting	quickdraw	sketch
Base	24.52	6.43	1.59	6.03	0.78	2.28
Real Guidance	42.14	22.38	5.85	17.04	3.21	11.36
GIF	46.81	26.75	7.30	22.32	5.09	18.49
DiffuseMix	32.43	10.49	3.04	10.72	1.07	5.45
Diff-Aug	48.93	26.93	8.14	20.72	3.91	14.81
Diff-Mix	49.68	31.59	11.01	24.67	4.99	24.90

Table 7: Top-1 accuracy (%) of different DiffDA methods on the DomainNet dataset across six domains. Each method is trained on the real domain and evaluated on both in-domain ( real ) and out-of-domain (OOD) settings.

Datasets	$s=0.1$	$s=0.3$	$s=0.5$	$s=0.7$	$s=0.9$
Caltech-101	54.96	56.48	58.70	64.31	65.38
CIFAR-100	58.10	59.21	60.06	65.16	66.24
ImageNet-100	66.30	68.16	70.67	76.70	77.12

Table 9: Top-1 accuracy (%) of Real Guidance under different image-to-image transition strengths on coarse-grained datasets.

Figure 12: Grid search results for classifier hyperparameters, including batch size and learning rate.

Real Data Size	Expansion Ratio
Real Data Size	1 $\times$	(1+1) $\times$	(1+5) $\times$	(1+10) $\times$	(1+20) $\times$
20%	14.27	21.64	34.96	41.98	49.58
40%	24.19	31.27	54.56	61.42	66.06
60%	34.05	47.93	67.08	69.58	74.05
80%	40.68	55.30	72.08	74.90	78.53
100%	54.33	60.23	76.49	79.21	82.04

Table 10: Top-1 accuracy (%) of the GIF method on conventional classification tasks of the Caltech-101 dataset with various combinations of real and generated data sizes.

Figure 13: Examples of “bear” images from the CIFAR-100 dataset generated by the Real Guidance method with different generative backbones at various resolutions. Failure cases of SD2.1 caused by resolution mismatch are indicated with red boxes.

Figure 14: Examples of “bobolink” images from the Birds dataset generated by the Diff-Aug method with different generative backbones. Incorrectly synthesized details and complete failure cases are indicated with red boxes.

Dataset	SD1.5	SD2.1		SD3.5
Dataset	$512\times 512$	$512\times 512$	$768\times 768$	$512\times 512$	$1024\times 1024$
Caltech-101	65.38	63.34	65.27	64.76	65.44
CIFAR-100	66.24	59.87	66.86	65.10	66.17

Table 11: Top-1 accuracy (%) of Real Guidance with different generative backbones at various resolutions.

Method	Caltech-101	CIFAR-100	Birds (5-shot)	Birds (10-shot)
Diff-Aug (SD1.5)	71.67	67.73	60.36	71.71
Diff-Aug (SD2.1)	71.27	68.19	48.73	64.01
Diff-Aug (SD3.5)	71.52	66.97	44.65	58.13
Diff-Mix (SD1.5)	75.62	71.45	62.96	73.26
Diff-Mix (SD2.1)	75.35	71.57	51.14	67.75
Diff-Mix (SD3.5)	75.21	70.64	46.10	60.27

Table 12: Top-1 accuracy (%) of Diff-Aug and Diff-Mix with different generative backbones.

Methods	Caltech-101				CIFAR-100
Methods	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$
Baseline	-	-	-	0.50	-	-	-	0.55
Real Guidance	42.79	0.38	0.28	0.67	84.45	0.03	0.17	0.67
GIF	54.81	0.40	0.33	0.76	113.22	0.01	0.10	0.68
DiffuseMix	41.22	0.82	0.87	0.63	149.97	0.01	0.09	0.59
DA-Fusion	41.99	0.38	0.29	0.68	83.54	0.03	0.18	0.67

Table 13: Quantitative analysis of images generated by different DiffDA methods on coarse-grained datasets.

Methods	5-shot				10-shot
Methods	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$
Baseline	-	-	-	0.52	-	-	-	0.70
Real Guidance	6.54	0.99	0.05	0.53	4.32	0.99	0.10	0.70
GIF	16.20	0.73	0.19	0.52	15.58	0.71	0.22	0.68
DiffuseMix	28.69	0.63	0.27	0.54	28.79	0.62	0.27	0.68
DA-Fusion	17.83	0.41	0.23	0.59	15.80	0.32	0.24	0.71

Table 14: Quantitative analysis of images generated by different DiffDA methods on the Birds dataset.

Methods	5-shot				10-shot
Methods	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$	FID $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$
Baseline	-	-	-	0.32	-	-	-	0.53
Real Guidance	5.95	0.99	0.04	0.33	3.42	0.99	0.07	0.54
GIF	23.14	0.39	0.20	0.35	22.15	0.38	0.22	0.56
DiffuseMix	49.52	0.31	0.19	0.37	49.90	0.33	0.37	0.58
DA-Fusion	14.30	0.33	0.50	0.33	15.62	0.37	0.27	0.53

Table 15: Quantitative analysis of images generated by different DiffDA methods on the Aircraft dataset.

Method	FT	GEN			CLS			Total
Method	FT	CA	CI	IN	CA	CI	IN	CA	CI	IN
Baseline	0.00	0.00	0.00	0.00	0.18	0.54	1.02	0.18	0.54	1.02
Real Guidance	0.00	1.91	7.70	20.07	0.68	2.26	5.11	2.59	9.96	25.18
GIF	0.00	2.26	9.21	22.59	0.68	2.26	5.11	2.94	11.47	27.70
DiffuseMix	0.00	2.56	10.53	26.28	0.68	2.26	5.11	3.24	12.79	31.39
DA-Fusion	1.98	1.92	7.71	20.17	0.68	2.26	5.11	4.58	11.95	27.26

Table 17: GPU hours required by each DiffDA method on Caltech-101 (CA), CIFAR-100 (CI), and ImageNet-100 (IN) datasets, broken down into fine-tuning (FT), generation (GEN), and classification (CLS) stages.

Method	Trial	Caltech-101	CIFAR-100	Birds (5-shot)	Birds (10-shot)
Diff-Aug	base	71.67	67.73	60.36	71.71
Diff-Aug	suffix-enrich	70.24	66.91	54.87	66.95
Diff-Aug	suffix-dream	73.03	69.62	60.78	71.82
Diff-Aug	suffix-exchange	72.16	69.15	61.19	72.13
Diff-Mix	base	75.62	71.45	62.96	73.26
Diff-Mix	suffix-enrich	72.54	69.13	56.29	68.81

Table 18: Top-1 accuracy (%) of different prompt engineering trials.

Figure 15: Examples of “European Goldfinch” images from the Birds dataset generated by the Diff-Aug method with different numbers of diffusion steps $T$ .

Figure 16: Comparison of filtering strategies under various filtering ratios.

Diffusion-Based Data Augmentation

See all 11 papers

When Generative Augmentation Hurts: A Benchmark Study of GAN and Diffusion Models for Bias Correction in AI Classification Systems
Mar 17, 2026Shesh Narayan Gupta, Nik Bear BrownSynthetic Data Augmentation Class-Imbalanced Learning
College of Engineering, Northeastern University, Boston, MA 02115, USA
Difficulty-Aware Sample Allocation for Adaptive Data Augmentation in Semantic Segmentation
Aug 26, 2026Olasimbo Ayodeji Arigbabu, Abimbola Ismail ArigbabuData Augmentation Semantic Segmentation
Independent Researcher, Germany. · Department of Science and Technology Education, Olabisi Onabanjo University, Ago-Iwoye, Ogun State, Nigeria.
Data-Efficient Classifier Pretraining Through Diffusion Models
Oct 16, 2026Qinyu Li, Yee Whye Teh, Chris Holmes +1Data-Efficient Learning Self-Supervised Pre-Training
Department of Statistics, University of Oxford · Faculty of Medicine and Health Technology, Tampere University · Tampere Institute for Advanced Study, Tampere University

Mar 17, 2026J/K move · Enter open · S save

When Generative Augmentation Hurts: A Benchmark Study of GAN and Diffusion Models for Bias Correction in AI Classification Systems

Shesh Narayan Gupta, Nik Bear Brown

College of Engineering, Northeastern University, Boston, MA 02115, USA

Synthetic Data Augmentation Class-Imbalanced Learning Generative Adversarial Networks Image Classification Diffusion-Based Image Generation

Open paper AskPDF arXiv

Generative models are widely used to compensate for class imbalance in AI training pipelines, yet their failure modes under low-data conditions are poorly understood. This paper reports a controlled benchmark comparing three augmentation strategies applied to a fine-grained animal classification task: traditional transforms, FastGAN, and Stable Diffusion 1.5 fine-tuned with Low-Rank Adaptation (LoRA). Using the Oxford-IIIT Pet Dataset with eight artificially underrepresented breeds, we find that FastGAN augmentation does not merely underperform at very low training set sizes but actively increases classifier bias, with a statistically significant large effect across three random seeds (bias gap increase: +20.7%, Cohen's d = +5.03, p = 0.013). The effect size here is large enough to give confidence in the direction of the finding despite the small number of seeds. Feature embedding analysis using t-distributed Stochastic Neighbor Embedding reveals that FastGAN images for severe-minority breeds form tight isolated clusters outside the real image distribution, a pattern consistent with mode collapse. Stable Diffusion with Low-Rank Adaptation produced the best results overall, achieving the highest macro F1 (0.9125 plus or minus 0.0047) and a 13.1% reduction in the bias gap relative to the unaugmented baseline. The data suggest a sample-size boundary somewhere between 20 and 50 training images per class below which GAN augmentation becomes harmful in this setting, though further work across additional domains is needed to establish where that boundary sits more precisely. All experiments run on a consumer-grade GPU with 6 to 8 GB of memory, with no cloud compute required.

Abstract

Diffusion-Based Data Augmentation

Explore similar work

When Generative Augmentation Hurts: A Benchmark Study of GAN and Diffusion Models for Bias Correction in AI Classification Systems

Difficulty-Aware Sample Allocation for Adaptive Data Augmentation in Semantic Segmentation

Data-Efficient Classifier Pretraining Through Diffusion Models