Generative Data Augmentation Without External Data: An Empirical and Theoretical Study on Deep Networks
Organizations: Department of Statistics, University of British Columbia
Abstract
Generative data augmentation adds samples from generative models to a classifier's training set. Many influential studies use generators pre-trained on large external datasets, and so mix the value of synthetic data with knowledge transferred through pretraining. We study the case without external data, where each generator learns only from the classifier's own training set, and ask when and why adding generated samples to that training set raises or lowers the classifier's test accuracy. We train classifiers with four backbones on six datasets, two of them medical, adding samples from four generators. We bound the excess risk of a classifier trained with synthetic data, that is, how far its population risk can lie above the minimum risk within its hypothesis class. The bound charges synthetic data for the total variation distance between the real and generated distributions and credits synthetic data for a larger training set. The gain from adding synthetic data, the fall in population risk, is at most the excess risk of the real-only classifier, which trains on the real images alone. Read through the bound, the experiments suggest that synthetic data help most when our classifiers detect only a small discrepancy between the generated and real data and the real-only classifier has room to improve, and that synthetic data help little or hurt when the detected discrepancy is large. Generators with a larger lower bound on the total variation distance, a bound that we estimate from our own classifiers, tend to help less on five of the six datasets, and on the CIFAR datasets the most accurate backbone gains the least. The bound gives these directions but not the size of the largest gains, most of which traditional augmentation also reaches at the same number of optimizer steps.
Figures & tables
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| CIFAR-10 | MNIST | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | Generator | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID |
| ViT | VAE | 85.44 | 85.14 | 85.49 | 85.86 | 86.18 | 85.50 | 87.39 | 87.51 | 88.07 | 113.9 | 99.01 | 98.99 | 99.03 | 98.99 | 99.09 | 99.05 | 99.06 | 99.19 | 99.14 | 20.9 |
| DDPM | 85.44 | 86.78 | 87.41 | 87.93 | 88.83 | 88.82 | 90.12 | 91.96 | 93.31 | 11.0 | 99.01 | 98.88 | 98.97 | 98.91 | 98.90 | 98.86 | 98.85 | 98.92 | 98.94 | 5.0 | |
| VP-SDE | 85.44 | 86.38 | 87.59 | 88.24 | 88.63 | 89.09 | 90.50 | 91.54 | 92.73 | 6.7 | 99.01 | 98.99 | 99.07 | 99.04 | 99.09 | 99.11 | 99.08 | 99.21 | 99.21 | 8.0 | |
| Flow matching | 85.44 | 86.63 | 87.34 | 88.04 | 88.84 | 88.74 | 90.24 | 91.71 | 92.60 | 9.3 | 99.01 | 99.02 | 99.00 | 98.92 | 99.02 | 99.06 | 99.04 | 99.11 | 99.15 | 3.0 | |
| Trad. aug. | 85.44 | 86.27 | 86.49 | 86.85 | 87.18 | 87.60 | 89.39 | 90.46 | 91.89 | 37.5 | 99.01 | 99.23 | 99.28 | 99.42 | 99.46 | 99.42 | 99.49 | 99.60 | 99.66 | 8.9 | |
| CIFAR-100 | Fashion-MNIST | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | Generator | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID |
| ViT | VAE | 58.69 | 58.13 | 58.75 | 58.39 | 58.70 | 58.97 | 61.18 | 63.27 | 63.80 | 101.2 | 93.72 | 93.60 | 93.84 | 93.71 | 93.94 | 93.77 | 93.74 | 93.84 | 93.77 | 47.4 |
| DDPM | 58.69 | 59.84 | 61.29 | 61.18 | 61.79 | 63.06 | 65.53 | 69.62 | 73.09 | 9.4 | 93.72 | 93.68 | 93.87 | 93.88 | 93.61 | 93.82 | 93.87 | 93.79 | 93.65 | 4.3 | |
| VP-SDE | 58.69 | 59.85 | 61.42 | 61.46 | 62.74 | 63.07 | 66.37 | 69.09 | 72.02 | 6.3 | 93.72 | 93.80 | 93.87 | 93.89 | 94.03 | 94.02 | 93.85 | 94.22 | 94.10 | 5.8 | |
| Flow matching | 58.69 | 59.79 | 61.01 | 61.17 | 61.54 | 62.40 | 65.23 | 68.13 | 71.31 | 11.0 | 93.72 | 93.78 | 93.76 | 93.80 | 93.81 | 93.76 | 93.74 | 93.73 | 93.47 | 5.0 | |
| Trad. aug. | 58.69 | 59.10 | 60.60 | 60.41 | 60.70 | 61.29 | 62.65 | 65.42 | 68.55 | 39.7 | 93.72 | 93.65 | 93.88 | 93.88 | 94.01 | 94.08 | 94.12 | 94.22 | 94.48 | 19.1 | |
| DermaMNIST | BloodMNIST | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | Generator | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | 2.0 | 4.0 | 8.0 | FID |
| ViT | VAE | 77.64 | 76.43 | 76.92 | 76.79 | 77.52 | 77.92 | 77.42 | 78.50 | 77.97 | 54.2 | 97.83 | 97.87 | 97.97 | 97.90 | 97.79 | 97.85 | 98.06 | 97.95 | 97.88 | 127.3 |
| DDPM | 77.64 | 76.41 | 75.33 | 75.46 | 74.91 | 75.34 | 74.90 | 74.88 | 72.62 | 33.3 | 97.83 | 97.88 | 97.85 | 97.81 | 97.63 | 98.10 | 97.79 | 97.82 | 97.91 | 5.3 | |
| VP-SDE | 77.64 | 76.97 | 76.69 | 76.21 | 76.77 | 76.31 | 75.71 | 76.04 | 75.66 | 25.2 | 97.83 | 97.87 | 98.10 | 98.20 | 98.16 | 98.19 | 98.30 | 98.26 | 98.34 | 4.8 | |
| Flow matching | 77.64 | 77.69 | 77.11 | 76.82 | 76.23 | 76.44 | 75.48 | 74.78 | 75.05 | 21.8 | 97.83 | 98.06 | 98.02 | 98.19 | 98.02 | 98.19 | 98.29 | 98.35 | 98.35 | 3.4 | |
| Trad. aug. | 77.64 | 76.82 | 77.67 | 77.97 | 77.01 | 78.49 | 77.24 | 77.32 | 78.40 | 132.5 | 97.83 | 97.96 | 97.87 | 98.01 | 98.00 | 98.16 | 98.36 | 97.91 | 98.50 | 110.9 | |
| MNIST | CIFAR-10 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | VAE | DDPM | VP-SDE | FM | Trad. | VAE | DDPM | VP-SDE | FM | Trad. | |||
| ResNet-18 | 0.2 | 99.09 | 99.01 | 98.55 | 98.67 | 98.85 | 99.32 | 84.20 | 84.74 | 85.55 | 86.26 | 85.79 | 87.25 |
| 0.4 | 99.36 | 99.39 | 98.96 | 99.14 | 99.20 | 99.55 | 90.24 | 89.44 | 90.95 | 90.90 | 90.61 | 91.35 | |
| 0.6 | 99.43 | 99.45 | 99.24 | 99.36 | 99.35 | 99.63 | 92.50 | 91.51 | 92.57 | 92.61 | 92.62 | 93.36 | |
| 0.8 | 99.57 | 99.60 | 99.28 | 99.29 | 99.46 | 99.63 | 93.76 | 92.78 | 93.52 | 93.54 | 93.56 | 94.24 | |
| 1.0 | 99.58 | 99.64 | 99.25 | 99.43 | 99.43 | 99.67 | 94.85 | 93.38 | 94.51 | 94.44 | 94.31 | 95.11 | |
| CIFAR-100 | Fashion-MNIST | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classifier | VAE | DDPM | VP-SDE | FM | Trad. | VAE | DDPM | VP-SDE | FM | Trad. | |||
| ResNet-18 | 0.2 | 51.07 | 49.00 | 53.46 | 55.39 | 52.83 | 55.56 | 91.99 | 92.35 | 92.06 | 92.17 | 92.06 | 92.04 |
| 0.4 | 64.94 | 60.37 | 64.76 | 64.68 | 63.11 | 66.91 | 93.55 | 93.68 | 93.27 | 93.53 | 93.54 | 93.49 | |
| 0.6 | 71.10 | 66.89 | 69.78 | 69.77 | 67.79 | 71.83 | 94.45 | 94.34 | 93.91 | 94.01 | 94.01 | 94.24 | |
| 0.8 | 74.22 | 70.46 | 72.44 | 72.53 | 71.03 | 75.07 | 94.91 | 94.75 | 94.37 | 94.44 | 94.38 | 94.78 | |
| 1.0 | 76.66 | 72.62 | 75.93 | 75.85 | 74.92 | 76.85 | 95.07 | 94.78 | 94.65 | 95.02 | 94.75 | 95.04 | |
| Dataset | VAE | DDPM | VP-SDE | FM |
|---|---|---|---|---|
| MNIST | 0.03 | 0.13 | 0.14 | 0.03 |
| Fashion-MNIST | 0.12 | 0.10 | 0.15 | 0.06 |
| CIFAR-10 | 0.77 | 0.10 | 0.03 | 0.06 |
| CIFAR-100 | 0.70 | 0.17 | 0.08 | 0.16 |
| DermaMNIST | 0.40 | 0.40 | 0.23 | 0.21 |
| BloodMNIST | 0.12 | 0.05 | 0.01 | 0.01 |
| Dataset | VAE | DDPM | VP-SDE | FM | |
|---|---|---|---|---|---|
| MNIST | 0.06 | 0.72 | 0.52 | 0.31 | |
| 0.06 | 0.28 | 0.18 | 0.11 | ||
| 0.05 | 0.19 | 0.15 | 0.04 | ||
| 0.03 | 0.16 | 0.14 | 0.04 | ||
| 0.03 | 0.13 | 0.14 | 0.03 | ||
| Fashion-MNIST | 0.16 | 0.20 | 0.18 | 0.14 |
| Datasets | MNIST, Fashion-MNIST, CIFAR-10, |
|---|---|
| CIFAR-100, DermaMNIST, BloodMNIST | |
| Classifiers | ResNet-18, ViT, Swin-T, ConvNeXt-V2 |
| Generators | VAE, DDPM, VP-SDE, flow matching; |
| traditional augmentation baseline (no generator) | |
| Splits | official train/test, stratified tenth |
| of train held out for validation |
| Used for | Package and version | Source |
|---|---|---|
| Tensors, training and sampling | PyTorch 2.8.0, Python 3.9, CUDA 12 | pytorch/pytorch |
| ResNet-18, Swin-T, MNIST, Fashion-MNIST and CIFAR | torchvision 0.23.0 | pytorch/vision |
| Crop, flip, affine and RandAugment | torchvision 0.23.0 | pytorch/vision |
| ViT | vit-pytorch 1.24.1 | lucidrains/vit-pytorch |
| ConvNeXt-V2 Femto | timm 1.0.28 | huggingface/pytorch-image-models |
| Flow-matching path and ODE solver | flow_matching 1.0.10 | facebookresearch/flow_matching |
| Dataset | Official training | Training ( ) | Validation | Test |
|---|---|---|---|---|
| MNIST | 60,000 | 54,000 | 6,000 | 10,000 |
| Fashion-MNIST | 60,000 | 54,000 | 6,000 | 10,000 |
| CIFAR-10 | 50,000 | 45,000 | 5,000 | 10,000 |
| CIFAR-100 | 50,000 | 45,000 | 5,000 | 10,000 |
| DermaMNIST | 7,007 | 6,306 | 701 | 2,005 |
| BloodMNIST | 11,959 | 10,763 | 1,196 | 3,421 |
| Setting | ResNet-18 | ViT | Swin-T | ConvNeXt-V2 |
|---|---|---|---|---|
| input | native | native | 64 px | 64 px |
| stem / patch | 3 by 3 conv s1, no max pool | patch 4 (8 at 64 px) | patch 4, window 7 | 4 by 4 conv s4 |
| architecture | torchvision resnet18 | dim 256 / depth 6 / heads 8 / MLP 512 / dropout 0.1 | torchvision swin_t | timm convnextv2_femto |
| initialization | random | random | random | random |
| trained parameters | all | all | all | all |
| optimizer | SGD, momentum 0.9 | AdamW | AdamW | AdamW |
| Setting | MNIST | Fashion-MNIST | CIFAR-10/100 | DermaMNIST / BloodMNIST |
|---|---|---|---|---|
| base channels | 32 | 32 | 128 | 128 |
| channel multipliers | (1, 2) | (1, 2) | (1, 2, 2, 2) | (1, 2, 2, 4) |
| residual blocks per resolution | 2 | 2 | 2 | 2 |
| attention resolutions | (16, 14) | (16, 14) | (16, 14) | (16, 14) |
| dropout | 0.1 | 0.1 | 0.1 | 0.1 |
| optimizer | AdamW | AdamW | AdamW | AdamW |
| Generator | Setting | Value |
|---|---|---|
| DDPM | timesteps | 1000 |
| noise schedule | cosine | |
| label-drop probability (CFG training) | 0.1 | |
| guidance scale at sampling | 0.0 | |
| sampler | DDIM, 250 steps | |
| VAE | latent dimension | 128 ( datasets); 256 (others) |
| Generator | MNIST | Fashion-MNIST | CIFAR-10 | CIFAR-100 | DermaMNIST | BloodMNIST |
|---|---|---|---|---|---|---|
| VAE | 20.9 | 47.4 | 113.9 | 101.2 | 54.2 | 127.3 |
| DDPM | 5.0 | 4.3 | 11.0 | 9.4 | 33.3 | 5.3 |
| VP-SDE | 8.0 | 5.8 | 6.7 | 6.3 | 25.2 | 4.8 |
| Flow matching | 3.0 | 5.0 | 9.3 | 11.0 | 21.8 | 3.4 |
| Trad. aug. | 8.9 | 19.1 | 37.5 | 39.7 | 132.5 | 110.9 |
| Symbol | Meaning | Defined in |
|---|---|---|
| Data and distributions | ||
| number of images in the full real training set | Sec. 3.1 | |
| real-data fraction, the share of each class of that keeps | Sec. 3.1 | |
| number of real training points, | Sec. 3.1 | |
| mixing ratio, so synthetic points join the real ones | Sec. 3.1 | |
| true distribution of labeled images | Sec. 4 | |
Synthetic Data Augmentation
Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.
See all 478 papers