We examine the main claims of Zada, Benou, and Irani (2021), which states that in an image classification task with imbalanced training data, (i) using pure noise to augment minority-class images encourages generalization by improving minority-class accuracy. This method is paired with (ii) a new batch normalization layer that normalizes noise images using affine parameters learned from natural images, which improves the model's performance. Moreover, (iii) this improvement is robust to varying levels of data augmentation. Finally, the authors propose that (iv) adding pure noise images can improve classification even on balanced training data. We implemented the training pipeline from the description of the paper using PyTorch and integrated authors' code snippets for sampling pure noise images and batch normalizing noise and natural images separately. We reproduced the main claims that (i) oversampling with pure noise improves generalization by improving the minority-class accuracy, (ii) the proposed batch normalization (BN) method outperforms baselines, (iii) and this improvement is robust across data augmentations. Our results also support that (iv) adding pure noise images can improve classification on balanced training data. However, additional experiments suggest that the performance improvement from OPeN may be more orthogonal to the improvement caused by a bigger network or more complex data augmentation.
Figures & tables
Dataset
Imbalance Ratio (IR)
Number of training examples
CIFAR-10-LT
100
12406
CIFAR-10-LT
50
13996
CIFAR-100-LT
100
10847
CIFAR-100-LT
50
12608
Table 1: Different long-tail variants of the CIFAR-10/100 datasets. A higher imbalance ratio signifies that the dataset is more imbalanced.
Table 2: Default hyperparameters used for experiments. ∗ denote hyperparameters not described in [ 1 ] but confirmed through email. Check Section 5.2 for more details.
Source
Reported [ 1 ]
Ours
ERM
79.6
81.18
RS
75.1
74.82
DRS
83.0
83.22
OPeN
84.6
85.04
Table 3: Comparing accuracy of resampling schemes on CIFAR-10-LT (IR= 100 ) dataset. Reported accuracy are from Table 1 in [ 1 ] .
Dataset
CIFAR-10-LT
CIFAR-100-LT
Source
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
Standard BN [ 6 ]
81.45
81.81
49.18
49.26
Auxiliary BN [ 12 ]
83.38
82.23
50.13
51.27
DAR-BN [ 1 ]
84.64
85.04
51.50
52.12
Table 4: Ablation experiment: comparing DAR-BN with other Batch Normalization layers (IR= 100 ). Reported scores are from Table 4 in [ 1 ] .
Flip and Crop
Add Cutout
Add SimCLR
Source
Reported
Ours
Reported
Ours
Reported
Ours
ERM
74.3
74.6
77.7
78.7
79.6
80.7
DRS [ 2 ]
76.5
75.4
80.3
79.5
83.0
83.2
OPeN [ 1 ]
80.3
79.9
83.1
83.9
84.6
84.3
Table 5: Data augmentation ablation experiment. Reported accuracy are from Figure 3 in [ 1 ] .
Source
Improvement
Baseline Accuracy
Pure Noise Accuracy
Reported
+0.9%
-
-
Ours
+1.6%
87.16
88.57
Table 6: Performance improvement of OPeN on the full balanced CIFAR-10 dataset. The original paper reported the percentage improvement but not the baseline and pure noise accuracy.
Table 7: Performance of ResNet-32 models for CIFAR-10-LT (IR= 100 ). Note that for M2m [ 3 ] , a different dataset variant has been used. Check Section 4.2.2 for more information.
BN Layer
Accuracy
Standard BN [ 6 ]
74.37
Auxiliary BN [ 12 ]
75.08
DAR-BN [ 1 ]
77.52
Table 8: Batch normalization ablation experiment for OPeN. Same experiment setting as Table 4 , but with ResNet-32 and Flip and Crop augmentation.
Accuracy without pure noise
Accuracy with pure noise
Change
86.51
86.19
-0.37%
Table 9: Performance of adding pure noise to the full balanced CIFAR-10 dataset when using ResNet-32 model.
Source
ERM
DRS
OPeN
[ 8 ]
79.26
80.87
84.19
[ 3 ]
78.37
81.64
87.11
[ 2 ]
81.18
83.22
85.04
Reported by [ 1 ]
79.6
83.0
84.6
Table 10: Performance of models trained on different CIFAR-10-LT (IR=100) datasets from various sources.
Figure 1: t-SNE of 50 examples from each class in the CIFAR-10 validation dataset, 50 out-of-distribution images from the CIFAR-100 validation dataset, and 50 random pure noise. Images are embedded with a model trained with OPeN. Noise images are gathered into one cluster that is separable from any other class clusters, whereas OOD images are more dispersed across multiple classes. This phenomenon is not unique to OPeN, as it also appears with DRS (Figure 7 ).
Figure 2: Confusion matrices of models trained on CIFAR-10-LT (IR=100).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
CIFAR-10-LT
CIFAR-100-LT
IR
50
100
50
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
ERM
84.9
84.9
47.0
47.1
52.4
52.7
RS
82.2
80.9
42.5
41.6
48.0
46.5
DRS
-
86.9
-
50.8
-
55.8
OPeN
87.9
87.8
51.5
52.1
56.3
56.5
Appendix
Table 11: Comparison of accuracy on CIFAR-10-LT (IR=50) and CIFAR-100-LT (IR=100,50).
Source
Mean
Std
[ 9 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 2 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 3 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
CIFAR-10 †
(0.4914, 0.4822, 0.4465)
(0.2470, 0.2435, 0.2616)
CIFAR-10-LT (IR=50) †
(0.4978, 0.5003, 0.4840)
(0.2505, 0.2477, 0.2722)
CIFAR-10-LT (IR=100) †
(0.4989, 0.5044, 0.4926)
(0.2513, 0.2485, 0.2734)
Appendix
Table 12: Per-channel mean and standard deviations for experiments on CIFAR-10-LT. Values calculated in this paper are marked by † .
Source
Mean
Std
[ 9 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 2 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 3 ]
(0.5071, 0.4867, 0.4408)
(0.2675, 0.2565, 0.2761)
CIFAR-100 †
(0.5071, 0.4866, 0.4409)
(0.2673, 0.2564, 0.2762)
CIFAR-100-LT (IR=50) †
(0.5202, 0.4916, 0.4415)
(0.2676, 0.2609, 0.2778)
CIFAR-100-LT (IR=100) †
(0.5228, 0.4929, 0.4420)
(0.2677, 0.2617, 0.2780)
Appendix
Table 13: Per-channel mean and standard deviations for experiments on CIFAR-100-LT. Values calculated in this paper are marked by † .
Source
ERM
DRS
OPeN
Baseline [ 9 , 2 , 3 ]
81.18
83.22
85.04
CIFAR-10 †
80.50
82.39
84.72
CIFAR-10-LT (IR=100) †
79.28
81.03
84.12
Appendix
Table 14: Performance on different input normalization values on CIFAR-10-LT (IR=100). Values calculated in this paper are marked by † .
Batch size
Accuracy
ERM
DRS
OPeN
32
74.38
80.15
80.50
64
78.69
82.89
83.89
128
81.18
83.22
85.04
256
79.11
75.28
82.36
512
75.19
71.39
79.22
Appendix
Table 15: Hyperparameter search for batch size. Experiment was done on CIFAR-10-LT (IR=100) with the default setting in Table 2 except for the batch size. Results with highest accuracy for each method are boldfaced.
Noise ratio
\nicefrac16
\nicefrac26
\nicefrac36
\nicefrac46
\nicefrac56
\nicefrac66
Accuracy
83.23
85.04
84.34
84.23
83.31
80.01
Appendix
Table 16: Hyperparameter search for noise ratio. Experiment was done on CIFAR-10-LT (IR=100) with the default setting in Table 2 except for the noise ratio. Result with highest accuracy is boldfaced.
Figure 3: Venn diagram showing the intersections of training dataset used by [ 8 ] (denoted Cui ), [ 2 ] ( LDAM ), and [ 3 ] ( M2m ) for CIFAR-10-LT (IR=100).
Figure 4: Examples of automobile images only found in one but not the other 2 CIFAR-10-LT (IR=100) datasets.
Figure 5: Validation accuracy for CIFAR-10-LT and CIFAR-100-LT with IR=100. Classes partitioned into 5 groups, where Group 1 is the least frequent and Group 5 is the most frequent. Reproduction of Figure 6 .
Figure 6: Figure from [ 1 ] reporting validation accuracy for CIFAR-10-LT and CIFAR-100-LT with IR=100 where classes are partitioned into 5 groups. Group 1 is the least frequent and Group 5 is the most frequent.
Figure 7: t-SNE of 50 examples from each class in the CIFAR-10 validation dataset, 50 out-of-distribution images from CIFAR-100 validation dataset, and 50 random pure noise. Images are embedded with model trained with DRS.
Figure 8: Histogram of predicted classes for 1000 noise images.
Figure 9: Histogram of predicted classes for 1000 out-of-distribution images.
Generative models are widely used to compensate for class imbalance in AI training pipelines, yet their failure modes under low-data conditions are poorly understood. This paper reports a controlled benchmark comparing three augmentation strategies applied to a fine-grained animal classification task: traditional transforms, FastGAN, and Stable Diffusion 1.5 fine-tuned with Low-Rank Adaptation (LoRA). Using the Oxford-IIIT Pet Dataset with eight artificially underrepresented breeds, we find that FastGAN augmentation does not merely underperform at very low training set sizes but actively increases classifier bias, with a statistically significant large effect across three random seeds (bias gap increase: +20.7%, Cohen's d = +5.03, p = 0.013). The effect size here is large enough to give confidence in the direction of the finding despite the small number of seeds. Feature embedding analysis using t-distributed Stochastic Neighbor Embedding reveals that FastGAN images for severe-minority breeds form tight isolated clusters outside the real image distribution, a pattern consistent with mode collapse. Stable Diffusion with Low-Rank Adaptation produced the best results overall, achieving the highest macro F1 (0.9125 plus or minus 0.0047) and a 13.1% reduction in the bias gap relative to the unaugmented baseline. The data suggest a sample-size boundary somewhere between 20 and 50 training images per class below which GAN augmentation becomes harmful in this setting, though further work across additional domains is needed to establish where that boundary sits more precisely. All experiments run on a consumer-grade GPU with 6 to 8 GB of memory, with no cloud compute required.
Shesh Narayan Gupta, Nik Bear Brown
College of Engineering, Northeastern University, Boston, MA 02115, USA
Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical analysis of noise injection, which confirms that corruption by impulse noise approximates a Jacobian regularization, whereas Gaussian noise acts as a curvature penalty. This regularization behavior has been invoked to explain why noise injection increases model robustness. But the interleaved nature of our proposed schedule produces superior results even for the optimization objective: mixing phases of noisy data permits the optimizer to escape local minima and increase exploration without the risk of catastrophically forgetting the important features from the clean data. To stabilize this training scheme against the rapid changes of the loss when switching between clean and noisy data, we introduce a gradient-norm stabilization technique that scales noisy updates based on clean gradient magnitudes. We compare this method with other common augmentation methods and find substantial improvements in corruption tolerance and robustness to real-world distribution shifts on CIFAR-100-C, ImageNet-C, and ImageNet-R for ResNet and ViT architectures, with the best results being achieved by stacking our method on top of other augmentations. Through saliency and attention maps we show that the effect of interleaved noise injection stems from penalizing the failure modes encouraged by the inductive bias of the models: impulse noise works against the locality bias of convolutional (ResNet) architectures, and Gaussian noise reduces the tendency of attention-based models to pick up large-scale spurious features. Interleaved noise injection is therefore an effective tool to improve the test performance on clean, noisy, and out-of-distribution data at essentially zero computational cost.
Matt L. Wiemann, Peter Melchior, Andrew K. Saydjari
Data augmentation is a key technique for improving the robustness of image classification models. However, many recent approaches rely on diffusion-based synthesis or complex feature mixing strategies, which introduce substantial computational overhead or require external datasets. In this work, we explore a different direction: procedural augmentation based on analytic interference patterns. Unlike conventional augmentation methods that rely on stochastic noise, feature mixing, or generative models, our approach exploits Moire interference to generate structured perturbations spanning a wide range of spatial frequencies. We propose a lightweight augmentation method that procedurally generates Moire textures on-the-fly using a closed-form mathematical formulation. The patterns are synthesized directly in memory with negligible computational cost (0.0026 seconds per image), mixed with training images during training, and immediately discarded, enabling a storage-free augmentation pipeline without external data. Extensive experiments with Vision Transformers demonstrate that the proposed method consistently improves robustness across multiple benchmarks, including ImageNet-C, ImageNet-R, and adversarial benchmarks, outperforming standard augmentation baselines and existing external-data-free augmentation approaches. These results suggest that analytic interference patterns provide a practical and efficient alternative to data-driven generative augmentation methods.
Yuto Matsuo, Yoshihiro Fukuhara, Yuki M. Asano +3
National Institute of Advanced Industrial Science and Technology (AIST) · Tokyo Denki University · Waseda University +2