We examine the main claims of Zada, Benou, and Irani (2021), which states that in an image classification task with imbalanced training data, (i) using pure noise to augment minority-class images encourages generalization by improving minority-class accuracy. This method is paired with (ii) a new batch normalization layer that normalizes noise images using affine parameters learned from natural images, which improves the model's performance. Moreover, (iii) this improvement is robust to varying levels of data augmentation. Finally, the authors propose that (iv) adding pure noise images can improve classification even on balanced training data. We implemented the training pipeline from the description of the paper using PyTorch and integrated authors' code snippets for sampling pure noise images and batch normalizing noise and natural images separately. We reproduced the main claims that (i) oversampling with pure noise improves generalization by improving the minority-class accuracy, (ii) the proposed batch normalization (BN) method outperforms baselines, (iii) and this improvement is robust across data augmentations. Our results also support that (iv) adding pure noise images can improve classification on balanced training data. However, additional experiments suggest that the performance improvement from OPeN may be more orthogonal to the improvement caused by a bigger network or more complex data augmentation.
Figures & tables
Dataset
Imbalance Ratio (IR)
Number of training examples
CIFAR-10-LT
100
12406
CIFAR-10-LT
50
13996
CIFAR-100-LT
100
10847
CIFAR-100-LT
50
12608
Table 1: Different long-tail variants of the CIFAR-10/100 datasets. A higher imbalance ratio signifies that the dataset is more imbalanced.
Table 2: Default hyperparameters used for experiments. ∗ denote hyperparameters not described in [ 1 ] but confirmed through email. Check Section 5.2 for more details.
Source
Reported [ 1 ]
Ours
ERM
79.6
81.18
RS
75.1
74.82
DRS
83.0
83.22
OPeN
84.6
85.04
Table 3: Comparing accuracy of resampling schemes on CIFAR-10-LT (IR= 100 ) dataset. Reported accuracy are from Table 1 in [ 1 ] .
Dataset
CIFAR-10-LT
CIFAR-100-LT
Source
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
Standard BN [ 6 ]
81.45
81.81
49.18
49.26
Auxiliary BN [ 12 ]
83.38
82.23
50.13
51.27
DAR-BN [ 1 ]
84.64
85.04
51.50
52.12
Table 4: Ablation experiment: comparing DAR-BN with other Batch Normalization layers (IR= 100 ). Reported scores are from Table 4 in [ 1 ] .
Flip and Crop
Add Cutout
Add SimCLR
Source
Reported
Ours
Reported
Ours
Reported
Ours
ERM
74.3
74.6
77.7
78.7
79.6
80.7
DRS [ 2 ]
76.5
75.4
80.3
79.5
83.0
83.2
OPeN [ 1 ]
80.3
79.9
83.1
83.9
84.6
84.3
Table 5: Data augmentation ablation experiment. Reported accuracy are from Figure 3 in [ 1 ] .
Source
Improvement
Baseline Accuracy
Pure Noise Accuracy
Reported
+0.9%
-
-
Ours
+1.6%
87.16
88.57
Table 6: Performance improvement of OPeN on the full balanced CIFAR-10 dataset. The original paper reported the percentage improvement but not the baseline and pure noise accuracy.
Table 7: Performance of ResNet-32 models for CIFAR-10-LT (IR= 100 ). Note that for M2m [ 3 ] , a different dataset variant has been used. Check Section 4.2.2 for more information.
BN Layer
Accuracy
Standard BN [ 6 ]
74.37
Auxiliary BN [ 12 ]
75.08
DAR-BN [ 1 ]
77.52
Table 8: Batch normalization ablation experiment for OPeN. Same experiment setting as Table 4 , but with ResNet-32 and Flip and Crop augmentation.
Accuracy without pure noise
Accuracy with pure noise
Change
86.51
86.19
-0.37%
Table 9: Performance of adding pure noise to the full balanced CIFAR-10 dataset when using ResNet-32 model.
Source
ERM
DRS
OPeN
[ 8 ]
79.26
80.87
84.19
[ 3 ]
78.37
81.64
87.11
[ 2 ]
81.18
83.22
85.04
Reported by [ 1 ]
79.6
83.0
84.6
Table 10: Performance of models trained on different CIFAR-10-LT (IR=100) datasets from various sources.
Figure 1: t-SNE of 50 examples from each class in the CIFAR-10 validation dataset, 50 out-of-distribution images from the CIFAR-100 validation dataset, and 50 random pure noise. Images are embedded with a model trained with OPeN. Noise images are gathered into one cluster that is separable from any other class clusters, whereas OOD images are more dispersed across multiple classes. This phenomenon is not unique to OPeN, as it also appears with DRS (Figure 7 ).
Figure 2: Confusion matrices of models trained on CIFAR-10-LT (IR=100).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
CIFAR-10-LT
CIFAR-100-LT
IR
50
100
50
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
Reported [ 1 ]
Ours
ERM
84.9
84.9
47.0
47.1
52.4
52.7
RS
82.2
80.9
42.5
41.6
48.0
46.5
DRS
-
86.9
-
50.8
-
55.8
OPeN
87.9
87.8
51.5
52.1
56.3
56.5
Appendix
Table 11: Comparison of accuracy on CIFAR-10-LT (IR=50) and CIFAR-100-LT (IR=100,50).
Source
Mean
Std
[ 9 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 2 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 3 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
CIFAR-10 †
(0.4914, 0.4822, 0.4465)
(0.2470, 0.2435, 0.2616)
CIFAR-10-LT (IR=50) †
(0.4978, 0.5003, 0.4840)
(0.2505, 0.2477, 0.2722)
CIFAR-10-LT (IR=100) †
(0.4989, 0.5044, 0.4926)
(0.2513, 0.2485, 0.2734)
Appendix
Table 12: Per-channel mean and standard deviations for experiments on CIFAR-10-LT. Values calculated in this paper are marked by † .
Source
Mean
Std
[ 9 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 2 ]
(0.4914, 0.4822, 0.4465)
(0.2023, 0.1994, 0.2010)
[ 3 ]
(0.5071, 0.4867, 0.4408)
(0.2675, 0.2565, 0.2761)
CIFAR-100 †
(0.5071, 0.4866, 0.4409)
(0.2673, 0.2564, 0.2762)
CIFAR-100-LT (IR=50) †
(0.5202, 0.4916, 0.4415)
(0.2676, 0.2609, 0.2778)
CIFAR-100-LT (IR=100) †
(0.5228, 0.4929, 0.4420)
(0.2677, 0.2617, 0.2780)
Appendix
Table 13: Per-channel mean and standard deviations for experiments on CIFAR-100-LT. Values calculated in this paper are marked by † .
Source
ERM
DRS
OPeN
Baseline [ 9 , 2 , 3 ]
81.18
83.22
85.04
CIFAR-10 †
80.50
82.39
84.72
CIFAR-10-LT (IR=100) †
79.28
81.03
84.12
Appendix
Table 14: Performance on different input normalization values on CIFAR-10-LT (IR=100). Values calculated in this paper are marked by † .
Batch size
Accuracy
ERM
DRS
OPeN
32
74.38
80.15
80.50
64
78.69
82.89
83.89
128
81.18
83.22
85.04
256
79.11
75.28
82.36
512
75.19
71.39
79.22
Appendix
Table 15: Hyperparameter search for batch size. Experiment was done on CIFAR-10-LT (IR=100) with the default setting in Table 2 except for the batch size. Results with highest accuracy for each method are boldfaced.
Noise ratio
\nicefrac16
\nicefrac26
\nicefrac36
\nicefrac46
\nicefrac56
\nicefrac66
Accuracy
83.23
85.04
84.34
84.23
83.31
80.01
Appendix
Table 16: Hyperparameter search for noise ratio. Experiment was done on CIFAR-10-LT (IR=100) with the default setting in Table 2 except for the noise ratio. Result with highest accuracy is boldfaced.
Figure 3: Venn diagram showing the intersections of training dataset used by [ 8 ] (denoted Cui ), [ 2 ] ( LDAM ), and [ 3 ] ( M2m ) for CIFAR-10-LT (IR=100).
Figure 4: Examples of automobile images only found in one but not the other 2 CIFAR-10-LT (IR=100) datasets.
Figure 5: Validation accuracy for CIFAR-10-LT and CIFAR-100-LT with IR=100. Classes partitioned into 5 groups, where Group 1 is the least frequent and Group 5 is the most frequent. Reproduction of Figure 6 .
Figure 6: Figure from [ 1 ] reporting validation accuracy for CIFAR-10-LT and CIFAR-100-LT with IR=100 where classes are partitioned into 5 groups. Group 1 is the least frequent and Group 5 is the most frequent.
Figure 7: t-SNE of 50 examples from each class in the CIFAR-10 validation dataset, 50 out-of-distribution images from CIFAR-100 validation dataset, and 50 random pure noise. Images are embedded with model trained with DRS.
Figure 8: Histogram of predicted classes for 1000 noise images.
Figure 9: Histogram of predicted classes for 1000 out-of-distribution images.