Variational Rectification Inference for Learning with Noisy Labels
Authors: Haoliang Sun, Qi Wei, Lei Feng, Yupeng Hu, Fan Liu, Hehe Fan, Yilong Yin
Organizations: School of Software, Shandong University, Jinan, China. · School of Computer Science and Engineering, Nanyang Technological University, Singapore. · School of Computing, National University of Singapore, Singapore. · School of Computer Science and Technology, Zhejiang University, Hangzhou, China.
Label noise has been broadly observed in real-world datasets. To mitigate the negative impact of overfitting to label noise for deep models, effective strategies (e.g., re-weighting, or loss rectification) have been broadly applied in prevailing approaches, which have been generally learned under the meta-learning scenario. Despite the robustness of noise achieved by the probabilistic meta-learning models, they usually suffer from model collapse that degenerates generalization performance. In this paper, we propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions as an amortized variational inference problem and derive the evidence lower bound under the meta-learning framework. Specifically, VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable, which can rectify the loss of the noisy sample with the extra randomness regularization and is, therefore, more robust to label noise. To achieve the inference of the rectifying vector, we approximate its conditional posterior with an amortization meta-network. By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function, which can significantly improve the generalization performance. The elaborated meta-network and prior network adhere to the smoothness assumption, enabling the generation of reliable rectification vectors. Given a set of clean meta-data, VRI can be efficiently meta-learned within the bi-level optimization programming. Besides, theoretical analysis guarantees that the meta-network can be efficiently learned with our algorithm. Comprehensive comparison experiments and analyses validate its effectiveness for robust learning with noisy labels, particularly in the presence of open-set noise.
Figures & tables
Figure 1: The meta-network can generate the rectifying vector to integrate into the inference of the classification network. The variational module can avoid model collapse via a prior network.
Figure 2: Flowchart of the learning algorithm. The solid and dashed lines denote forward and backward propagation, respectively. For each iteration, the meta-network ϕ generates the distribution of v and then produces multiple examples via the sampling module to estimate the predictive distribution. By computing the gradient through the update step 4, the meta-network can be trained in step 5. The prior network is also jointly optimized in step 6. The classification network θ will be updated with support of the learned meta-network in step 7.
Output size
Layers
1024
Input ConCat (sample features, embedding labels)
Vϕ(⋅)
512
fully connected, tanh
Number of classes
fully connected, Sigmoid to μv , logσv2
1024
Input sample features
Hω(⋅)
512
fully connected, tanh
Number of classes
fully connected, Sigmoid to μv , logσv2
Table 1: Architectures of applied meta-net in experiments.
Dataset
CIFAR-10
CIFAR-100
Clothing1M
Food-101N
ANIMAL-10N
Sampling Number
2
2
1
1
1
Batch Size
100
100
128
128
128
Optimizer
SGD
SGD
SGD
Adam
Adam
Initial Learning Rate
0.02
0.02
0.02
3e-4
3e-4
Decay Rate
5e-4
5e-4
5e-4
-
-
Total Epoch Number
160
160
10
30
30
Table 2: Hyperparameters of the classification network in our experiments on different datasets.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
Baseline
ResNet-32
76.83 ± 0.3
70.77 ± 2.3
50.86 ± 0.3
43.01 ± 1.2
MW-Net ( Shu et al., 2019 )
(NeurIPS19)
ResNet-32
90.33 ± 0.6
87.54 ± 0.2
64.22 ± 0.3
58.64 ± 0.5
MLC ( Wang et al., 2020 )
(CVPR20)
ResNet-32
90.07 ± 0.2
88.97 ± 0.5
64.91 ± 0.4
59.96 ± 0.6
CORES* ( Cheng et al., 2021 )
(ICLR21)
ResNet-32
91.41 ± 0.4
89.47 ± 0.3
64.82 ± 0.5
62.76 ± 0.4
PMW-Net ( Zhao et al., 2023 )
(TNNLS23)
ResNet-32
90.47 ± 0.1
87.69 ± 0.3
64.95 ± 0.2
58.72 ± 0.2
Table 3: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Flip label noise. The best and second-best performances are highlighted with blod and underline , respectively.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
ELR ( Liu et al., 2020 )
(NeurIPS20)
ResNet-34
91.43 ± 0.2
88.87 ± 0.2
68.43 ± 0.4
60.05 ± 0.9
MSLC ( Wu et al., 2021 )
(AAAI21)
ResNet-34
91.42
87.25
68.70
60.25
FaMUS ( Xu et al., 2021a )
(CVPR21)
ResNet-18
90.50
85.80
69.40
62.90
SFT ( Wei et al., 2022 )
(ECCV22)
ResNet-18
89.54 ± 0.3
-
69.72 ± 0.3
-
SOP ( Liu et al., 2022 )
(ICML22)
ResNet-34
90.09 ± 0.3
86.78 ± 0.2
70.12 ± 0.5
60.06 ± 0.4
Table 4: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Uniform label noise. The best and second-best performances are highlighted with blod and underline , respectively.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
Baseline
ResNet-18 / 34
85.10 ± 0.6
77.00 ± 2.1
52.19 ± 1.4
42.26 ± 1.2
Co-teaching ( Han et al., 2018b )
(NeurIPS18)
ResNet-18 / 34
86.54 ± 0.1
79.98 ± 0.3
57.24 ± 0.6
45.69 ± 0.9
Peer loss ( Liu and Guo, 2020 )
(ICML20)
ResNet-18 / 34
88.19 ± 0.5
81.53 ± 0.7
63.82 ± 0.3
47.91 ± 0.5
CORES* ( Cheng et al., 2021 )
(ICLR21)
ResNet-18 / 34
89.67 ± 0.3
82.99 ± 0.5
64.86 ± 0.5
49.62 ± 0.7
WarPI ( Sun et al., 2022a )
(PR22)
ResNet-18 / 34
89.76 ± 0.4
87.57 ± 0.9
65.08 ± 0.6
57.38 ± 1.0
Table 5: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Instance-dependent label noise. Note that “ResNet-18/34” denotes applying ResNet-18 for CIFAR-10 and ResNet-34 for CIFAR-100. The best and second-best performances are highlighted with blod and underline , respectively.
Clothing1M
Food-101N
MWNet ( Shu et al., 2019 )
73.72
DivideMix ( Li et al., 2020 )
74.76
Base Model
81.67
CNeth ( Lee et al., 2018 )
83.47
ELR ( Liu et al., 2020 )
74.81
CAL ( Zhu et al., 2021 )
74.17
MWNet ( Shu et al., 2019 )
84.72
SMP ( Han et al., 2019 )
85.11
PLC ( Zhang et al., 2021a )
73.24
WarPI ( Sun et al., 2022a )
74.98
NRank ( Sharma et al., 2020 )
85.20
ELR+ ( Liu et al., 2020 )
85.77
JNPL ( Kim et al., 2021 )
74.15
CoDis ( Xia et al., 2023 )
74.92
PLC ( Zhang et al., 2021a )
85.28
WarPI ( Sun et al., 2022a )
85.91
NCR ( Iscen et al., 2022 )
74.42
VRI (Ours)
75.19
CoDis ( Xia et al., 2023 )
86.13
VRI (Ours)
86.24
Table 6: Testing Accuracy (%) on real-world noise , including Clothing1M and Food-101N.
Methods
Unif 20%
Unif 50%
Unif 80%
Flip 40%
Cross-Entropy
-
42.57
27.06
9.27
22.25
Decoupling ( Malach and Shalev-Shwartz, 2017 )
NIPS 2017
43.49
-
10.01
33.74
Co-teaching ( Han et al., 2018b )
NIPS 2018
60.38
-
16.59
42.42
Co-teaching+ ( Yu et al., 2019 )
ICML 2019
53.97
-
12.29
43.01
JoCoR ( Wei et al., 2020 )
CVPR 2020
59.99
-
12.85
39.36
MoPro ( Li et al., 2021 )
ICLR 2021
65.60
-
30.29
60.22
Table 7: Average test accuracy on CIFAR80N, an open-set label noisy dataset, over the last 10 epochs.
Figure 3: (a) As the noise ratio rises, the rectification effect becomes more obvious since the area of the original loss increases. (b) We almost achieve the unbiased estimation for the initialized transition matrix of flip noise with varying noise ratio ρ .
Figure 4: The performance as the noise ratio increases is compared. VRI continues to deliver good performance even at significantly higher noise ratios.
Figure 5: Our algorithm achieves a stable convergence and displays robustness on flip noise with a high ratio (e.g., 60% ).
Figure 6: Model collapse of MC approximation.
Figure 7: MC method is more sensitive about the sample number compared with our proposed VRI.
k
Time (min./epoch)
Test Acc. (%)
1
2.17
88.23
MC
3
4.32
89.45
5
7.04
89.87
VRI
1
2.20
90.20
Table 8: VRI yields higher performance than MC approximation with an efficient inference.
Structure: {512+C^,h1,...,hn,C}
Accuracy
{512+C^,128,C}
89.76
{512+C^,512,512,C}
90.69
{512+C^,1024,512,C} (ours)
91.21
{512+C^,1024,1024,C}
90.89
{512+C^,1024,1024,512,512,C}
90.61
Table 9: Test accuracy (%) of different architectures of the meta-net on CIFAR-10 Flip 40%. C and C^ denote the number of classes and the dimension of the embedding vector of the label, respectively.
Homogeneous functions
Accuracy
Sigmoid (ours)
91.29
Tanh
88.17
w/o Sigmoid
86.90
Table 10: Test accuracy (%) of different activation functions in the last layer of the meta-net on CIFAR-10 Unif 40%. A ResNet-18 is used.
Noise
Method
Test Acc.
CIFAR-10
VRI
91.29
Unif. 40%
Non-Bayesian VRI
89.27
CIFAR-100
VRI
68.92
Unif. 40%
Non-Bayesian VRI
66.96
CIFAR-10
VRI
90.60
Inst. 40%
Non-Bayesian VRI
87.01
Table 11: Performance comparison (%) of VRI and its correspondingly non-Bayesian version.
Figure 8: (a) We obtain the best performance on with λ = 0.001. (b) The performance improves as the number of meta samples increases.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
40%
60%
40%
60%
Average gap
Uniform
Baseline
ResNet-18
68.07 ± 1.23
53.12 ± 3.03
51.11 ± 0.42
30.92 ± 0.33
26.0
VRI (-)
ResNet-18
80.23 ± 1.42
74.54 ± 2.46
59.39 ± 0.73
49.39 ± 0.46
10.1
VRI (+)
ResNet-18
91.24 ± 1.42
87.45 ± 2.16
66.39 ± 0.44
58.60 ± 0.56
1.0
VRI (Aug.)
ResNet-18
90.78 ± 1.56
87.98 ± 2.12
66.78 ± 0.68
58.79 ± 0.76
0.8
VRI (Standard)
ResNet-18
91.58 ± 0.17
88.68 ± 0.22
67.92 ± 0.19
59.32 ± 0.31
0
Table 12: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with uniform noise (top) and flip noise (bottom) when the accessibility of meta-data is restricted.
Baseline
Song et al. (2019)
Zhang et al. (2021a)
Chen et al. (2021)
Englesson (2021)
Chen et al. (2022)
VRI (-)
VRI (+)
79.4
81.8
83.4
84.1
84.2
84.5
81.4
85.8
Table 13: Testing accuracy (%) of VRI without given meta-data on ANIMAL-10N.