Variational Rectification Inference for Learning with Noisy Labels
Authors: Haoliang Sun, Qi Wei, Lei Feng, Yupeng Hu, Fan Liu, Hehe Fan, Yilong Yin
Organizations: School of Software, Shandong University, Jinan, China. · School of Computer Science and Engineering, Nanyang Technological University, Singapore. · School of Computing, National University of Singapore, Singapore. · School of Computer Science and Technology, Zhejiang University, Hangzhou, China.
Label noise has been broadly observed in real-world datasets. To mitigate the negative impact of overfitting to label noise for deep models, effective strategies (e.g., re-weighting, or loss rectification) have been broadly applied in prevailing approaches, which have been generally learned under the meta-learning scenario. Despite the robustness of noise achieved by the probabilistic meta-learning models, they usually suffer from model collapse that degenerates generalization performance. In this paper, we propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions as an amortized variational inference problem and derive the evidence lower bound under the meta-learning framework. Specifically, VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable, which can rectify the loss of the noisy sample with the extra randomness regularization and is, therefore, more robust to label noise. To achieve the inference of the rectifying vector, we approximate its conditional posterior with an amortization meta-network. By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function, which can significantly improve the generalization performance. The elaborated meta-network and prior network adhere to the smoothness assumption, enabling the generation of reliable rectification vectors. Given a set of clean meta-data, VRI can be efficiently meta-learned within the bi-level optimization programming. Besides, theoretical analysis guarantees that the meta-network can be efficiently learned with our algorithm. Comprehensive comparison experiments and analyses validate its effectiveness for robust learning with noisy labels, particularly in the presence of open-set noise.
Figures & tables
Figure 1: The meta-network can generate the rectifying vector to integrate into the inference of the classification network. The variational module can avoid model collapse via a prior network.
Figure 2: Flowchart of the learning algorithm. The solid and dashed lines denote forward and backward propagation, respectively. For each iteration, the meta-network ϕ generates the distribution of v and then produces multiple examples via the sampling module to estimate the predictive distribution. By computing the gradient through the update step 4, the meta-network can be trained in step 5. The prior network is also jointly optimized in step 6. The classification network θ will be updated with support of the learned meta-network in step 7.
Output size
Layers
1024
Input ConCat (sample features, embedding labels)
Vϕ(⋅)
512
fully connected, tanh
Number of classes
fully connected, Sigmoid to μv , logσv2
1024
Input sample features
Hω(⋅)
512
fully connected, tanh
Number of classes
fully connected, Sigmoid to μv , logσv2
Table 1: Architectures of applied meta-net in experiments.
Dataset
CIFAR-10
CIFAR-100
Clothing1M
Food-101N
ANIMAL-10N
Sampling Number
2
2
1
1
1
Batch Size
100
100
128
128
128
Optimizer
SGD
SGD
SGD
Adam
Adam
Initial Learning Rate
0.02
0.02
0.02
3e-4
3e-4
Decay Rate
5e-4
5e-4
5e-4
-
-
Total Epoch Number
160
160
10
30
30
Table 2: Hyperparameters of the classification network in our experiments on different datasets.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
Baseline
ResNet-32
76.83 ± 0.3
70.77 ± 2.3
50.86 ± 0.3
43.01 ± 1.2
MW-Net ( Shu et al., 2019 )
(NeurIPS19)
ResNet-32
90.33 ± 0.6
87.54 ± 0.2
64.22 ± 0.3
58.64 ± 0.5
MLC ( Wang et al., 2020 )
(CVPR20)
ResNet-32
90.07 ± 0.2
88.97 ± 0.5
64.91 ± 0.4
59.96 ± 0.6
CORES* ( Cheng et al., 2021 )
(ICLR21)
ResNet-32
91.41 ± 0.4
89.47 ± 0.3
64.82 ± 0.5
62.76 ± 0.4
PMW-Net ( Zhao et al., 2023 )
(TNNLS23)
ResNet-32
90.47 ± 0.1
87.69 ± 0.3
64.95 ± 0.2
58.72 ± 0.2
Table 3: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Flip label noise. The best and second-best performances are highlighted with blod and underline , respectively.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
ELR ( Liu et al., 2020 )
(NeurIPS20)
ResNet-34
91.43 ± 0.2
88.87 ± 0.2
68.43 ± 0.4
60.05 ± 0.9
MSLC ( Wu et al., 2021 )
(AAAI21)
ResNet-34
91.42
87.25
68.70
60.25
FaMUS ( Xu et al., 2021a )
(CVPR21)
ResNet-18
90.50
85.80
69.40
62.90
SFT ( Wei et al., 2022 )
(ECCV22)
ResNet-18
89.54 ± 0.3
-
69.72 ± 0.3
-
SOP ( Liu et al., 2022 )
(ICML22)
ResNet-34
90.09 ± 0.3
86.78 ± 0.2
70.12 ± 0.5
60.06 ± 0.4
Table 4: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Uniform label noise. The best and second-best performances are highlighted with blod and underline , respectively.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
20%
40%
20%
40%
Baseline
ResNet-18 / 34
85.10 ± 0.6
77.00 ± 2.1
52.19 ± 1.4
42.26 ± 1.2
Co-teaching ( Han et al., 2018b )
(NeurIPS18)
ResNet-18 / 34
86.54 ± 0.1
79.98 ± 0.3
57.24 ± 0.6
45.69 ± 0.9
Peer loss ( Liu and Guo, 2020 )
(ICML20)
ResNet-18 / 34
88.19 ± 0.5
81.53 ± 0.7
63.82 ± 0.3
47.91 ± 0.5
CORES* ( Cheng et al., 2021 )
(ICLR21)
ResNet-18 / 34
89.67 ± 0.3
82.99 ± 0.5
64.86 ± 0.5
49.62 ± 0.7
WarPI ( Sun et al., 2022a )
(PR22)
ResNet-18 / 34
89.76 ± 0.4
87.57 ± 0.9
65.08 ± 0.6
57.38 ± 1.0
Table 5: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with Instance-dependent label noise. Note that “ResNet-18/34” denotes applying ResNet-18 for CIFAR-10 and ResNet-34 for CIFAR-100. The best and second-best performances are highlighted with blod and underline , respectively.
Clothing1M
Food-101N
MWNet ( Shu et al., 2019 )
73.72
DivideMix ( Li et al., 2020 )
74.76
Base Model
81.67
CNeth ( Lee et al., 2018 )
83.47
ELR ( Liu et al., 2020 )
74.81
CAL ( Zhu et al., 2021 )
74.17
MWNet ( Shu et al., 2019 )
84.72
SMP ( Han et al., 2019 )
85.11
PLC ( Zhang et al., 2021a )
73.24
WarPI ( Sun et al., 2022a )
74.98
NRank ( Sharma et al., 2020 )
85.20
ELR+ ( Liu et al., 2020 )
85.77
JNPL ( Kim et al., 2021 )
74.15
CoDis ( Xia et al., 2023 )
74.92
PLC ( Zhang et al., 2021a )
85.28
WarPI ( Sun et al., 2022a )
85.91
NCR ( Iscen et al., 2022 )
74.42
VRI (Ours)
75.19
CoDis ( Xia et al., 2023 )
86.13
VRI (Ours)
86.24
Table 6: Testing Accuracy (%) on real-world noise , including Clothing1M and Food-101N.
Methods
Unif 20%
Unif 50%
Unif 80%
Flip 40%
Cross-Entropy
-
42.57
27.06
9.27
22.25
Decoupling ( Malach and Shalev-Shwartz, 2017 )
NIPS 2017
43.49
-
10.01
33.74
Co-teaching ( Han et al., 2018b )
NIPS 2018
60.38
-
16.59
42.42
Co-teaching+ ( Yu et al., 2019 )
ICML 2019
53.97
-
12.29
43.01
JoCoR ( Wei et al., 2020 )
CVPR 2020
59.99
-
12.85
39.36
MoPro ( Li et al., 2021 )
ICLR 2021
65.60
-
30.29
60.22
Table 7: Average test accuracy on CIFAR80N, an open-set label noisy dataset, over the last 10 epochs.
Figure 3: (a) As the noise ratio rises, the rectification effect becomes more obvious since the area of the original loss increases. (b) We almost achieve the unbiased estimation for the initialized transition matrix of flip noise with varying noise ratio ρ .
Figure 4: The performance as the noise ratio increases is compared. VRI continues to deliver good performance even at significantly higher noise ratios.
Figure 5: Our algorithm achieves a stable convergence and displays robustness on flip noise with a high ratio (e.g., 60% ).
Figure 6: Model collapse of MC approximation.
Figure 7: MC method is more sensitive about the sample number compared with our proposed VRI.
k
Time (min./epoch)
Test Acc. (%)
1
2.17
88.23
MC
3
4.32
89.45
5
7.04
89.87
VRI
1
2.20
90.20
Table 8: VRI yields higher performance than MC approximation with an efficient inference.
Structure: {512+C^,h1,...,hn,C}
Accuracy
{512+C^,128,C}
89.76
{512+C^,512,512,C}
90.69
{512+C^,1024,512,C} (ours)
91.21
{512+C^,1024,1024,C}
90.89
{512+C^,1024,1024,512,512,C}
90.61
Table 9: Test accuracy (%) of different architectures of the meta-net on CIFAR-10 Flip 40%. C and C^ denote the number of classes and the dimension of the embedding vector of the label, respectively.
Homogeneous functions
Accuracy
Sigmoid (ours)
91.29
Tanh
88.17
w/o Sigmoid
86.90
Table 10: Test accuracy (%) of different activation functions in the last layer of the meta-net on CIFAR-10 Unif 40%. A ResNet-18 is used.
Noise
Method
Test Acc.
CIFAR-10
VRI
91.29
Unif. 40%
Non-Bayesian VRI
89.27
CIFAR-100
VRI
68.92
Unif. 40%
Non-Bayesian VRI
66.96
CIFAR-10
VRI
90.60
Inst. 40%
Non-Bayesian VRI
87.01
Table 11: Performance comparison (%) of VRI and its correspondingly non-Bayesian version.
Figure 8: (a) We obtain the best performance on with λ = 0.001. (b) The performance improves as the number of meta samples increases.
Dataset
CIFAR-10
CIFAR-100
Noise Ratio
Structure
40%
60%
40%
60%
Average gap
Uniform
Baseline
ResNet-18
68.07 ± 1.23
53.12 ± 3.03
51.11 ± 0.42
30.92 ± 0.33
26.0
VRI (-)
ResNet-18
80.23 ± 1.42
74.54 ± 2.46
59.39 ± 0.73
49.39 ± 0.46
10.1
VRI (+)
ResNet-18
91.24 ± 1.42
87.45 ± 2.16
66.39 ± 0.44
58.60 ± 0.56
1.0
VRI (Aug.)
ResNet-18
90.78 ± 1.56
87.98 ± 2.12
66.78 ± 0.68
58.79 ± 0.76
0.8
VRI (Standard)
ResNet-18
91.58 ± 0.17
88.68 ± 0.22
67.92 ± 0.19
59.32 ± 0.31
0
Table 12: Testing Accuracy (%) on CIFAR-10 and CIFAR-100 with uniform noise (top) and flip noise (bottom) when the accessibility of meta-data is restricted.
Baseline
Song et al. (2019)
Zhang et al. (2021a)
Chen et al. (2021)
Englesson (2021)
Chen et al. (2022)
VRI (-)
VRI (+)
79.4
81.8
83.4
84.1
84.2
84.5
81.4
85.8
Table 13: Testing accuracy (%) of VRI without given meta-data on ANIMAL-10N.
Most methods for learning with noisy labels require privileged knowledge such as noise transition matrices, clean subsets or pretrained feature extractors, resources typically unavailable when robustness is most needed. We propose Conformal Margin Risk Minimization (CMRM), a plug-and-play envelope framework that improves any classification loss under label noise by adding a single quantile-calibrated regularization term, with no privileged knowledge or training pipeline modification. CMRM measures the confidence margin between the observed label and competing labels, and thresholds it with a conformal quantile estimated per batch to focus training on high-margin samples while suppressing likely mislabeled ones. We derive a learning bound for CMRM under arbitrary label noise requiring only mild regularity of the margin distribution. Across five base methods and six benchmarks with synthetic and real-world noise, CMRM consistently improves accuracy (up to +3.39%), reduces conformal prediction set size (up to -20.44%) and does not hurt under 0% noise, showing that CMRM captures a method-agnostic uncertainty signal that existing mechanisms did not exploit.
Yuanjie Shi, Peihong Li, Zijian Zhang +2
School of EECS, Washington State University, Pullman, WA, USA
Training a deep neural network with noisy labels could reduce data annotation cost but may introduce noise into the learned model. In meta label correction approaches, an additional meta model besides the main model is trained with a small, clean dataset to correct the large, noisy dataset. However, the update of the meta model requires the computation of hypergradients at the inner step of the main model which signif- icantly increases the computational cost. To improve the training efficiency, we first introduce the dynamic barrier gradient descent into standard meta label correction. While this naive extenstion is able to speed up the training process to approximately first- order complexity, it lacks mechanisms to prevent the leakage of noisy signals to the main model and to stabilize the learning of the meta model. Based on this observation, we propose the EBOMLC method, which is designed with three key improvements including one-step inner loop update, mixture upper loss and alignment- aware dynamic barrier. Empirical results on CIFAR-10 and CIFAR-100 demonstrate that EBOMLC consistently outperforms other baselines, especially under high noise rate settings, while reducing training time of the meta label correction approach.
Ba Hoang Anh Nguyen, Viet Cuong Ta
Human-Machine Interaction Laboratory, VNU University of Engineering and Technology, Hanoi, Vietnam
Training deep neural networks with noisy labels remains a significant challenge, often leading to degraded performance. Existing methods for handling label noise typically rely on either transition matrix, noise detection, or meta-learning techniques, but they often exhibit low utilization efficiency of noisy samples and incur high computational costs. In this paper, we propose a self-training label correction framework using decoupled bilevel optimization, where a classifier and neural correction function co-evolve. Leveraging a small clean dataset, our method employs noisy posterior simulation and intermediate features to transfer ground-truth knowledge, forming a closed-loop feedback system that prevents error amplification. Theoretical guarantees underpin the stability of our approach, and extensive experiments on benchmark datasets like CIFAR and Clothing1M confirm state-of-the-art performance with reduced training time, highlighting its practical applicability for learning from noisy labels.