Bayesian neural networks (BNNs) are a useful tool for uncertainty quantification, but require substantially more computational resources than conventional neural networks. For non-Bayesian networks, the Lottery Ticket Hypothesis (LTH) posits the existence of sparse subnetworks that can train to the same or even surpassing accuracy as the original dense network. Such sparse networks can lower the demand for computational resources at inference, and during training. The existence of the LTH and corresponding sparse subnetworks in BNNs could motivate the development of sparse training algorithms and provide valuable insights into the underlying training process. Towards this end, we translate the LTH experiments to a Bayesian setting using common computer vision models. We investigate the defining characteristics of Bayesian lottery tickets, and extend our study towards a transplantation method connecting BNNs with deterministic Lottery Tickets. We generally find that the LTH holds in BNNs, and winning tickets of matching and surpassing accuracy are present independent of model size, with degradation at very high sparsities. However, the pruning strategy should rely primarily on magnitude, secondly on standard deviation. Furthermore, our results demonstrate that models rely on mask structure and weight initialization to varying degrees.
Figures & tables
Figure 1: Test accuracy and MACE vs. percentage of weights remaining for ResNet18, VGG11 and ViT-tiny models trained on CIFAR10. Shown in color are the different scoring functions for pruning; in black are the non-Bayesian models pruned by magnitude. The x-axis is shown on a logarithmic scale; the y-axis is adjusted to include the full range of observed accuracies.
Figure 2: Layer-wise sparsity vs. layer index for ResNet (left), VGG (center), and ViT (right) training and pruning 20 times. The x-axis denotes the layer index, and the y-axis shows the resulting sparsity in each layer. Each plot shows the non-Bayesian model and the Bayesian model.
Figure 3: Visualization of the different randomizations applied to the weights (magnitude symbolized through color) and masks (binary, i.e. black and white) of a winning ticket.
Figure 4: Test accuracy vs. percentage of weights remaining (log scale) for ResNet (left), VGG (middle), and ViT (right) trained on CIFAR10. The black line denotes the original lottery ticket obtained through IMP, the colored lines represent the accuracy achieved by reinitializing or shuffling and training the ticket of that sparsity level. The Bayesian and non-Bayesian models are shown in separate plots.
Figure 5: Test accuracy vs. percentage of weights remaining (log scale) for ResNet (left), VGG (middle), and ViT (right) trained on CIFAR10. Shown are LTs generated through deterministic IMP, a fully Bayesian approach, and the transplantation method.
ResNet20
VGG11
ViT-Tiny
Deterministic
12
13
9
Bayesian
33
72
51
Table 1: Average runtime for a singular training level of models trained in Section 5 in minutes.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Test accuracy vs. Percentage of weights remaining (log scale) for ResNet, VGG, and ViT of different sizes trained on CIFAR10. Shown are non-Bayesian and Bayesian models.
Figure 7: Test accuracy vs. Percentage of weights remaining (log scale) for ResNet, VGG, and ViT trained on CIFAR10. The colored lines represent different initial learning rates. Included as a subplot are the equivalent non-Bayesian models.
Figure 8: Test accuracy vs. percentage of weights remaining (log scale) for ResNet, VGG, and ViT trained on CIFAR10. The lines represent tickets obtained through either Learning Rate Rewinding or Iterative Magnitude Pruning. Shown are non-Bayesian and Bayesian models.
Figure 9: MACE vs. percentage of weights remaining for transplanted LTs (see Section 8 ) and fully Bayesian LTs.