Over-parameterized neural networks incur prohibitive memory and computational costs for resource-constrained deployment. The Strong Lottery Ticket (SLT) hypothesis suggests that randomly initialized networks contain sparse subnetworks achieving competitive accuracy without weight training. Existing SLT methods, notably edge-popup, rely on non-differentiable score-based selection, limiting optimization efficiency and scalability. We propose using continuously relaxed Bernoulli gates to discover SLTs through fully differentiable, end-to-end optimization - training only gating parameters while keeping all network weights frozen at their initialized values. Continuous relaxation enables direct gradient-based optimization of an ℓ0-regularization objective, eliminating the need for non-differentiable gradient estimators or iterative pruning cycles. To our knowledge, this is the first fully differentiable approach for SLT discovery that avoids straight-through estimator approximations. Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss - nearly double the sparsity achieved by edge-popup at comparable accuracy - establishing a scalable framework for pre-training network sparsification.
Figures & tables
Fig. 1: Pre-training sparsification on LeNet-300-100. Blue region: percentage of pruned weights. Green region: retained weights. Red line: test accuracy on MNIST as sparsification progresses. The method achieves 96% accuracy at 45% sparsification.
Method
Description
Architecture
Accuracy
Weights Trained
Sparsity
Weak Lottery Ticket Methods (weights trained)
Baseline [ 31 ]
Standard training with ℓ2 loss
784-300-100
96.95%
Yes
0%
Sparse VD [ 33 ]
Variational dropout sparsification
512-114-72
98.2%
Yes
97.8%
BC-GNJ [ 34 ]
Bayesian compression with GNJ prior
278-98-13
98.2%
Yes
89.2%
BC-GHS [ 28 ]
Bayesian compression with GHS prior
311-86-14
98.2%
Yes
–
Strong Lottery Ticket Methods (weights frozen)
TABLE I: Comparison of pruning methods on MNIST. Methods are grouped by whether weights are trained (weak LT) or frozen (strong LT). Our method achieves the highest accuracy among SLT methods while using a smaller base network than [ 32 ] .
Method
Architecture
Accuracy
Sparsity
Edge-popup [ 18 ]
Wide-ResNet50
88%
50%
Ours (CRBG)
Wide-ResNet50
88%
90.5%
Ours (CRBG)
ResNet50
83.1%
91.5%
TABLE II: CNN sparsification on CIFAR-10. Our method matches edge-popup accuracy while achieving significantly higher sparsification.
Fig. 2: Per-layer sparsification of ResNet50 on CIFAR-10. Later layers exhibit higher sparsification rates, consistent with prior findings that early layers require more weights for low-level feature extraction.
Method
Architecture
Accuracy
Sparsity
Weights Trained
Fully trained
Swin-T
87.26%
0%
Yes
[ 37 ]
ViT-base
85.09%
30%
Yes (after pruning)
Ours (CRBG)
ViT-base
76%
90%
No
Ours (CRBG)
Swin-T
80%
50%
No
TABLE III: Transformer sparsification on CIFAR-10.
Fig. 3: Robustness to base network size (LeNet on MNIST). Blue: pruned weights. Green: retained weights. Red: test accuracy. SLTs can be discovered even in base networks at 20% of the original size.
Bayesian neural networks (BNNs) are a useful tool for uncertainty quantification, but require substantially more computational resources than conventional neural networks. For non-Bayesian networks, the Lottery Ticket Hypothesis (LTH) posits the existence of sparse subnetworks that can train to the same or even surpassing accuracy as the original dense network. Such sparse networks can lower the demand for computational resources at inference, and during training. The existence of the LTH and corresponding sparse subnetworks in BNNs could motivate the development of sparse training algorithms and provide valuable insights into the underlying training process. Towards this end, we translate the LTH experiments to a Bayesian setting using common computer vision models. We investigate the defining characteristics of Bayesian lottery tickets, and extend our study towards a transplantation method connecting BNNs with deterministic Lottery Tickets. We generally find that the LTH holds in BNNs, and winning tickets of matching and surpassing accuracy are present independent of model size, with degradation at very high sparsities. However, the pruning strategy should rely primarily on magnitude, secondly on standard deviation. Furthermore, our results demonstrate that models rely on mask structure and weight initialization to varying degrees.
Nicholas Kuhn, Arvid Weyrauch, Lars Heyen +3
Scientific Computing Center (SCC), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany · Helmholtz AI, Eggenstein-Leopoldshafen, Germany
Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles. This work evaluates progressive magnitude-based pruning as a single-cycle alternative. The method gradually increases sparsity during training using a linear schedule and updates pruning masks based on active weight magnitudes. We conduct systematic experiments on CIFAR-10 and MNIST across ResNet, VGG-style, and LeNet architectures, comparing the proposed method with representative iterative and initialization-based pruning baselines, including LTH, SNIP, and GraSP. On CIFAR-10, the method achieves 95.12% accuracy on ResNet-18 at 72.9% sparsity, compared with 90.5% reported for LTH. At extreme sparsity, it achieves 93.13% accuracy on a VGG-like architecture at 97% sparsity, compared with approximately 92.0% for SNIP, and 93.44% accuracy on VGG-19 at 97.97% sparsity, compared with 92.19% for GraSP at 98% sparsity. A sparsity-accuracy analysis on ResNet-18 further shows that accuracy remains within 0.1 percentage points of the dense baseline across 70--85% sparsity. These results indicate that progressive magnitude-based pruning provides an effective single-cycle approach for neural network sparsification under the evaluated settings.
Romana Qureshi, Hafida Benhidour, Said Kerrache +1
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia
The Strong Lottery Ticket Hypothesis (SLTH) asserts that sufficiently overparameterized, randomly initialized neural networks contain sparse subnetworks that, even without any training, can match the performance of a small trained network on a given dataset. A key mathematical tool in the theoretical study of SLTH has been the Random Subset Sum Problem (RSSP). The SLTH has recently been extended to the quantized setting, where the network weights are sampled from a discrete set rather than from a continuous interval. These new results are however far from those in arbitrary-precision setting in several ways. In this work, we provide an analysis of the RSSP in the discrete setting, and use it to derive tight SLTH guarantees in the quantized case. Our analysis obtain tight bounds on the failure probability of finding a strong lottery ticket in the quantized regime, providing an exponential improvement over previous results. Most importantly, it unifies the literature by showing that both approximate representations in the continuous setting and exact representations in quantized settings naturally emerge as limiting cases of our results. This perspective not only sharpens existing bounds but also provides a cohesive framework that simultaneously handles approximation and rounding errors.
Aakash Kumar, Emanuele Natale
COATI, CNRS, Inria, I3S, Université Coté d’Azur, France