Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
Authors: Leonid Berlyand, Theo Bourdais, Houman Owhadi, Yitzchak Shmalo
Abstract
We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component R has small propagated logit effect Ls∥Rψ1(s)∥∞, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive L2-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge σ+ gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16 2:4+ToMe reaches 83.41% top-1 (−1.70 pp from dense) at 59.81% sparse-execution MAC reduction, with 1.388× best-observed A40 native-2:4 backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives 2.705×. At structured sparsity, ViT-B/16 6:12 reaches 83.74%, ViT-L/16 8:16 dense+permutation reaches 85.33% (−0.51 pp), and ConvNeXtV2-Base 12:16 reaches 86.35% (−0.37 pp). For CNNs, ResNet50 8:16 dense+permutation reaches 75.87% (−0.26 pp), and ResNet152d CAST-conv+permutation reaches 81.33% (−1.53 pp) at ∼50% MAC accounting with a 1.62× A40 im2col+2:4 sparse-GEMM audit.
Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles. This work evaluates progressive magnitude-based pruning as a single-cycle alternative. The method gradually increases sparsity during training using a linear schedule and updates pruning masks based on active weight magnitudes. We conduct systematic experiments on CIFAR-10 and MNIST across ResNet, VGG-style, and LeNet architectures, comparing the proposed method with representative iterative and initialization-based pruning baselines, including LTH, SNIP, and GraSP. On CIFAR-10, the method achieves 95.12% accuracy on ResNet-18 at 72.9% sparsity, compared with 90.5% reported for LTH. At extreme sparsity, it achieves 93.13% accuracy on a VGG-like architecture at 97% sparsity, compared with approximately 92.0% for SNIP, and 93.44% accuracy on VGG-19 at 97.97% sparsity, compared with 92.19% for GraSP at 98% sparsity. A sparsity-accuracy analysis on ResNet-18 further shows that accuracy remains within 0.1 percentage points of the dense baseline across 70--85% sparsity. These results indicate that progressive magnitude-based pruning provides an effective single-cycle approach for neural network sparsification under the evaluated settings.
Romana Qureshi, Hafida Benhidour, Said Kerrache +1
Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model's accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, STARFISH, that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network's internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing 50% of the weights, STARFISH healing improves the recovered accuracy by up to 22% over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating 75% of the weights in a DeiT-B network for ImageNet, STARFISH uses only 0.4% of the number of training images as a calibration set and recovers 82% of the original dense accuracy, whereas competing recovery techniques reach only 40% of the dense model accuracy.
Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel L0 method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.
Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu