Authors: Joshua Offergeld, John Chidiac, Marcel van Gerven, Nasir Ahmad
Organizations: Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands
Node pruning is the art of removing computational units such as neurons, filters, attention heads, or even entire layers to significantly reduce inference time while retaining network performance. In this work, we show that a broad family of importance measures, from activation-based to Fisher-based, share a common quadratic form. Through a lower-triangular change of basis, this form yields a pruning subspace in which entire nodes can be removed while the impact of the lost units is simultaneously recovered via linear least squares. Furthermore, the importance measures within our subspace are decoupled and additive through an orthogonalization of the layer activations, allowing multi-node pruning within layers without interference between importance scores. Our method, Subspace Node Pruning (SNP), outperforms the compared baselines on VGG-16 and ResNet-50 on ImageNet. On OPT-125M and LLaMA-2 7B, SNP matches the state-of-the-art LLM Surgeon at a 40-shot schedule and outperforms it whenever fewer shots are used, all without any retraining after pruning.
Figures & tables
Squared input activation ( Hu et al., 2016 ; Berisha et al., 2025 )
Al(k) : I
Il : ∑kxl(k)⊤xl(k)
Il,i : ∑k(xl,i(k))2
Squared (layer) output activation ( He et al., 2017 ; Luo et al., 2017 ; Sun et al., 2023 )
Al(k) : Wl
Il : ∑kxl(k)⊤Wl⊤Wlxl(k)
Il,i : ∥wl,i∥2∑k(xl,i(k))2
First-order Taylor ( Molchanov et al., 2016 ; Theis et al., 2018 ; Molchanov et al., 2019 )
Al(k) : gl(k)⊤
Il : ∑k(gl(k)⊤xl(k))2
Il,i : ∑k(gl,i(k)xl,i(k))2
Second-order Taylor / Fisher
Table 1: Common importance measures expressed in terms of the activities at layer l , transformed by a sample-wise sensitivity matrix Al(k) . The per-unit importance Il,i is the contribution of unit i to Il , excluding cross terms between units. Here gl(k)=∂L/∂xl(k) is the gradient of the loss with respect to the unit activities, and Hl(k) is a curvature estimate ( Martens, 2020 ) of the loss with respect to the layer’s output yl+1(k) : the Gauss–Newton Hessian, the true Fisher, the empirical Fisher δl(k)δl(k)⊤ with δl(k)=∂L/∂yl+1(k) , or its sample average Hˉl as in K-FAC ( Martens and Grosse, 2015 ) . Since gl(k)=Wl⊤δl(k) , the empirical Fisher recovers the first-order measure up to a factor of \ifrac12 . The per-unit importance Il,i is the contribution of unit i to the layer-wide importance Il , excluding cross terms between units, where xl,i(k) and gl,i(k) are the i -th elements of xl(k) and gl(k) , and wl,i is the i -th column of Wl . Citations indicate methods using these measures or close variants thereof.
Figure 1: Why the subspace must be constructed with a lower-triangular matrix. Left: with a dense Ml , removing latent units only lowers the rank of the layer (number of nodes in the latent space), and every original input unit remains in use. Right: with a lower-triangular Ml , removing the bottom row of Ml removes the last original unit.
Figure 2: Node ordering and pruning performance on VGG-16. (a) Subspace importances I^l,j of layer 12, for units ordered randomly (left), by the summed absolute weights (SAW) of Li et al. (2016) (middle), and by our score Il,i⊥ (right). Units to the right are pruned first. (b) Accuracy over a range of pruning ratios for our method (SNP) and baseline methods (Appendix A ). SNP-Act uses the squared input activation with the I⊥ ordering, and SNP-Fisher is SNP with the empirical Fisher. PFA-EN is the only baseline that uses PCA to determine global importance.
Figure 3: SNP vs. Intra-Fusion for reconstruction. A comparison of pruning different groups of layers of ResNet-50. We compare our SNP method with the squared input activation measure against Intra-Fusion (IF). SNP 1024 and SNP wn estimate Cl from 1024 images and from white noise inputs, respectively, to show robustness to the samples used. SNP’s LLS reconstruction outperforms IntraFusions Optimal Transport approach over the range tested.
OPT-125M
LLaMA-2 7B
Shots
Sparsity
SNP
LLM Surgeon
SNP
LLM Surgeon
Dense
27.65
5.12
1
10%
28.48
29.30
5.89
7237
30%
55.13
3357
19.14
4791
50%
579.72
4258
241.77
13169
5
10%
27.85
28.97
5.49
5.27
Table 2: Pruning LLMs without retraining. Perplexity on WikiText-2 after structured pruning of rows and columns with SNP and LLM Surgeon ( van der Ouderaa et al., 2024 ) , both with the empirical-Fisher K-FAC curvature, for different numbers of shots and sparsities (fraction of parameters removed). Lower is better; the better method in each pair is in bold. Results on OPT-125M for all numbers of shots are given in Table 4 .
Figure 4: True Loss Changes vs Loss Predictions of SNP vs LLM Surgeon. These plots show the loss change induced by pruning units (more units pruned from right to left) using the SNP with empirical Fisher vs LLM Surgeon (LS) method. These loss magnitude changes are shown for the middle block of an OPT-125M model and for all of its linear layers including two fully connected (fc) layers of the MLP, an output projection layer (out_proj), as well as the Q, K, and V matrices.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
VGG-16
ResNet-50
OPT-125M
LLaMA-2 7B
Pretrained weights
torchvision
torchvision
Hugging Face
Hugging Face
Calibration data
ImageNet train
ImageNet train
WikiText-2 train
WikiText-2 train
Calibration samples
Images
Images
128 sequences
128 sequences
Sequence length
–
–
2048 tokens
2048 tokens
Evaluation data
ImageNet val
ImageNet val
WikiText-2 test
WikiText-2 test
Shots T
1
1
up to 40
up to 40
Appendix
Table 3: Hyperparameters for all experiments. For the language models, the damping follows the LLM Surgeon defaults ( van der Ouderaa et al., 2024 ) , with λC and λH added to Cl and Hˉl relative to their diagonals.
10%
20%
30%
40%
50%
Shots
SNP
LS
SNP
LS
SNP
LS
SNP
LS
SNP
LS
1
28.48
29.30
34.95
221.06
55.13
3357
132.01
3875
579.72
4258
2
27.93
28.70
30.01
32.01
34.89
44.59
45.13
270.25
66.19
2699
3
27.87
30.15
29.08
31.33
34.14
34.97
43.21
50.92
63.08
106.89
4
27.98
28.31
29.64
30.51
34.19
37.13
41.40
45.07
56.47
69.32
5
27.85
28.97
29.07
31.50
32.41
33.54
41.06
45.05
54.21
59.79
Appendix
Table 4: Perplexity on WikiText-2 of OPT-125M (dense: 27.65) after structured pruning of rows and columns with SNP and LLM Surgeon (LS), for all numbers of shots and sparsities, extending Table 2 . Lower is better; the better method in each pair is in bold.
OPT-125M
SNP
Sparsity
5 shots
40 shots
SliceGPT †
K-OBD †
LS †
Dense
27.65
27.64
27.65
27.65
10%
27.85
27.87
29.33 (1)
27.97 (8)
28.01 (8)
20%
29.07
28.62
34.53 (1)
29.89 (16)
28.73 (16)
30%
32.41
31.74
44.61 (1)
36.54 (24)
31.82 (24)
Appendix
Table 5: Perplexity on WikiText-2 of SNP with 5 and 40 shots, compared with results reported for SliceGPT ( Ashkboos et al., 2024 ) , and for K-OBD and LLM Surgeon (LS) by van der Ouderaa et al. (2024) , on OPT-125M (top) and LLaMA-2 7B (bottom). Lower is better; the best value in each row is in bold, and – marks settings not evaluated. † Values in italics are reported by other works. Numbers in parentheses give the number of shots used by each reported method: SliceGPT is single-shot, and K-OBD and LS are reported from a single run with a linear 40-shot schedule to 50% sparsity, reaching lower sparsities after fewer shots. Only at 50% sparsity do the reported K-OBD and LS results use the same number of shots as SNP with 40 shots; all other comparisons are indicative only.
Figure 5: Predicted and true loss change when pruning units of all blocks of OPT-125M. Rows correspond to blocks 7-12 and columns to the linear layers of each block (fc1, fc2, out_proj, and the Q, K and V projections). Units are pruned from right to left in a single shot; curves show the true change in loss and the change predicted by SNP-Fisher and by LLM Surgeon. As in the main text, the SNP-Fisher prediction stays close to the true loss as more units are pruned, whereas LLM Surgeon’s prediction diverges.
Figure 6: Predicted and true loss change when pruning units of all blocks of OPT-125M. Rows correspond to blocks 1-6 and columns to the linear layers of each block (fc1, fc2, out_proj, and the Q, K and V projections). Units are pruned from right to left in a single shot; curves show the true change in loss and the change predicted by SNP-Fisher and by LLM Surgeon. As in the main text, the SNP-Fisher prediction stays close to the true loss as more units are pruned, whereas LLM Surgeon’s prediction diverges.
Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel L0 method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.
Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca
Weight pruning is a common technique for compressing large neural networks. We focus on the challenging post-training one-shot setting, where a pre-trained model is compressed without any retraining. Existing one-shot pruning methods typically optimize a single objective, such as a layer-wise reconstruction loss or a second-order Taylor approximation of the training loss. We highlight that neither objective alone is consistently the most effective across architectures and sparsity levels. Motivated by this insight, we propose MOONSHOT, a general and flexible framework that extends any single-objective pruning method into a multi-objective formulation by jointly optimizing both the layer-wise reconstruction error and second-order Taylor approximation of the training loss. MOONSHOT acts as a wrapper around existing pruning algorithms. To enable this integration while maintaining scalability to billion-parameter models, we propose modeling decisions and introduce an efficient procedure for computing the inverse Hessian, preserving the efficiency of state-of-the-art one-shot pruners. When combined with state-of-the-art pruning methods on Llama-3.2 and Llama-2 models, MOONSHOT reduces C4 perplexity by up to 32.6% at 2:4 sparsity and improves zero-shot mean accuracy across seven classification benchmarks by up to 4.9 points. On Vision Transformers, it improves accuracy on ImageNet-1k by over 5 points at 70% sparsity, and on ResNet-50, it yields a 4-point gain at 90% sparsity.
Gabriel Afriat, Xiang Meng, Shibal Ibrahim +2
Operations Research Center Massachusetts Institute of Technology · Google · OpenAI +1
One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to ±5% around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, p<0.001). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.