cs.LGMay 26, 2024

Subspace Node Pruning

Authors: Joshua Offergeld, John Chidiac, Marcel van Gerven, Nasir Ahmad

Organizations: Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands

Abstract

Node pruning is the art of removing computational units such as neurons, filters, attention heads, or even entire layers to significantly reduce inference time while retaining network performance. In this work, we show that a broad family of importance measures, from activation-based to Fisher-based, share a common quadratic form. Through a lower-triangular change of basis, this form yields a pruning subspace in which entire nodes can be removed while the impact of the lost units is simultaneously recovered via linear least squares. Furthermore, the importance measures within our subspace are decoupled and additive through an orthogonalization of the layer activations, allowing multi-node pruning within layers without interference between importance scores. Our method, Subspace Node Pruning (SNP), outperforms the compared baselines on VGG-16 and ResNet-50 on ImageNet. On OPT-125M and LLaMA-2 7B, SNP matches the state-of-the-art LLM Surgeon at a 40-shot schedule and outperforms it whenever fewer shots are used, all without any retraining after pruning.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hyperflux: Pruning Reveals Importance

    Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural NetworksNeural Network Training Dynamics

  2. MOONSHOT : A Framework for Multi-Objective Pruning of Vision and Large Language Models

    Apr 14, 2026Gabriel Afriat, Xiang Meng, Shibal Ibrahim +2LLM PruningNeural Network Pruning

  3. PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

    Jul 8, 2026Yazdan Jamshidi, Alexey ShvetsLLM PruningLLM Compression