cs.LGSep 6, 2026

How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

Authors: Fangbo Li, Junpeng Zhang, Qihan Ren, Quanshi Zhang

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, i.e., model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.

Figures & tables

Figure 1: (a) It is proven [ 3 ] that the complex prediction logic can be mathematically decomposed into a small set of AND/OR interactions. (b) Some interactions are removed (shown in light color), while others are preserved (shown in dark color) after pruning. (c) When we progressively increase the pruning ratio, we find that the test accuracy does not decrease noticeably until the pruning operation begins to remove generalizable interactions.
Figure 2: Illustration of removed, preserved, and emerged interactions after pruning.
Figure 3: Changes in the distributions of removed interaction ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and the distributions of preserved interaction ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) when the pruning ratio increases. These changes can be divided into three phases. The four selected checkpoint pruning ratios correspond to the inflection points of the accuracy curve, and represent the boundaries of the three phases in the progressive parameter pruning process. Please see Appendix F.1 for results on more pruning ratios.
Figure 4: (left) Changes in the ratio of preserved low/high-order interactions ρlow\rho_{\textnormal{low}} / ρhigh\rho_{\textnormal{high}} and the classification accuracy when the pruning ratio increases. (right) Changes in the offsetting ratio of removed interactions κ\kappa and the classification accuracy when the pruning ratio increases.
Figure 5: (top) The ratio of preserved low/high-order interactions ρlow\rho_{\textnormal{low}} / ρhigh\rho_{\textnormal{high}} after pruning each module. (bottom) The offsetting ratio of removed interactions κ\kappa after pruning. After pruning sensitive modules, low-order interactions are significantly removed, and the removed interactions exhibit poorer offsetting ratio. Please see Appendix G.2 for more results.
Figure 6: The distributions of generalizable interactions (denoted by Igeneral(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{general}} and Igeneral(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{general}} ) and all interactions (denoted by I(m⁡),+{\bf I}^{(\operatorname{m}),+} and I(m⁡),−{\bf I}^{(\operatorname{m}),-} ) over different orders. Low-order interactions exhibit stronger generalizability than high-order ones. Please see Appendix H for results on more DNNs.
Figure 7: (left) The ratio of removed generalizable interactions ρremoved\rho_{\textnormal{removed}} after pruning each module. Pruning insensitive modules removes much less generalizable interactions than pruning sensitive modules. Please see Appendix G for results on more DNNs. (right) The ratio of removed generalizable interactions ρremoved\rho_{\textnormal{removed}} when the pruning ratio progressively increases. The removal of generalizable interactions leads to DNN performance degradation.
Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Top-5 classification accuracy of pruned VGG-16 and pruned AlexNet on TinyImageNet under different pruning ratio.
Modelτ\tau
LLaMA-7B0.60
LLaMA-3.1-8B1.00
DeepSeek-R1-Distill-LLaMA-8B0.70
Qwen-2.5-7B0.70
ResNet-500.50
ResNet-1010.45
Appendix
Table 1: Threshold τ\tau used for different DNNs.
Figure 9: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by pruned ResNet-56 at different pruning ratios. The distributions are arranged in ascending order of the pruning ratio.
Figure 10: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by pruned VGG-19 at different pruning ratios. The distributions are arranged in ascending order of the pruning ratio.
Figure 11: The ratio of preserved low-order/high-order interactions ρlow\rho_{\textnormal{low}} / ρhigh\rho_{\textnormal{high}} during progressive parameter pruning on different DNNs.
Figure 12: The offsetting ratio of removed interactions κ\kappa during progressive parameter pruning on different DNNs.
Figure 13: The ratio of removed generalizable interactions ρremoved\rho_{\textnormal{removed}} during progressive parameter pruning on different DNNs.
PerspectivesPhase 1Phase 2Phase 3
Testing accuracybarely affectedsignificantly decreasedremains low
Low-order interactionsslightly removedsignificantly removedmost have been removed
High-order interactionssignificantly removedsignificantly removedmost have been removed
Offsetting of the removedextremely highcomparatively lowcomparatively low
Generalizability of the removedslightly increasesignificantly increasemost have been removed
Appendix
Table 2: Analysis of three-phase dynamics of interactions from different perspectives.
Figure 14: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by LLaMA-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 15: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by LLaMA-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 16: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by LLaMA-3.1-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 17: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by LLaMA-3.1-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 18: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by DeepSeek-R1-Distill-LLaMA-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 19: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by DeepSeek-R1-Distill-LLaMA-8B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 20: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by Qwen-2.5-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 21: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by Qwen-2.5-7B after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 22: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-50 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 23: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-50 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 24: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-101 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 25: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-101 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 26: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-152 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 27: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ResNet-152 after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 28: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Small after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 29: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Small after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 30: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Base after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 31: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Base after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 32: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Large after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the first of two subsets of results.
Figure 33: The distributions of removed interactions ( Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} and Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} ) and preserved interactions ( Ipreserved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{preserved}} and Ipreserved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{preserved}} ) over different orders encoded by ViT-Large after pruning each sensitive/insensitive module. The distributions are arranged in ascending order of the increase in test loss after pruning. For clarity, this figure presents the second of two subsets of results.
Figure 34: The ratio of preserved low-order/high-order interactions ρlow\rho_{\textnormal{low}} / ρhigh\rho_{\textnormal{high}} after pruning sensitive/insensitive modules on different DNNs.
Figure 35: The offsetting ratio of removed interactions κ\kappa after pruning sensitive/insensitive modules on different DNNs.
Figure 36: The ratio of removed generalizable interactions ρremoved\rho_{\textnormal{removed}} after pruning sensitive/insensitive modules on different DNNs.
Figure 37: The distributions of interactions and generalizable interactions I(m⁡),+{\bf I}^{(\operatorname{m}),+} , I(m⁡),−{\bf I}^{(\operatorname{m}),-} , Igeneral(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{general}} and Igeneral(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{general}} encoded by twelve DNNs. Interactions of different orders exhibit different generalizability. Low-order interactions exhibit stronger generalizability than high-order ones.
Figure 38: The distributions of removed interactions Iremoved(m⁡),+{\bf I}^{(\operatorname{m}),+}_{\textnormal{removed}} , Iremoved(m⁡),−{\bf I}^{(\operatorname{m}),-}_{\textnormal{removed}} at the end of Phase 1. Interactions of different orders exhibit different generalizability. Removed interactions exhibit strong offsetting effects.
Figure 39: Sorted interaction strength distribution on different input samples. Most interactions exhibit negligible strength.
Figure 40: The logical model representing the inference logic of the DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “New York Department of Health recommends that all people should wear N95, KN95, or KF94 masks in all public.” The predicted next word is “settings.”
Figure 41: The logical model representing the inference logic of the Qwen-2.5-7B model on the input prompt “New York Department of Health recommends that all people should wear N95, KN95, or KF94 masks in all public.” The predicted next word is “settings.”
Figure 42: The logical model representing the inference logic of the DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “On June 11, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained.” The predicted next word is “transformer.”
Figure 43: The logical model representing the inference logic of the Qwen-2.5-7B model on the input prompt “On June 11, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained.” The predicted next word is “transformer.”
Figure 44: The logical model representing the inference logic of the original DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “Earth’s greenhouse effect would be too weak to keep temperature above freezing without.” The predicted next word is “carbon dioxide.”
Figure 45: The logical model representing the inference logic of the pruned DeepSeek-R1-Distill-LLaMA-8B model (under a pruning ratio of 41.01) on the input prompt “Earth’s greenhouse effect would be too weak to keep temperature above freezing without.” The predicted next word is “carbon dioxide.”
Figure 46: The logical model representing the inference logic of the original DeepSeek-R1-Distill-LLaMA-8B model on the input prompt “The government borrows money by selling financial products called.” The predicted next word is “bonds.”
Figure 47: The logical model representing the inference logic of the pruned DeepSeek-R1-Distill-LLaMA-8B model (under a pruning ratio of 41.01) on the input prompt “The government borrows money by selling financial products called.” The predicted next word is “bonds.”
More in this topic

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 289 papers

Explore similar work

CardsList
Jun 20, 2026

Data Pruning: Redundant, Problematic, and Interdependent Samples

The performance of deep learning models is affected by not only data quantity but also data quality. Data pruning is a process by which practitioners can reduce the size of a dataset by only keeping the most important training data points, thereby achieving similar test set performance. We empirically investigate two popular data pruning methods under noisy and noiseless conditions and show that these methods fail in the presence of significant label noise. We highlight that the success of data pruning is distinctly affected by three factors: redundancy in the dataset, the presence of problematic samples, and interdependence between samples. We perform a detailed investigation on commonly used benchmark classification datasets and neural network architectures. We find that our observations are consistent across data distributions and training protocols.

Leon Freese, Marthinus W. Theunissen

Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

Dataset PruningTraining Data Selection
AskPDF
Feb 24, 2026

Elimination-compensation pruning for fully-connected neural networks

The unmatched ability of Deep Neural Networks in capturing complex patterns in large and noisy datasets is often associated with their large hypothesis space, and consequently to the vast amount of parameters that characterize model architectures. Pruning techniques affirmed themselves as valid tools to extract sparse representations of neural networks parameters, carefully balancing between compression and preservation of information. However, a fundamental assumption behind pruning is that expendable weights should have small impact on the error of the network, while highly important weights should tend to have a larger influence on the inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias, efficiently computable by automatic differentiation. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios. Finally, our findings are discussed and the theoretical implications of our results are presented.

Enrico Ballini, Luca Muscarnera, Alessio Fumagalli +2

MOX, Department of Mathematics, Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, Italy

Neural Network PruningNeural Network Compression
AskPDF
Nov 28, 2024

Is Oracle Pruning the True Oracle?

Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has served as the foundation for most neural network pruning methods for over thirty-five years, while few (if any) have thought about how much the foundation really holds. This paper, for the first time, attempts to systematically examine its validity on deep neural networks through empirical correlation analyses and provides meta-framework reflections on the field of neural network pruning. Specifically, this paper focuses on the pruning algorithms with three stages: training, pruning, and retraining. We analyze the correlation in model performance before and after the retraining stage. Extensive experiments (37K models are trained) across a wide spectrum of models (LeNet5, VGG, ResNets, ViT, MLLM) and datasets (MNIST, CIFAR10/CIFAR100, ImageNet-1K, MLLM data) are conducted. For large-scale experiments, we adopt approximate oracle pruning due to the prohibitive cost of exact oracle pruning. The results point to a counterintuitive conclusion: for deep learning models of nontrivial size (already at the scale of ResNet56 on CIFAR-10), pre-retraining performance is negligibly correlated with post-retraining performance. In other words, the weights identified by oracle pruning can scarcely guarantee strong performance following retraining. This further suggests that existing works that derive pruning criteria from oracle pruning may rest on a questionable foundational premise. Further studies suggest that rising task complexity is a primary factor behind the invalidity of oracle pruning nowadays. Finally, given the evidence, we argue that the retraining stage in a pruning algorithm should be accounted for when developing pruning criteria.

Sicheng Feng, Keda Tao, Huan Wang

Westlake University, Hangzhou, China · Nankai University, Tianjin, China

Neural Network Pruning
AskPDF