cs.LGNov 28, 2024

Is Oracle Pruning the True Oracle?

Authors: Sicheng Feng, Keda Tao, Huan Wang

Organizations: Westlake University, Hangzhou, China · Nankai University, Tianjin, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has served as the foundation for most neural network pruning methods for over thirty-five years, while few (if any) have thought about how much the foundation really holds. This paper, for the first time, attempts to systematically examine its validity on deep neural networks through empirical correlation analyses and provides meta-framework reflections on the field of neural network pruning. Specifically, this paper focuses on the pruning algorithms with three stages: training, pruning, and retraining. We analyze the correlation in model performance before and after the retraining stage. Extensive experiments (37K models are trained) across a wide spectrum of models (LeNet5, VGG, ResNets, ViT, MLLM) and datasets (MNIST, CIFAR10/CIFAR100, ImageNet-1K, MLLM data) are conducted. For large-scale experiments, we adopt approximate oracle pruning due to the prohibitive cost of exact oracle pruning. The results point to a counterintuitive conclusion: for deep learning models of nontrivial size (already at the scale of ResNet56 on CIFAR-10), pre-retraining performance is negligibly correlated with post-retraining performance. In other words, the weights identified by oracle pruning can scarcely guarantee strong performance following retraining. This further suggests that existing works that derive pruning criteria from oracle pruning may rest on a questionable foundational premise. Further studies suggest that rising task complexity is a primary factor behind the invalidity of oracle pruning nowadays. Finally, given the evidence, we argue that the retraining stage in a pruning algorithm should be accounted for when developing pruning criteria.

Figures & tables

Figure 1: Analysis framework of this work. We study the validity of oracle pruning in this paper, by examining the correlation between the pruned train loss and the final test performance ( i.e. , test accuracy or test loss). We apply this analysis framework to a wide range of networks and datasets (from toy networks like LeNet5-Mini to large ones like TinyLLaVA-3.1B) in order to have a comprehensive evaluation. The key finding of this work is that, to our surprise, oracle pruning becomes invalid on modern networks and datasets (starting from the CIFAR level), challenging the conventional belief in network pruning for the past 35 years.
Network ( Dataset)KendallAnomalyCounterexample
correlationratioratio
LeNet5-Mini ( MNIST)✓✓✗
ResNet56 ( CIFAR10)✓✓✗
VGG19 ( CIFAR100)✓✓✗
ResNet18 ( ImageNet-1K)✓✓✗
ViT-B/16 ( ImageNet-1K)✓✗✓
Table 1: Summary of the analysis methods for checking the validity of oracle pruning on different networks and datasets. LeNet5-Mini is a small enough network to achieve oracle pruning exactly; for other networks, we randomly sample enough models to analyze Kendall correlation, anomaly ratio, and counterexample ratio.
Figure 2: Pruned train loss vs. final test accuracy on MNIST with LeNet5-Mini. The subcaptions correspond to the pruning settings. The blue star indicates the reference oracle pruning result ( i.e. , the one with the smallest pruned train loss). The points with final test accuracy higher than the reference oracle pruning result are marked in red (anomaly points), and those lower are marked in green.
Pr.Conv1Conv2Conv3
0.2-0.60 / 4.9e-09-0.76 / 2.0e-13-0.71 / 7.0e-12
0.3-0.51 / 1.3e-16-0.61 / 4.8e-23-0.59 / 9.8e-22
0.4-0.48 / 1.3e-24-0.61 / 7.4e-39-0.58 / 2.2e-35
0.5-0.51 / 2.1e-33-0.51 / 1.8e-33-0.60 / 2.5e-45
0.6-0.41 / 6.1e-19-0.42 / 3.5e-19-0.47 / 1.2e-23
0.7-0.19 / 1.8e-03-0.44 / 1.4e-12-0.19 / 2.1e-03
Table 2: Kendall correlation between pruned train loss and final test accuracy, by exhaustively pruning LeNet5-Mini network on MNIST dataset. Each entry in the table is arranged as Kendall coefficient / p-value. Pr. means the pruning ratio for the corresponding layer combination of Conv1, Conv2, and Conv3. The red indicates the results where oracle pruning is invalid as defined in Sec. 3.1 .
Figure 3: Pruned train loss vs. final test accuracy with ResNet56, VGG19, and ResNet18. The subcaptions correspond to the pruning settings.
Method#ParamsPruned train lossFinal test performance
Dense model3.1B/77.15
ONP2B1.3576.70
UMP2B1.2476.75
GMP2B1.1776.30
PNP2B1.1676.28
Table 3: Pruning results on TinyLLaVA-3.1B. The final test performance is averaged on 5 benchmark datasets following standard practices (see Tab. A6 in the Appendix C for the detailed results).
Figure 4: Pruned train loss vs. final test accuracy on the variants of MNIST dataset, with LeNet5-Mini network (pruning ratio 0.5, Conv1 layer). FMNIST and KMNIST are two drop-in replacements of MNIST, which are more complex. As seen, the correlation becomes weaker on more challenging datasets.
Figure 5: Pruned train loss vs. final test accuracy on MNIST with different variants of LeNet5-Mini (pruning ratio 0.5, Conv1 layer). The original LeNet5-Mini (Base) has 5 layers (D5), and each layer has 10 neurons (W10). Here, we change the model width and depth to obtain different variants. As seen, the correlation becomes weaker when pruning more complex networks.
Figure 6: Pruned train loss vs. final test loss on MNIST with different variants of LeNet5-Mini (pruning ratio 0.5, Conv1 layer). The original LeNet5-Mini (Base) has 5 layers (D5), and each layer has 10 neurons (W10). Here, we change the model width and depth to obtain different variants. As seen, the correlation becomes weaker when pruning more complex networks.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Network & DataSolverLR policyTotal epochBatch size
  (pre-train and retrain)  
LeNet5-MiniSGD (0.9, 1e-4)Multi-step (0:1e-2, 20:1e-3)30256
(MNIST) Multi-step (0:1e-3, 20:1e-4)30256
ResNet56SGD (0.9, 5e-4)Multi-step (0:1e-1, 100:1e-2, 150:1e-3)200128
(CIFAR10) Multi-step (0:1e-2, 60:1e-3, 90:1e-4)120256
VGG19SGD (0.9, 5e-4)Multi-step (0:1e-1, 100:1e-2, 150:1e-3)200256
Appendix
Table A1 : Training setting summary. For the solver, the momentum and weight decay are in brackets. For CIFAR10, batch size 256 is used for retraining instead of 128, which is for saving training time. For LR policy, total epoch, and batch size, the first one is for the pre-training stage, the second is for the retraining stage.
DatasetNetworkPruning ratio
MNISTLeNet5-Mini[0.2-0.8, 0, 0]
  [0, 0.2-0.8, 0]
  [0, 0, 0.2-0.8]
  [0, 0.5, 0.5]
  [0.5, 0, 0.5]
  [0.5, 0.5, 0]
Appendix
Table A2 : Pruning ratio summary. See more detailed statements in Appendix A.3 .
IterationPruned Train Loss vs Final Test LossPruned Train Loss vs Final Test Acc.
LR1 (first four iterations: 0.001; final stage: 0:0.1, 20:0.01, 40:0.001)
1/50.05 / 0.310.03 / 0.58
2/5-0.05 / 0.320.04 / 0.47
3/50.02 / 0.710.04 / 0.41
4/50.10 / 0.060.08 / 0.13
5/50.05 / 0.340.03 / 0.52
Appendix
Table A3 : Kendall correlation between pruned train loss at each iterative pruning step and final test performance on ResNet50 with CIFAR100. Each entry in the table is arranged as coefficient / p-value.
Pr.Conv1Conv2Conv3
0.20.57 / 3.1e-080.74 / 6.0e-130.66 / 1.4e-10
0.30.53 / 7.3e-180.67 / 2.9e-270.55 / 3.5e-19
0.40.48 / 1.2e-240.65 / 1.8e-440.54 / 7.3e-32
0.50.54 / 2.6e-370.56 / 1.5e-400.57 / 4.8e-42
0.60.45 / 2.8e-220.44 / 5.5e-210.46 / 3.5e-23
0.70.20 / 1.0e-030.45 / 2.4e-130.20 / 1.1e-03
Appendix
Table A4 : Kendall correlation between pruned train loss and final test loss, by exhaustively pruning LeNet5-Mini network on MNIST dataset. Each entry in the table is arranged as Kendall coefficient / p-value. Pr. means the pruning ratio for the corresponding layer combination of Conv1, Conv2, and Conv3. The red entries mean these results pose weak or counterintuitive correlations.
Figure A1 : Pruned train loss vs. final test loss on MNIST with LeNet5-Mini. The subcaptions correspond to the pruning rates of each image. The star denotes the oracle pruning results, where points with final test loss lower than the oracle pruning are marked in red, and those lower are marked in green.
Figure A2 : Pruned train loss vs. final test accuracy on MNIST with LeNet5-Mini.
Figure A3 : Pruned train loss vs. final test loss on MNIST with LeNet5-Mini.
CombinationPruned train lossFinal test accuracyFinal test loss
16.976876.24451.9714
27.051575.97311.9781
37.044276.06621.9826
47.016276.32231.9689
56.998974.07482.0530
67.070975.70661.9901
Appendix
Table A5 : Results of ViT-B/16 on ImageNet-1K.
Figure A4 : Pruned train loss vs. final test loss with ResNet56 (on CIFAR10), VGG19 (on CIFAR100), ResNet18 (on ImageNet-1K).
Methods#ParamsVision-EncoderRes.PTLSQATextVQAGQAMM-VetPOPEAvg.
Dense model3.1BSigLIP-0.4B384/69.159.1623286.477.15
UMP2BSigLIP-0.4B3841.2469.456.760.134.286.676.75
GMP2BSigLIP-0.4B3841.1769.955.860.13386.476.30
ONP2BSigLIP-0.4B3841.3569.855.761.53386.876.70
PNP2BSigLIP-0.4B3841.1669.555.161.533.18676.28
Appendix
Table A6 : For the pruning results of MLLMs, the table presents the performance of the Base model and the models pruned using the four strategies. We present the model’s performance across five benchmarks. PTL stands for the pruned train loss.
Figure A5 : Pruned train loss vs. final test loss on the variants of MNIST dataset, with LeNet5-Mini network (pruning ratio 0.5, Conv1 layer). FMNIST and KMNIST are two drop-in replacements of MNIST, which are more complex. As seen, the correlation becomes weaker on more challenging datasets. See more discussions in Sec. 4 .
Figure A6 : Test accuracy (10% epochs) vs. test accuracy (100% epochs).
More in this topic

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Explore similar work

CardsList
May 23, 2026

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component RR has small propagated logit effect Ls∥Rψ1(s)∥∞L_s \| R ψ_1(s) \|_\infty, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive L2L_2-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge σ+σ_+ gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16 2:4+2{:}4{+}ToMe reaches 83.41%83.41\% top-1 (−1.70-1.70 pp from dense) at 59.81%59.81\% sparse-execution MAC reduction, with 1.388×1.388\times best-observed A40 native-2:42{:}4 backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives 2.705×2.705\times. At structured sparsity, ViT-B/16 6:126{:}12 reaches 83.74%83.74\%, ViT-L/16 8:168{:}16 dense+permutation reaches 85.33%85.33\% (−0.51-0.51 pp), and ConvNeXtV2-Base 12:1612{:}16 reaches 86.35%86.35\% (−0.37-0.37 pp). For CNNs, ResNet50 8:168{:}16 dense+permutation reaches 75.87%75.87\% (−0.26-0.26 pp), and ResNet152d CAST-conv+permutation reaches 81.33%81.33\% (−1.53-1.53 pp) at ∼50%{\sim}50\% MAC accounting with a 1.62×1.62\times A40 im2col+2:4+2{:}4 sparse-GEMM audit.

Leonid Berlyand, Theo Bourdais, Houman Owhadi +1

Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA

Structured PruningSparse Neural NetworksImage ClassificationNeural Network Pruning
AskPDF
Jun 20, 2026

Data Pruning: Redundant, Problematic, and Interdependent Samples

The performance of deep learning models is affected by not only data quantity but also data quality. Data pruning is a process by which practitioners can reduce the size of a dataset by only keeping the most important training data points, thereby achieving similar test set performance. We empirically investigate two popular data pruning methods under noisy and noiseless conditions and show that these methods fail in the presence of significant label noise. We highlight that the success of data pruning is distinctly affected by three factors: redundancy in the dataset, the presence of problematic samples, and interdependence between samples. We perform a detailed investigation on commonly used benchmark classification datasets and neural network architectures. We find that our observations are consistent across data distributions and training protocols.

Leon Freese, Marthinus W. Theunissen

Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

Dataset PruningTraining Data Selection
AskPDF
Feb 24, 2026

Elimination-compensation pruning for fully-connected neural networks

The unmatched ability of Deep Neural Networks in capturing complex patterns in large and noisy datasets is often associated with their large hypothesis space, and consequently to the vast amount of parameters that characterize model architectures. Pruning techniques affirmed themselves as valid tools to extract sparse representations of neural networks parameters, carefully balancing between compression and preservation of information. However, a fundamental assumption behind pruning is that expendable weights should have small impact on the error of the network, while highly important weights should tend to have a larger influence on the inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias, efficiently computable by automatic differentiation. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios. Finally, our findings are discussed and the theoretical implications of our results are presented.

Enrico Ballini, Luca Muscarnera, Alessio Fumagalli +2

MOX, Department of Mathematics, Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, Italy

Neural Network PruningNeural Network Compression
AskPDF