Is Oracle Pruning the True Oracle?

Sicheng Feng; Keda Tao; Huan Wang

cs.LGNov 28, 2024

Is Oracle Pruning the True Oracle?

Authors: Sicheng Feng, Keda Tao, Huan Wang

Organizations: Westlake University, Hangzhou, China · Nankai University, Tianjin, China

Abstract

Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has served as the foundation for most neural network pruning methods for over thirty-five years, while few (if any) have thought about how much the foundation really holds. This paper, for the first time, attempts to systematically examine its validity on deep neural networks through empirical correlation analyses and provides meta-framework reflections on the field of neural network pruning. Specifically, this paper focuses on the pruning algorithms with three stages: training, pruning, and retraining. We analyze the correlation in model performance before and after the retraining stage. Extensive experiments (37K models are trained) across a wide spectrum of models (LeNet5, VGG, ResNets, ViT, MLLM) and datasets (MNIST, CIFAR10/CIFAR100, ImageNet-1K, MLLM data) are conducted. For large-scale experiments, we adopt approximate oracle pruning due to the prohibitive cost of exact oracle pruning. The results point to a counterintuitive conclusion: for deep learning models of nontrivial size (already at the scale of ResNet56 on CIFAR-10), pre-retraining performance is negligibly correlated with post-retraining performance. In other words, the weights identified by oracle pruning can scarcely guarantee strong performance following retraining. This further suggests that existing works that derive pruning criteria from oracle pruning may rest on a questionable foundational premise. Further studies suggest that rising task complexity is a primary factor behind the invalidity of oracle pruning nowadays. Finally, given the evidence, we argue that the retraining stage in a pruning algorithm should be accounted for when developing pruning criteria.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis framework of this work. We study the validity of oracle pruning in this paper, by examining the correlation between the pruned train loss and the final test performance ( i.e. , test accuracy or test loss). We apply this analysis framework to a wide range of networks and datasets (from toy networks like LeNet5-Mini to large ones like TinyLLaVA-3.1B) in order to have a comprehensive evaluation. The key finding of this work is that, to our surprise, oracle pruning becomes invalid on modern networks and datasets (starting from the CIFAR level), challenging the conventional belief in network pruning for the past 35 years.

Network ( Dataset)	Kendall	Anomaly	Counterexample
Network ( Dataset)	correlation	ratio	ratio
LeNet5-Mini ( MNIST)	✓	✓	✗
ResNet56 ( CIFAR10)	✓	✓	✗
VGG19 ( CIFAR100)	✓	✓	✗
ResNet18 ( ImageNet-1K)	✓	✓	✗
ViT-B/16 ( ImageNet-1K)	✓	✗	✓

Table 1: Summary of the analysis methods for checking the validity of oracle pruning on different networks and datasets. LeNet5-Mini is a small enough network to achieve oracle pruning exactly; for other networks, we randomly sample enough models to analyze Kendall correlation, anomaly ratio, and counterexample ratio.

Figure 2: Pruned train loss vs. final test accuracy on MNIST with LeNet5-Mini. The subcaptions correspond to the pruning settings. The blue star indicates the reference oracle pruning result ( i.e. , the one with the smallest pruned train loss). The points with final test accuracy higher than the reference oracle pruning result are marked in red (anomaly points), and those lower are marked in green.

Pr.	Conv1	Conv2	Conv3
0.2	-0.60 / 4.9e-09	-0.76 / 2.0e-13	-0.71 / 7.0e-12
0.3	-0.51 / 1.3e-16	-0.61 / 4.8e-23	-0.59 / 9.8e-22
0.4	-0.48 / 1.3e-24	-0.61 / 7.4e-39	-0.58 / 2.2e-35
0.5	-0.51 / 2.1e-33	-0.51 / 1.8e-33	-0.60 / 2.5e-45
0.6	-0.41 / 6.1e-19	-0.42 / 3.5e-19	-0.47 / 1.2e-23
0.7	-0.19 / 1.8e-03	-0.44 / 1.4e-12	-0.19 / 2.1e-03

Table 2: Kendall correlation between pruned train loss and final test accuracy, by exhaustively pruning LeNet5-Mini network on MNIST dataset. Each entry in the table is arranged as Kendall coefficient / p-value. Pr. means the pruning ratio for the corresponding layer combination of Conv1, Conv2, and Conv3. The red indicates the results where oracle pruning is invalid as defined in Sec. 3.1 .

Figure 3: Pruned train loss vs. final test accuracy with ResNet56, VGG19, and ResNet18. The subcaptions correspond to the pruning settings.

Method	#Params	Pruned train loss	Final test performance
Dense model	3.1B	/	77.15
ONP	2B	1.35	76.70
UMP	2B	1.24	76.75
GMP	2B	1.17	76.30
PNP	2B	1.16	76.28

Table 3: Pruning results on TinyLLaVA-3.1B. The final test performance is averaged on 5 benchmark datasets following standard practices (see Tab. A6 in the Appendix C for the detailed results).

Figure 4: Pruned train loss vs. final test accuracy on the variants of MNIST dataset, with LeNet5-Mini network (pruning ratio 0.5, Conv1 layer). FMNIST and KMNIST are two drop-in replacements of MNIST, which are more complex. As seen, the correlation becomes weaker on more challenging datasets.

Figure 5: Pruned train loss vs. final test accuracy on MNIST with different variants of LeNet5-Mini (pruning ratio 0.5, Conv1 layer). The original LeNet5-Mini (Base) has 5 layers (D5), and each layer has 10 neurons (W10). Here, we change the model width and depth to obtain different variants. As seen, the correlation becomes weaker when pruning more complex networks.

Figure 6: Pruned train loss vs. final test loss on MNIST with different variants of LeNet5-Mini (pruning ratio 0.5, Conv1 layer). The original LeNet5-Mini (Base) has 5 layers (D5), and each layer has 10 neurons (W10). Here, we change the model width and depth to obtain different variants. As seen, the correlation becomes weaker when pruning more complex networks.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Network & Data	Solver	LR policy	Total epoch	Batch size
		(pre-train and retrain)
LeNet5-Mini	SGD (0.9, 1e-4)	Multi-step (0:1e-2, 20:1e-3)	30	256
(MNIST)		Multi-step (0:1e-3, 20:1e-4)	30	256
ResNet56	SGD (0.9, 5e-4)	Multi-step (0:1e-1, 100:1e-2, 150:1e-3)	200	128
(CIFAR10)		Multi-step (0:1e-2, 60:1e-3, 90:1e-4)	120	256
VGG19	SGD (0.9, 5e-4)	Multi-step (0:1e-1, 100:1e-2, 150:1e-3)	200	256

Appendix

Table A1 : Training setting summary. For the solver, the momentum and weight decay are in brackets. For CIFAR10, batch size 256 is used for retraining instead of 128, which is for saving training time. For LR policy, total epoch, and batch size, the first one is for the pre-training stage, the second is for the retraining stage.

Dataset	Network	Pruning ratio
MNIST	LeNet5-Mini	[0.2-0.8, 0, 0]
		[0, 0.2-0.8, 0]
		[0, 0, 0.2-0.8]
		[0, 0.5, 0.5]
		[0.5, 0, 0.5]
		[0.5, 0.5, 0]

Appendix

Table A2 : Pruning ratio summary. See more detailed statements in Appendix A.3 .

Iteration	Pruned Train Loss vs Final Test Loss	Pruned Train Loss vs Final Test Acc.
LR1 (first four iterations: 0.001; final stage: 0:0.1, 20:0.01, 40:0.001)
1/5	0.05 / 0.31	0.03 / 0.58
2/5	-0.05 / 0.32	0.04 / 0.47
3/5	0.02 / 0.71	0.04 / 0.41
4/5	0.10 / 0.06	0.08 / 0.13
5/5	0.05 / 0.34	0.03 / 0.52

Appendix

Table A3 : Kendall correlation between pruned train loss at each iterative pruning step and final test performance on ResNet50 with CIFAR100. Each entry in the table is arranged as coefficient / p-value.

Pr.	Conv1	Conv2	Conv3
0.2	0.57 / 3.1e-08	0.74 / 6.0e-13	0.66 / 1.4e-10
0.3	0.53 / 7.3e-18	0.67 / 2.9e-27	0.55 / 3.5e-19
0.4	0.48 / 1.2e-24	0.65 / 1.8e-44	0.54 / 7.3e-32
0.5	0.54 / 2.6e-37	0.56 / 1.5e-40	0.57 / 4.8e-42
0.6	0.45 / 2.8e-22	0.44 / 5.5e-21	0.46 / 3.5e-23
0.7	0.20 / 1.0e-03	0.45 / 2.4e-13	0.20 / 1.1e-03

Appendix

Table A4 : Kendall correlation between pruned train loss and final test loss, by exhaustively pruning LeNet5-Mini network on MNIST dataset. Each entry in the table is arranged as Kendall coefficient / p-value. Pr. means the pruning ratio for the corresponding layer combination of Conv1, Conv2, and Conv3. The red entries mean these results pose weak or counterintuitive correlations.

Figure A1 : Pruned train loss vs. final test loss on MNIST with LeNet5-Mini. The subcaptions correspond to the pruning rates of each image. The star denotes the oracle pruning results, where points with final test loss lower than the oracle pruning are marked in red, and those lower are marked in green.

Figure A2 : Pruned train loss vs. final test accuracy on MNIST with LeNet5-Mini.

Figure A3 : Pruned train loss vs. final test loss on MNIST with LeNet5-Mini.

Combination	Pruned train loss	Final test accuracy	Final test loss
1	6.9768	76.2445	1.9714
2	7.0515	75.9731	1.9781
3	7.0442	76.0662	1.9826
4	7.0162	76.3223	1.9689
5	6.9989	74.0748	2.0530
6	7.0709	75.7066	1.9901

Appendix

Table A5 : Results of ViT-B/16 on ImageNet-1K.

Figure A4 : Pruned train loss vs. final test loss with ResNet56 (on CIFAR10), VGG19 (on CIFAR100), ResNet18 (on ImageNet-1K).

Methods	#Params	Vision-Encoder	Res.	PTL	SQA	TextVQA	GQA	MM-Vet	POPE	Avg.
Dense model	3.1B	SigLIP-0.4B	384	/	69.1	59.1	62	32	86.4	77.15
UMP	2B	SigLIP-0.4B	384	1.24	69.4	56.7	60.1	34.2	86.6	76.75
GMP	2B	SigLIP-0.4B	384	1.17	69.9	55.8	60.1	33	86.4	76.30
ONP	2B	SigLIP-0.4B	384	1.35	69.8	55.7	61.5	33	86.8	76.70
PNP	2B	SigLIP-0.4B	384	1.16	69.5	55.1	61.5	33.1	86	76.28

Appendix

Table A6 : For the pruning results of MLLMs, the table presents the performance of the Base model and the models pruned using the four strategies. We present the model’s performance across five benchmarks. PTL stands for the pruned train loss.

Figure A5 : Pruned train loss vs. final test loss on the variants of MNIST dataset, with LeNet5-Mini network (pruning ratio 0.5, Conv1 layer). FMNIST and KMNIST are two drop-in replacements of MNIST, which are more complex. As seen, the correlation becomes weaker on more challenging datasets. See more discussions in Sec. 4 .

Figure A6 : Test accuracy (10% epochs) vs. test accuracy (100% epochs).

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
May 23, 2026Leonid Berlyand, Theo Bourdais, Houman Owhadi +1Structured Pruning Sparse Neural Networks
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA
Data Pruning: Redundant, Problematic, and Interdependent Samples
Jun 20, 2026Leon Freese, Marthinus W. TheunissenDataset Pruning Training Data Selection
Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa
Elimination-compensation pruning for fully-connected neural networks
Feb 24, 2026Enrico Ballini, Luca Muscarnera, Alessio Fumagalli +2Neural Network Pruning Neural Network Compression
MOX, Department of Mathematics, Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, Italy

May 23, 2026J/K move · Enter open · S save

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Leonid Berlyand, Theo Bourdais, Houman Owhadi, Yitzchak Shmalo

Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA

Structured Pruning Sparse Neural Networks Image Classification Neural Network Pruning

Open paper AskPDF arXiv

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component

R

has small propagated logit effect

L_s \| R ψ_1(s) \|_\infty

, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive

L_2

-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge

σ_+

gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16

2{:}4{+}

ToMe reaches

83.41\%

top-1 (

-1.70

pp from dense) at

59.81\%

sparse-execution MAC reduction, with

1.388\times

best-observed A40 native-

2{:}4

backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives

2.705\times

. At structured sparsity, ViT-B/16

6{:}12

reaches

83.74\%

, ViT-L/16

8{:}16

dense+permutation reaches

85.33\%

(

-0.51

pp), and ConvNeXtV2-Base

12{:}16

reaches

86.35\%

(

-0.37

pp). For CNNs, ResNet50

8{:}16

dense+permutation reaches

75.87\%

(

-0.26

pp), and ResNet152d CAST-conv+permutation reaches

81.33\%

(

-1.53

pp) at

{\sim}50\%

MAC accounting with a

1.62\times

A40 im2col

+2{:}4

sparse-GEMM audit.

Abstract

Neural Network Pruning

Explore similar work

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Data Pruning: Redundant, Problematic, and Interdependent Samples

Elimination-compensation pruning for fully-connected neural networks