Optimal Pruning for Neural Architectures using Fisher Information Distances

David S. Berman; Yen-Yu Fu; Edward Hirst; Thelma Chiwete Obirai

cs.AISep 14, 2026

Optimal Pruning for Neural Architectures using Fisher Information Distances

Authors: David S. Berman, Yen-Yu Fu, Edward Hirst, Thelma Chiwete Obirai

Organizations: Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, UK · Institute of Mathematics, Statistics and Scientific Computing, University of Campinas (Unicamp), 13083-859, Brazil

Abstract

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete $0$ - $100\%$ pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

Figures & tables

Architecture	Dataset	Parameters	Epochs	Dense accuracy [%]
SimpleNN	MNIST	55 050	50	$97.59\pm 0.27$
SimpleNN	CIFAR-10	201 482	50	$50.46\pm 0.63$
SimpleViT	MNIST	539 914	50	$98.70\pm 0.10$
SimpleViT	CIFAR-10	545 930	50	$81.22\pm 0.47$

Table 4.1 : The four testbeds. Dense accuracy is on the test split, mean

\pm

standard deviation over the five seeds. SimpleNN ’s parameter count differs between datasets because the flattened input dimension does; SimpleViT ’s differs only through the patch-embedding convolution.

SimpleNN — MNIST
Scheme	AUC (acc.)	floor-corr.	AUC (MCC)	vs. mag. (acc. / MCC)	Compute
Magnitude	$0.907\pm 0.007$	$0.896\pm 0.008$	$0.898\pm 0.007$	$1.00\times$ / $1.00\times$	6.3 s
FIM	$0.743\pm 0.022$	$0.714\pm 0.024$	$0.718\pm 0.023$	$0.82\times$ / $0.80\times$	9.9 s
F-dist (one-shot)	$0.891\pm 0.008$	$0.879\pm 0.009$	$0.881\pm 0.009$	$0.98\times$ / $0.98\times$	9.9 s
F-dist (iterative)	$0.924\pm 0.003$	$0.916\pm 0.003$	$0.916\pm 0.003$	$1.02\times$ / $1.02\times$	9.9 s
F-dist (global)	$0.930\pm 0.002$	$0.922\pm 0.002$	$0.922\pm 0.002$	$1.02\times$ / $1.03\times$	17.0 s

Table 4.2 : Area under the normalised performance curve over the full

0

–

100\%

pruning range, mean

\pm

standard deviation over five seeds. The floor-corrected column rescales chance to

0

; it is omitted for MCC, whose floor is identically

0

. Compute is the median wall-clock of the pruning sweep over the five seeds. Here and in the figures, F-dist abbreviates Fisher-distance . † One SimpleViT /MNIST seed ran on a contended node and took

6.41

d against

3.36

–

3.38

d for the other four; the median is quoted rather than the mean, which that single outlier inflates by

18\%

Figure 4.1 : Normalised test accuracy against pruning ratio for all six schemes on the four testbeds. Lines are means over five seeds and bands are

\pm

one standard deviation. The exact Fisher-distance scheme is drawn beneath a wide translucent halo; where a cheaper scheme reproduces it, that curve is visibly contained within the halo (the iterative scheme is largely hidden beneath it on three of the four panels).

Figure 4.2 : Area under the normalised accuracy curve against the wall-clock cost of computing it, on logarithmic axes; error bars are

\pm

one standard deviation over five seeds. The annotated arrow marks the cost ratio between the iterative and exact Fisher-distance measures, which on the transformer testbeds sit at the same height four decades apart.

$K$	probes/coord.	AUC (acc.)	$\Delta$ vs. $K\!=\!2$	Compute
2	1	$0.9283\pm 0.0020$	$+0.0000$	57.3 s
3	2	$0.9285\pm 0.0015$	$+0.0002$	78.9 s
5	4	$0.9272\pm 0.0033$	$-0.0010$	1.9 min
9	8	$0.9292\pm 0.0032$	$+0.0010$	3.5 min
17	16	$0.9269\pm 0.0042$	$-0.0014$	5.5 min

Table 4.3 : Dependence of the exact Fisher-distance measure on the path resolution

K

, for SimpleNN /MNIST over five seeds. The probe count per coordinate is

K-1

; compute is the median wall-clock of the full sweep.

Figure 4.3 : Area under the normalised accuracy curve (above) and sweep cost (below) against path resolution

K

. The shaded band is the

K=2

result

\pm

one standard deviation; every richer grid lands inside it, while the cost rises by a factor of

5.8

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Normalised Matthews correlation coefficient against pruning ratio, for all six schemes on the four testbeds. Lines are means over five seeds and bands are

\pm

one standard deviation. Compare Figure 4.1 : the curves are qualitatively identical, displaced downwards by an amount which grows with the degree of degradation.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 284 papers

Feb 20, 2026

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Olga Saukh, Dong Wang, Haris Šikić +2

Graz University of Technology · Complexity Science Hub · Swiss Data Science Center +1

Model Compression Neural Network Compression Neural Network Pruning

Ask PDF

May 23, 2026

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component

R

has small propagated logit effect

L_s \| R ψ_1(s) \|_\infty

, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive

L_2

-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge

σ_+

gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16

2{:}4{+}

ToMe reaches

83.41\%

top-1 (

-1.70

pp from dense) at

59.81\%

sparse-execution MAC reduction, with

1.388\times

best-observed A40 native-

2{:}4

backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives

2.705\times

. At structured sparsity, ViT-B/16

6{:}12

reaches

83.74\%

, ViT-L/16

8{:}16

dense+permutation reaches

85.33\%

(

-0.51

pp), and ConvNeXtV2-Base

12{:}16

reaches

86.35\%

(

-0.37

pp). For CNNs, ResNet50

8{:}16

dense+permutation reaches

75.87\%

(

-0.26

pp), and ResNet152d CAST-conv+permutation reaches

81.33\%

(

-1.53

pp) at

{\sim}50\%

MAC accounting with a

1.62\times

A40 im2col

+2{:}4

sparse-GEMM audit.

Leonid Berlyand, Theo Bourdais, Houman Owhadi +1

Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA

Structured Pruning Sparse Neural Networks Image Classification Neural Network Pruning

Ask PDF

Apr 6, 2025

Hyperflux: Pruning Reveals Importance

Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel

L_0

method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.

Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu

Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

Sparse Neural Networks Neural Network Training Dynamics Neural Network Pruning

Ask PDF

Abstract

Neural Network Pruning

Explore similar work

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Hyperflux: Pruning Reveals Importance