Optimal Pruning for Neural Architectures using Fisher Information Distances

David S. Berman; Yen-Yu Fu; Edward Hirst; Thelma Chiwete Obirai

cs.AISep 14, 2026

Optimal Pruning for Neural Architectures using Fisher Information Distances

Authors: David S. Berman, Yen-Yu Fu, Edward Hirst, Thelma Chiwete Obirai

Organizations: Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, UK · Institute of Mathematics, Statistics and Scientific Computing, University of Campinas (Unicamp), 13083-859, Brazil

Abstract

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete $0$ - $100\%$ pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

Figures & tables

Architecture	Dataset	Parameters	Epochs	Dense accuracy [%]
SimpleNN	MNIST	55 050	50	$97.59\pm 0.27$
SimpleNN	CIFAR-10	201 482	50	$50.46\pm 0.63$
SimpleViT	MNIST	539 914	50	$98.70\pm 0.10$
SimpleViT	CIFAR-10	545 930	50	$81.22\pm 0.47$

Table 4.1 : The four testbeds. Dense accuracy is on the test split, mean

\pm

standard deviation over the five seeds. SimpleNN ’s parameter count differs between datasets because the flattened input dimension does; SimpleViT ’s differs only through the patch-embedding convolution.

SimpleNN — MNIST
Scheme	AUC (acc.)	floor-corr.	AUC (MCC)	vs. mag. (acc. / MCC)	Compute
Magnitude	$0.907\pm 0.007$	$0.896\pm 0.008$	$0.898\pm 0.007$	$1.00\times$ / $1.00\times$	6.3 s
FIM	$0.743\pm 0.022$	$0.714\pm 0.024$	$0.718\pm 0.023$	$0.82\times$ / $0.80\times$	9.9 s
F-dist (one-shot)	$0.891\pm 0.008$	$0.879\pm 0.009$	$0.881\pm 0.009$	$0.98\times$ / $0.98\times$	9.9 s
F-dist (iterative)	$0.924\pm 0.003$	$0.916\pm 0.003$	$0.916\pm 0.003$	$1.02\times$ / $1.02\times$	9.9 s
F-dist (global)	$0.930\pm 0.002$	$0.922\pm 0.002$	$0.922\pm 0.002$	$1.02\times$ / $1.03\times$	17.0 s

Table 4.2 : Area under the normalised performance curve over the full

0

–

100\%

pruning range, mean

\pm

standard deviation over five seeds. The floor-corrected column rescales chance to

0

; it is omitted for MCC, whose floor is identically

0

. Compute is the median wall-clock of the pruning sweep over the five seeds. Here and in the figures, F-dist abbreviates Fisher-distance . † One SimpleViT /MNIST seed ran on a contended node and took

6.41

d against

3.36

–

3.38

d for the other four; the median is quoted rather than the mean, which that single outlier inflates by

18\%

Figure 4.1 : Normalised test accuracy against pruning ratio for all six schemes on the four testbeds. Lines are means over five seeds and bands are

\pm

one standard deviation. The exact Fisher-distance scheme is drawn beneath a wide translucent halo; where a cheaper scheme reproduces it, that curve is visibly contained within the halo (the iterative scheme is largely hidden beneath it on three of the four panels).

Figure 4.2 : Area under the normalised accuracy curve against the wall-clock cost of computing it, on logarithmic axes; error bars are

\pm

one standard deviation over five seeds. The annotated arrow marks the cost ratio between the iterative and exact Fisher-distance measures, which on the transformer testbeds sit at the same height four decades apart.

$K$	probes/coord.	AUC (acc.)	$\Delta$ vs. $K\!=\!2$	Compute
2	1	$0.9283\pm 0.0020$	$+0.0000$	57.3 s
3	2	$0.9285\pm 0.0015$	$+0.0002$	78.9 s
5	4	$0.9272\pm 0.0033$	$-0.0010$	1.9 min
9	8	$0.9292\pm 0.0032$	$+0.0010$	3.5 min
17	16	$0.9269\pm 0.0042$	$-0.0014$	5.5 min

Table 4.3 : Dependence of the exact Fisher-distance measure on the path resolution

K

, for SimpleNN /MNIST over five seeds. The probe count per coordinate is

K-1

; compute is the median wall-clock of the full sweep.

Figure 4.3 : Area under the normalised accuracy curve (above) and sweep cost (below) against path resolution

K

. The shaded band is the

K=2

result

\pm

one standard deviation; every richer grid lands inside it, while the cost rises by a factor of

5.8

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Normalised Matthews correlation coefficient against pruning ratio, for all six schemes on the four testbeds. Lines are means over five seeds and bands are

\pm

one standard deviation. Compare Figure 4.1 : the curves are qualitatively identical, displaced downwards by an amount which grows with the degree of degradation.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 284 papers

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry
Feb 20, 2026Olga Saukh, Dong Wang, Haris Šikić +2Model Compression Neural Network Compression
Graz University of Technology · Complexity Science Hub · Swiss Data Science Center +1
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
May 23, 2026Leonid Berlyand, Theo Bourdais, Houman Owhadi +1Structured Pruning Sparse Neural Networks
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA
Hyperflux: Pruning Reveals Importance
Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural Networks Neural Network Training Dynamics
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

Feb 20, 2026J/K move · Enter open · S save

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Olga Saukh, Dong Wang, Haris Šikić, Yun Cheng+1

Graz University of Technology · Complexity Science Hub · Swiss Data Science Center+1

Model Compression Neural Network Compression Neural Network Pruning

Open paper AskPDF arXiv

Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Abstract

Neural Network Pruning

Explore similar work

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

Hyperflux: Pruning Reveals Importance