As neural networks continue to grow in scale, model compression is becoming increasingly important for efficient inference under limited computational resources. Structured pruning methods remove neurons or channels that are estimated to be less important, but the removed units may still contain useful information. From the viewpoint of coarse-graining a trained network, it is valuable to ask which information should be retained when multiple neuronal degrees of freedom are consolidated. In this paper, we discuss cluster-based merging methods for compression of trained neural networks. In addition to a data-free contribution-weighted averaging method, we propose neuron-merging methods in which neuron responses are mapped back to the pre-activation space via the inverse activation function, and the weights and biases of each representative neuron are estimated using the least-squares method. We also examine both a data-assisted strategy with actual training inputs and a data-free strategy using randomly generated inputs. The comparisons provide empirical evidence, in the tested sigmoid networks, that weight information is particularly useful for clustering whereas activation information is useful for representative-neuron reconstruction in the merging process.
Figures & tables
Figure 1: Structured pruning. In unstructured pruning, connections between neurons are removed individually. By contrast, structured pruning removes neurons and all their associated connections, thereby deleting the corresponding rows or columns from the weight matrices. Hence, the overall sizes of the weight matrices are reduced.
Item
Setting
Model
784–1024–1024–1024–10
Hidden activation
Sigmoid function
Optimizer
Adadelta with learning rate 1.0
Epochs
20
Batch size
128
Table 1: Training settings.
Figure 2: (Color online) Comparison of compression and accuracy through several structured pruning methods. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. The ℓ1 - and ℓ2 -norm weight-magnitude pruning methods are data-free and utilize only the weight information. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 3: Conceptual illustration of neuron merging. Here, similar neurons 1 and 3 are replaced by one representative neuron depicted by the shaded circle. The representative incoming and outgoing parameters are reconstructed.
Figure 4: (Color online) Test accuracy after neuron merging in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. For reference, this figure also shows the results of “Weight magnitude pruning (L2)”, as depicted in Fig. 2 . Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 5: (Color online) One-time compression time in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the computational time on a logarithmic scale. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 6: (Color online) Comparison of test accuracy using data-free procedures in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 7: (Color online) Test accuracy after neuron merging in the Fashion-MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. For readability, panel (a) shows the results of the conventional pruning methods, data-free methods, and data-dependent methods, whereas panel (b) shows the training-data-free methods with randomly generated datasets; “C-weight/M-av” is repeated in both panels as a common reference. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Neural network compression is commonly achieved by pruning parameters based on local importance scores, e.g., magnitude-based pruning. We propose a complementary approach that compresses models by aggregating neurons with similar functional behavior rather than removing weights independently. Our method encodes a trained network as a polynomial ODE system and applies a lumping method called Approximate Forward Differential Equivalence to identify neurons with approximately matching induced dynamics. A single tolerance parameter, ε, controls the compression level and induces a smooth trade-off between model size and predictive accuracy. We evaluate the method on synthetic datasets derived from nonlinear dynamical systems with known ground-truth behavior and on public regression benchmarks. Across both settings, the proposed approach achieves substantial parameter reduction while preserving accuracy, and consistently compares favorably with magnitude-based pruning and Wanda at similar compression levels. These results suggest that differential equivalence-based aggregation is a principled and effective alternative to conventional weight-centric pruning.
Ravi Dhiman, Andrea Passarella, Mirco Tribastone +1
IMT School for Advanced Studies Lucca 55100, Italy · IIT CNR, Pisa 56124, Italy
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.
Maria Matveev, Pascal Esser, Ayush Bharadwaj +2
Department of Mathematics, LMU Munich · Munich Center for Machine Learning (MCML) · Independent +3
The unmatched ability of Deep Neural Networks in capturing complex patterns in large and noisy datasets is often associated with their large hypothesis space, and consequently to the vast amount of parameters that characterize model architectures. Pruning techniques affirmed themselves as valid tools to extract sparse representations of neural networks parameters, carefully balancing between compression and preservation of information. However, a fundamental assumption behind pruning is that expendable weights should have small impact on the error of the network, while highly important weights should tend to have a larger influence on the inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias, efficiently computable by automatic differentiation. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios. Finally, our findings are discussed and the theoretical implications of our results are presented.