As neural networks continue to grow in scale, model compression is becoming increasingly important for efficient inference under limited computational resources. Structured pruning methods remove neurons or channels that are estimated to be less important, but the removed units may still contain useful information. From the viewpoint of coarse-graining a trained network, it is valuable to ask which information should be retained when multiple neuronal degrees of freedom are consolidated. In this paper, we discuss cluster-based merging methods for compression of trained neural networks. In addition to a data-free contribution-weighted averaging method, we propose neuron-merging methods in which neuron responses are mapped back to the pre-activation space via the inverse activation function, and the weights and biases of each representative neuron are estimated using the least-squares method. We also examine both a data-assisted strategy with actual training inputs and a data-free strategy using randomly generated inputs. The comparisons provide empirical evidence, in the tested sigmoid networks, that weight information is particularly useful for clustering whereas activation information is useful for representative-neuron reconstruction in the merging process.
Figures & tables
Figure 1: Structured pruning. In unstructured pruning, connections between neurons are removed individually. By contrast, structured pruning removes neurons and all their associated connections, thereby deleting the corresponding rows or columns from the weight matrices. Hence, the overall sizes of the weight matrices are reduced.
Item
Setting
Model
784–1024–1024–1024–10
Hidden activation
Sigmoid function
Optimizer
Adadelta with learning rate 1.0
Epochs
20
Batch size
128
Table 1: Training settings.
Figure 2: (Color online) Comparison of compression and accuracy through several structured pruning methods. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. The ℓ1 - and ℓ2 -norm weight-magnitude pruning methods are data-free and utilize only the weight information. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 3: Conceptual illustration of neuron merging. Here, similar neurons 1 and 3 are replaced by one representative neuron depicted by the shaded circle. The representative incoming and outgoing parameters are reconstructed.
Figure 4: (Color online) Test accuracy after neuron merging in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. For reference, this figure also shows the results of “Weight magnitude pruning (L2)”, as depicted in Fig. 2 . Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 5: (Color online) One-time compression time in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the computational time on a logarithmic scale. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 6: (Color online) Comparison of test accuracy using data-free procedures in the MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.
Figure 7: (Color online) Test accuracy after neuron merging in the Fashion-MNIST dataset experiments. The horizontal axis shows the node-retention ratio in the hidden layers, and the vertical axis shows the accuracy on the test data. For readability, panel (a) shows the results of the conventional pruning methods, data-free methods, and data-dependent methods, whereas panel (b) shows the training-data-free methods with randomly generated datasets; “C-weight/M-av” is repeated in both panels as a common reference. Symbols and error bars denote the mean and standard deviation over ten independently trained networks.