Lattice-Based Pruning of Recurrent Neural Networks via Poset Modeling: A Corrected Theory and a Matched-Sparsity Evaluation
Organizations: School of Interwoven Arts and Sciences, Krea University, Sri City 517541, Andhra Pradesh, India
Abstract
Magnitude pruning of recurrent neural networks (RNNs) ignores how information flows between neurons. An earlier version proposed to model RNN dependencies as a partially ordered set (poset) and to retain the meet-irreducible elements of its dependency lattice. By Birkhoff's representation theorem, the meet-irreducible ideals of a finite poset are in bijection with its elements, so retaining them retains everything; the critical-set criterion used earlier selects precisely the elements comparable to all others, of which a dense recurrent layer has none. Condensing a thresholded dependency graph into strongly connected components yields a well-defined dependency order, a graded order score and an acyclic pruning rule; for Cohen-Grossberg networks and LSTMs, acyclic pruning makes local stability decompose over modules. For a 128-unit LSTM pruned at matched sparsity on row-wise and chunked sequential MNIST (three seeds, equal training budgets), the original rules remove every recurrent weight or equal random column pruning; the order score is no better than degree, PageRank or norm; trained weights show little directional bias; and imposing acyclicity from initialisation costs 2.1 and 10.5 percentage points at 50% and 90% sparsity. In the networks studied, order structure must be imposed rather than discovered.
Figures & tables
| Granularity | Score | Rule |
|---|---|---|
| Column | Order score | lowest ( Definition 3.7 ) removed first |
| Degree | lowest in- plus out-degree in | |
| PageRank | lowest PageRank of the reversed weighted (influence) | |
| Norm | lowest norm of the column of | |
| Random | uniformly random columns | |
| Neuron pair | Magnitude | lowest ( 1 ) |
| Model | sparsity (%) | One-shot acc. (%) | Final acc. (%) |
|---|---|---|---|
| Row-MNIST | |||
| Unpruned, epochs (reference) | 0.0 | — | 98.5 0.2 |
| Unpruned, epochs (equal-budget reference) | 0.0 | — | 98.6 0.0 |
| No recurrence ( ), epochs | 100.0 | — | 92.1 0.4 |
| Continuous rule, | 100.0 | 16.7 2.5 | 89.6 0.4 |
| Binary rule (first half of neurons kept) | 50.0 | 62.4 8.6 | 98.4 0.2 |
| Row-MNIST | Chunked seq. MNIST | ||||||
|---|---|---|---|---|---|---|---|
| Granularity | Score | 25% | 50% | 75% | 90% | 50% | 90% |
| Unstructured | Magnitude | 98.6 0.1 | 98.7 0.1 | 98.6 0.2 | 98.2 0.2 | 96.4 0.2 | 92.6 0.7 |
| Neuron pair | Magnitude | 98.6 0.1 | 98.7 0.2 | 98.4 0.1 | 97.8 0.3 | 96.1 0.3 | 86.2 2.8 |
| Random | 98.6 0.1 | 98.5 0.1 | 97.9 0.2 | 96.9 0.4 | 94.5 1.1 | 75.5 0.9 | |
| Acyclic (cascade) | FAS order | 98.6 0.2 | 98.3 0.1 | 98.0 0.1 | 97.1 0.1 | 89.7 1.0 | 76.3 1.8 |
| Random order | 98.7 0.1 | 98.3 0.2 | 97.8 0.1 | 96.9 0.1 | 87.9 2.0 | 70.7 5.0 | |
| Row-MNIST | Chunked seq. MNIST | ||||
|---|---|---|---|---|---|
| Mask held fixed from initialisation | 50% | 75% | 90% | 50% | 90% |
| Random pair mask | 98.3 0.1 | 98.1 0.2 | 97.8 0.1 | 91.5 2.2 | 87.6 0.6 |
| Random acyclic mask | 98.3 0.1 | 98.2 0.1 | 97.2 0.3 | 89.4 2.1 | 77.1 3.8 |
| Rewound pair-magnitude mask | 98.5 0.1 | 98.3 0.2 | 97.9 0.1 | 93.8 0.4 | 86.4 2.4 |
| Rewound acyclic FAS mask | 98.4 0.0 | 98.2 0.1 | 97.3 0.2 | 88.7 0.9 | 78.5 2.0 |
| Rewound unstructured magnitude mask | 98.7 0.1 | 98.6 0.1 | 98.1 0.1 | — | — |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Row-MNIST | Chunked seq. MNIST | ||||||
|---|---|---|---|---|---|---|---|
| Granularity | Score | 25% | 50% | 75% | 90% | 50% | 90% |
| Unstructured | Magnitude | 98.5 0.2 | 98.2 0.4 | 90.4 3.5 | 43.9 2.6 | 89.1 1.2 | 10.5 0.5 |
| Neuron pair | Magnitude | 98.2 0.2 | 95.0 1.1 | 61.8 2.6 | 31.1 6.4 | 49.9 9.9 | 9.8 1.4 |
| Random | 95.8 1.1 | 67.6 4.5 | 30.9 5.1 | 22.0 1.6 | 12.3 1.3 | 10.7 0.1 | |
| Acyclic (cascade) | FAS order | 85.7 3.0 | 43.3 1.7 | 34.7 2.9 | 26.7 3.5 | 11.4 0.1 | 10.6 0.1 |
| Random order | 82.0 4.8 | 43.6 7.3 | 34.0 1.2 | 23.0 4.2 | 10.9 0.8 | 10.7 0.0 | |
Neural Network Pruning
Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.
See all 303 papers