A Deep State-Space Model Compression Method using Upper Bound on Output Error
Organizations: Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8656, Japan
Abstract
We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the -error norms between the layerwise LQO systems. In particular, we show that reducing the approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error. Next, we formulate an optimization problem for the derived upper bound and develop a gradient-based MOR method. In the numerical experiments, using the IMDb task from the LRA benchmark, we demonstrate the effectiveness of the proposed upper-bound-based compression method. In particular, we show that the number of trainable parameters can be reduced by approximately 60% without retraining while maintaining the performance of the original model.
Figures & tables
| Method | Relative error for ( 39 ) | TestAcc. | |
| TLBT | |||
| TLH2 | |||
| Alg. 1 (TLBT init.) | {\color[rgb]{0,0,0}6.203{\times}10^{-1}} | {\color[rgb]{0,0,0}0.7621} | |
| Alg. 1 (TLH2 init.) | {\color[rgb]{0,0,0}6.101{\times}10^{-1}} | {\color[rgb]{0,0,0}0.7642} | |
| TLBT | |||
| Alg. 1 (TLBT init.) | {\color[rgb]{0,0,0}3.073{\times}10^{-1}} | {\color[rgb]{0,0,0}0.8138} |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Relative error for ( 39 ) | TestPPL. | |
| {\color[rgb]{0,0,0}111.93} | ||
| \mathbf{{\color[rgb]{0,0,0}9.833{\times}10^{-2}}} |
Model Compression
Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.
See all 131 papers