Structured Matrix

Structured matrices are being extensively investigated as replacements for dense matrices in deep learning models to improve computational efficiency and reduce memory requirements. Current research focuses on developing novel matrix structures (e.g., Block Tensor-Train, Monarch matrices) and algorithms for efficiently training and utilizing these structures within various architectures, including Transformers and Mixture-of-Experts models. This work aims to achieve comparable or even superior performance to dense models while significantly reducing computational cost, impacting areas such as large language models, image processing, and time series forecasting. The ultimate goal is to enable the training and deployment of larger and more complex models with reduced resource consumption.