Low Rank Training

Low-rank training aims to reduce the computational cost and memory requirements of training large neural networks, particularly large language models (LLMs) and vision transformers, by representing weight matrices with lower-rank factorizations. Current research focuses on developing efficient algorithms that minimize performance loss compared to full-rank training, including methods that dynamically adjust rank during training or leverage online subspace projections to avoid computationally expensive singular value decompositions. This approach holds significant promise for enabling the training and deployment of larger and more complex models on resource-constrained hardware, thereby accelerating progress in various AI applications.