Low Rank Factorization

Low-rank factorization aims to decompose large matrices into products of smaller matrices, thereby reducing computational complexity and memory requirements for large models, particularly in natural language processing and machine learning. Current research focuses on improving the efficiency and stability of factorization techniques, including exploring novel algorithms like Householder reflections and activation-aware singular value decomposition (ASVD), and optimizing the allocation of low-rank dimensions using Bayesian optimization. These advancements are crucial for deploying and scaling large language models and other computationally intensive applications, offering significant improvements in efficiency without substantial performance loss.

Papers

June 17, 2024

An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers
Ashim Gupta, Sina Mahdipour Saravani, P. Sadayappan, Vivek Srikumar
Transformer Based Model Matrix Factorization Pre Trained Transformer Low Rank Factorization

May 17, 2024

Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang
Large Language Model Bayesian Optimization Low Rank Compression Low Rank Factorization

May 13, 2024

Efficient Matrix Factorization Via Householder Reflections
Anirudh Dash, Aditya Siripuram
Matrix Factorization Dictionary Learning Robust Recovery Low Rank Factorization

April 13, 2024

Concentration properties of fractional posterior in 1-bit matrix completion
The Tien Mai
Matrix Completion Concentration Bound Low Rank Factorization Fractional Posterior

February 10, 2024

Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning
Kang Du, Yu Xiang
Self Supervised Learning Supervised Learning Average Approximation SSL Algorithm Approximation Theory Redundancy Reduction Regularization Based Method Low Rank Factorization

February 8, 2024

Checking the Sufficiently Scattered Condition using a Global Non-Convex Optimization Software
Nicolas Gillis, Robert Luce
Hyperspectral Image Matrix Factorization Non Convex Low Rank Factorization

December 10, 2023

ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, Guangyu Sun
Singular Value Decomposition Low Rank Decomposition Low Rank Factorization

October 9, 2023

Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterization
Cong Ma, Xingyu Xu, Tian Tong, Yuejie Chi
Gradient Descent Low Rank Gradient Based Global Convergence Tensor Factorization Overparametrization Bound Low Rank Matrix Estimation Low Rank Factorization

June 30, 2022

Language model compression with weighted low-rank factorization
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, Hongxia Jin
Model Compression Singular Value Decomposition Compact Model Low Rank Factorization Language Model Compression Matrix Learning

February 1, 2022

Graph-based Neural Acceleration for Nonnegative Matrix Factorization
Jens Sjölund, Maria Bånkestad
Graph Neural Network Sparse Matrix Low Rank Factorization Matrix Computation

November 29, 2021

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization
Dezhong Yao, Wanning Pan, Michael J O'Neill, Yutong Dai, Yao Wan, Hai Jin, Lichao Sun
Federated Learning Model Compression Heterogeneous Federated Learning Heterogeneous Model Low Rank Factorization

November 28, 2021

Towards Robust and Automatic Hyper-Parameter Tunning
Mathieu Tuli, Mahdi S. Hosseini, Konstantinos N. Plataniotis
Bayesian Optimization Hyper Parameter Hyper Parameter Optimization Hyper Parameter Tuning Low Rank Factorization

Low Rank Factorization

Papers

An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers

Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization

Efficient Matrix Factorization Via Householder Reflections

Concentration properties of fractional posterior in 1-bit matrix completion

Low-Rank Approximation of Structural Redundancy for Self-Supervised Learning

Checking the Sufficiently Scattered Condition using a Global Non-Convex Optimization Software

ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterization

Language model compression with weighted low-rank factorization

Graph-based Neural Acceleration for Nonnegative Matrix Factorization

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Towards Robust and Automatic Hyper-Parameter Tunning