Low Rank Compression

Low-rank compression aims to reduce the size and computational cost of large machine learning models, such as transformers, convolutional neural networks, and large language models, without significantly sacrificing performance. Current research focuses on developing efficient algorithms for low-rank matrix factorization and decomposition, often incorporating techniques like adaptive compression, Bayesian optimization, and error feedback to improve accuracy and speed. These advancements are crucial for deploying sophisticated models on resource-constrained devices and improving the efficiency of training and inference in various applications, including speech recognition, recommendation systems, and computer vision. The resulting compact models offer significant benefits in terms of reduced energy consumption and faster processing times.

Papers

September 13, 2024

Anytime Continual Learning for Open Vocabulary Classification
Zhen Zhu, Yiming Gong, Derek Hoiem
Open Vocabulary Low Rank Compression

June 13, 2024

AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers
Emil Biju, Anirudh Sriram, Mert Pilanci
Transformer Megatron Decepticons Transformer Based Model Transformer Attention Speaker Independent Whisper Model Low Rank Compression Adaptive Compression

May 28, 2024

Unified Low-rank Compression Framework for Click-through Rate Prediction
Hao Yu, Minghao Fu, Jiandong Ding, Yusheng Zhou, Jianxin Wu
Click Through Rate Prediction Low Rank Compression

May 24, 2024

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
Yang Li, Changsheng Zhao, Hyungtak Lee, Ernie Chang, Yangyang Shi, Vikas Chandra
Large Language Model Microbial Decomposition Pre Trained Large Language Model Inference Cost Low Rank Decomposition Multiple Target Low Rank Compression Deep Compression

May 17, 2024

Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang
Large Language Model Bayesian Optimization Low Rank Compression Low Rank Factorization

February 8, 2024

Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
Seyedarmin Azizi, Mahdi Nazemi, Massoud Pedram
Vision Transformer Model Compression Memory Efficient Low Rank Compression Tensor Approximation Rank Adaptive Tensor Optimization

January 18, 2024

ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks
Yang Sui, Miao Yin, Yu Gong, Jinqi Xiao, Huy Phan, Bo Yuan
Training Data Low Rank Low Rank Compression Low Rank Training

January 15, 2024

Low-Rank Gradient Compression with Error Feedback for MIMO Wireless Federated Learning
Mingzhao Guo, Dongzhu Liu, Osvaldo Simeone, Dingzhu Wen
Federated Learning Matrix Factorization Error Feedback Multiple Input Multiple Output Gradient Compression Air Computation Low Rank Compression

August 28, 2023

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Samuel Horvath, Stefanos Laskaridis, Shashank Rajput, Hongyi Wang
Low Rank DNN Architecture Simple Structure Low Rank Structure Learning Decomposition Low Rank Compression

June 9, 2023

Error Feedback Can Accurately Compress Preconditioners
Ionut-Vlad Modoranu, Aleksei Kalinov, Eldar Kurtic, Elias Frantar, Dan Alistarh
Error Feedback Adaptive Preconditioner Low Rank Compression Full Matrix AdaGrad

June 4, 2023

Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
Ye Xue, Vincent Lau
Model Compression Low Rank Low Dimensional Manifold Riemannian Geometry Approach Low Rank Compression Low Rank Constraint Air Aggregation

January 20, 2023

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks
Jinqi Xiao, Chengming Zhang, Yu Gong, Miao Yin, Yang Sui, Lizhi Xiang, Dingwen Tao, Bo Yuan
Model Compression Compact Neural Network Low Rank Compression

April 12, 2022

Compact Model Training by Low-Rank Projection with Energy Transfer
Kailing Guo, Zhenquan Lin, Xiaofen Xing, Fang Liu, Xiangmin Xu
Low Dimensional Manifold Projection Bias Network Compression Projection Matrix Heat Transfer Low Rank Compression

November 30, 2021

A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank
Moonjung Eo, Suhyun Kang, Wonjong Rhee
Deep Neural Network Stable Rank Compression Performance Low Rank Compression

Low Rank Compression

Papers

Anytime Continual Learning for Open Vocabulary Classification

AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers

Unified Low-rank Compression Framework for Click-through Rate Prediction

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications

Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization

Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy

ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks

Low-Rank Gradient Compression with Error Feedback for MIMO Wireless Federated Learning

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

Error Feedback Can Accurately Compress Preconditioners

Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

Compact Model Training by Low-Rank Projection with Energy Transfer

A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank