Low Rank

Low-rank techniques aim to reduce the computational cost and memory requirements of large-scale machine learning models by representing high-dimensional data or model parameters using lower-dimensional structures. Current research focuses on applying low-rank methods to improve the efficiency of large language models (LLMs) and other deep learning architectures, often through techniques like low-rank adaptation (LoRA) and its variants, as well as matrix and tensor factorization. These advancements are significant because they enable the training and deployment of larger and more powerful models on resource-constrained devices, improving performance in various applications such as natural language processing, computer vision, and recommendation systems. Furthermore, theoretical work is exploring the inherent low-rank properties of trained models to better understand and optimize these methods.

Papers

November 12, 2024

November 11, 2024

Beating Adversarial Low-Rank MDPs with Unknown Transition and Bandit Feedback
Haolin Liu, Zakaria Mhammedi, Chen-Yu Wei, Julian Zimmert
Human Feedback Low Rank Bandit Feedback Adversarial Text O$ Regret Bandit Identification Regret Rate Transition Uncertainty

November 4, 2024

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
Xingtai Lv, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, Bowen Zhou
Large Language Model Low Rank Attention Layer Scalable Training

October 27, 2024

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S. Dhillon, Cho-Jui Hsieh, Sanjiv Kumar
Optimization Purpose Low Rank Parameter Efficient Finetuning Balance Control

October 23, 2024

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning
Yehonathan Refael, Jonathan Svirsky, Boris Shustin, Wasim Huleihel, Ofir Lindenbaum
Large Language Model Fine Tuning Low Rank Low Rank Adaptation Temporal Moment Large Language Model Training

October 22, 2024

Sample-Efficient Geometry Reconstruction from Euclidean Distances using Non-Convex Optimization
Ipsita Ghosh, Abiy Tasissa, Christian Kümmerle
Low Rank Non Convex Optimization Sampling Based Geometry Reconstruction Euclidean Distance Point Pair

October 21, 2024

Simplicity Bias via Global Convergence of Sharpness Minimization
Khashayar Gatmiry, Zhiyuan Li, Sashank J. Reddi, Stefanie Jegelka
Stochastic Gradient Descent Low Rank Sharpness Aware Minimization Two Layer Neural Network Global Convergence Simplicity Bias

October 17, 2024

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Chuanyu Tang, Yilong Chen, Zhenyu Zhang, Junyuan Shang, Wenyuan Zhang, Yong Huang, Tingwen Liu
Fine Tuning Mixture Component Mixture of Expert Low Rank Stable Rank

October 15, 2024

LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models
Hossein Abdi, Mingfei Sun, Andi Zhang, Samuel Kaski, Wei Pan
Machine Learning Parameter Efficient Fine Tuning Large Model Low Rank Kalman Filter Low Rank Decomposition Online Fine Tuning

October 10, 2024

The Plug-in Approach for Average-Reward and Discounted MDPs: Optimal Sample Complexity Analysis
Matthew Zurek, Yudong Chen
Markov Decision Process Sample Complexity Optimal Policy Low Rank Average Reward Markov Decision Process

October 5, 2024

LoRTA: Low Rank Tensor Adaptation of Large Language Models
Ignacio Hounie, Charilaos Kanatsoulis, Arnuv Tandon, Alejandro Ribeiro
Parameter Efficient Fine Tuning Low Rank Low Rank Adaptation Low Rank Tensor Fast LoRA Low Rank Tensor Model

October 2, 2024

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, Guoren Wang
Large Language Model Low Rank Grand Challenge Low Rank Constraint Low Rank Training

October 1, 2024

GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction
Zaid Ilyas, Naveed Akhtar, David Suter, Syed Zulqarnain Gilani
Self Attention Image Restoration Low Rank Multi Head Self Attention Hybrid CNN Transformer Spectral Reconstruction

September 29, 2024

Tailed Low-Rank Matrix Factorization for Similarity Matrix Completion
Changyi Ma, Runsheng Yu, Xiao Chen, Youzhi Zhang
Matrix Factorization Low Rank Matrix Completion Low Rank Regularization

September 25, 2024

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang
Low Rank LLM Fine Tuning Skeleton Sequence Skeleton Representation Rank One Update Multilingual Sequence to Sequence

September 19, 2024

Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization
Haemin Park, Diego Klabjan
Low Rank Implicit Regularization New Connection Parameter Update

September 14, 2024

Tensor-Based Synchronization and the Low-Rankness of the Block Trifocal Tensor
Daniel Miao, Gilad Lerman, Joe Kileel
Low Rank Tensor Based Order Tensor View Geometry High Dimensional Tensor

September 4, 2024

Low Rank

Papers

FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Beating Adversarial Low-Rank MDPs with Unknown Transition and Bandit Feedback

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

Sample-Efficient Geometry Reconstruction from Euclidean Distances using Non-Convex Optimization

Simplicity Bias via Global Convergence of Sharpness Minimization

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning

LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models

The Plug-in Approach for Average-Reward and Discounted MDPs: Optimal Sample Complexity Analysis

LoRTA: Low Rank Tensor Adaptation of Large Language Models

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction

Tailed Low-Rank Matrix Factorization for Similarity Matrix Completion

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning

Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization

Tensor-Based Synchronization and the Low-Rankness of the Block Trifocal Tensor

Few-shot Multi-Task Learning of Linear Invariant Features with Meta Subspace Pursuit

Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering