Low Rank Adaptation
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique for large pre-trained models, aiming to reduce computational costs and memory requirements while maintaining performance on downstream tasks. Current research focuses on improving LoRA's efficiency and effectiveness through methods like tensor decomposition, adaptive parameter allocation, and novel aggregation strategies for federated learning scenarios, often applied to transformer-based language and vision models. This approach holds significant promise for making large model fine-tuning more accessible and enabling the development of personalized and specialized models across diverse applications with limited resources.
Papers
Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models
Navyansh Mahla, Kshitij Sharad Jadhav, Ganesh Ramakrishnan
Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Wei Dong, Yuan Sun, Yiting Yang, Xing Zhang, Zhijun Lin, Qingsen Yan, Haokui Zhang, Peng Wang, Yang Yang, Hengtao Shen
CopRA: A Progressive LoRA Training Strategy
Zhan Zhuang, Xiequn Wang, Yulong Zhang, Wei Li, Yu Zhang, Ying Wei
Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients
Jabin Koo, Minwoo Jang, Jungseul Ok
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu
Closed-form merging of parameter-efficient modules for Federated Continual Learning
Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Jacopo Bonato, Luigi Sabetta, Simone Calderara
AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning
Yehonathan Refael, Jonathan Svirsky, Boris Shustin, Wasim Huleihel, Ofir Lindenbaum