LoRA Based

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique for large language models (LLMs) and other deep learning architectures, primarily aiming to reduce computational costs and memory requirements while maintaining or improving performance on downstream tasks. Current research focuses on improving LoRA's efficiency through methods like orthogonal initialization (SBoRA), dynamic rank selection (QDyLoRA), and heterogeneous rank allocation (HetLoRA) across distributed settings, as well as addressing challenges such as overfitting (LoRA Dropout) and adversarial attacks. These advancements are significant for deploying LLMs on resource-constrained devices and enabling more efficient and scalable model adaptation across diverse applications, including medical image analysis and federated learning.

Papers

October 22, 2024

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, Xiaojie Wang
Large Language Model Natural Language Processing Training Data Catastrophic Forgetting Adaptation Concern Subspace Learning Finetuning Method LoRA Based

July 16, 2024

LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors
Guanghua He, Wangang Cheng, Hancan Zhu, Gaohang Yu
X Vector Low Rank Tensor Fast LoRA Transformer Based UNet Hippocampus Segmentation LoRA Based Tensor Singular Value

July 7, 2024

SBoRA: Low-Rank Adaptation with Regional Weight Updates
Lai-Man Po, Yuyang Liu, Haoxuan Wu, Tianqi Zhang, Wing-Yin Yu, Zhuohan Wang, Zeyu Jiang, Kun Li
Adaptation Concern Low Rank Adaptation Weight Update Efficient Adaptation Memory Efficient Fine Tuning LoRA Based

June 24, 2024

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, Fei Wu
Large Language Model Retrieval Augmented Low Rank Adaptation Domain Specific Adapter LoRA Based

April 21, 2024

Mixture of LoRA Experts
Xun Wu, Shaohan Huang, Furu Wei
Natural Language Processing Mixture Component LoRA Fine Tuning Fast LoRA LoRA Based

April 15, 2024

LoRA Dropout as a Sparsity Regularizer for Overfitting Control
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, Hong Mei
Model Overfitting Safe LoRA Sparsity Regularization LoRA Based

March 18, 2024

Continual Forgetting for Pre-trained Vision Models
Hongbo Zhao, Bolin Ni, Haochen Wang, Junsong Fan, Fei Zhu, Yuxi Wang, Yuntao Chen, Gaofeng Meng, Zhaoxiang Zhang
Pre Trained Anti Forgetting LoRA Fine Tuning Run LoRA Run LoRA Based Group Sparse

February 29, 2024

LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario
Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, Xia Hu
LLM Safety LoRA Fine Tuning LoRA Based Transferable Backdoor Attack Collaborative Game

February 16, 2024

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
Large Language Model Adaptation Concern Low Rank Adaptation LoRA Based

January 12, 2024

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi
Fine Tuning Private Fine Tuning Device Model LoRA Module LoRA Based

November 14, 2023

$DA^3$: A Distribution-Aware Adversarial Attack against Language Models
Yibo Wang, Xiangjue Dong, James Caverlee, Philip S. Yu
Language Model Adversarial Attack Adversarial Example Attack Success Rate LoRA Based