Low Rank Adaptation

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique for large pre-trained models, aiming to reduce computational costs and memory requirements while maintaining performance on downstream tasks. Current research focuses on improving LoRA's efficiency and effectiveness through methods like tensor decomposition, adaptive parameter allocation, and novel aggregation strategies for federated learning scenarios, often applied to transformer-based language and vision models. This approach holds significant promise for making large model fine-tuning more accessible and enabling the development of personalized and specialized models across diverse applications with limited resources.

Papers

September 30, 2024

September 24, 2024

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, Fei Wu
Large Language Model Low Rank Adaptation Modularity Function Semantic Unit

September 22, 2024

Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape
Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, Xiaolin Huang
Parameter Efficient Fine Tuning Adaptation Concern Generalization Performance Low Rank Adaptation Large Scale Pre Trained Model

September 16, 2024

September 13, 2024

Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights
Dixi Yao
Diffusion Model Generative Model Supervised Autoencoder Low Rank Adaptation Risk Sensitive Fine Tuned Model Variational AutoEncoder

August 30, 2024

DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model
Mona Sheikh Zeinoddin, Chiara Lena, Jiongqi Qu, Luca Carlini, Mattia Magro, Seunghoi Kim, Elena De Momi, Sophia Bano, Matthew Grech-Sollars, Evangelos Mazomenos, Daniel C. Alexander, Danail Stoyanov, Matthew J. Clarkson, Mobarakol Islam
Foundation Model Depth Estimation Low Rank Adaptation Depth Anything Depth Perception

August 19, 2024

Customizing Language Models with Instance-wise LoRA for Sequential Recommendation
Xiaoyu Kong, Jiancan Wu, An Zhang, Leheng Sheng, Hui Lin, Xiang Wang, Xiangnan He
Sequential Recommendation Low Rank Adaptation Sequential Recommender System Recommendation Accuracy

August 18, 2024

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
Cheng Lin, Lujun Li, Dezhi Li, Jie Zou, Wei Xue, Yike Guo
Vision Language Model Fine Tuning Parameter Efficient Fine Tuning Adaptation Concern Low Rank Adaptation Task Adaptation Nested Entity

August 16, 2024

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS
Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, Albert Y. Zomaya
Federated Learning Low Rank Adaptation Heterogeneous Model Rank Aggregation

August 13, 2024

August 2, 2024

Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration
Donwon Park, Hayeon Kim, Se Young Chun
Full Model Image Restoration Adaptation Concern Low Rank Adaptation

July 24, 2024

Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
Ao Shen, Qiang Wang, Zhiquan Lai, Xionglve Li, Dongsheng Li
Large Language Model Low Rank Adaptation Fine Tuned Large Language Model Dynamic Balance

July 22, 2024

July 16, 2024

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach
Yuzhu Mao, Siqi Ping, Zihao Zhao, Yang Liu, Wenbo Ding
Large Language Model Strong Generalization Large Pre Trained Model Low Rank Adaptation Parameter Efficient Large Scale Model Gradient Masking

July 12, 2024

FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
Kai Zhao, Zhaohui Yang, Chongwen Huang, Xiaoming Chen, Zhaoyang Zhang
Large Language Model Low Rank Adaptation Split Learning Computer Network Federated Split Learning Splitfed Learning

Low Rank Adaptation

Papers

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models

UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

A Bayesian Interpretation of Adaptive Low-Rank Adaptation

Robust Bird's Eye View Segmentation by Adapting DINOv2

jina-embeddings-v3: Multilingual Embeddings With Task LoRA

Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights

DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model

Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS

DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion

Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models

Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration

Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance

Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters

Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

FedsLLM: Federated Split Learning for Large Language Models over Communication Networks