Parameter Efficient Fine Tuning

Parameter-efficient fine-tuning (PEFT) aims to adapt large pre-trained models to specific downstream tasks while minimizing the number of trainable parameters, thus reducing computational costs and memory requirements. Current research focuses on improving the efficiency and effectiveness of PEFT methods, exploring techniques like low-rank matrix and tensor decompositions (e.g., LoRA, its variants, and tensor-based adaptations), selective layer training, and novel parameter initialization strategies. These advancements are significant because they enable the deployment of large language models and other foundation models on resource-constrained devices and facilitate more efficient and sustainable model adaptation for diverse applications.

Papers

October 23, 2024

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu
Parameter Efficient Fine Tuning Adaptation Concern Mixture of Expert Large Language Low Rank Adaptation

October 22, 2024

FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation
Rohan Sukumaran, Aarash Feizi, Adriana Romero-Sorian, Golnoosh Farnadi
Fine Tuning Parameter Efficient Fine Tuning Adaptation Concern Vision Model Bias Mitigation Fairness Metric Fast LoRA

October 21, 2024

The effect of fine-tuning on language model toxicity
Will Hawkins, Brent Mittelstadt, Chris Russell
Language Model Fine Tuning Mixed Effect Parameter Efficient Fine Tuning Efficient Fine Tuning

October 20, 2024

MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models
Ahmed Elbakary, Chaouki Ben Issaid, Tamer ElBatt, Karim Seddik, Mehdi Bennis
Fine Tuning Full Model Multi Task Learning Practical Method Parameter Efficient Fine Tuning Large Language Federated Approach Federated Multi Task Learning

October 17, 2024

LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
Yiming Shi, Jiwei Wei, Yujia Wu, Ran Ran, Chengwei Sun, Shiyuan He, Yang Yang
Fine Tuning Parameter Efficient Fine Tuning Adaptation Concern Low Rank Matrix Low Rank Decomposition

October 16, 2024

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, Ramtin Pedarsani
Language Model Fine Tuning Parameter Efficient Fine Tuning

October 15, 2024

October 14, 2024

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates
Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Mojtaba Soltanalian, Niloofar Yousefi
Language Model High Efficiency Parameter Efficient Fine Tuning Neural Tangent Kernel Information System Update Separable Kernel

October 13, 2024

BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation
Peijia Qin, Ruiyi Zhang, Pengtao Xie
Large Language Model Language Generation Parameter Efficient Fine Tuning Adaptation Concern Multiple Level

October 11, 2024

October 9, 2024

October 8, 2024

PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches
Rana Muhammad Shahroz Khan, Pingzhi Li, Sukwon Yun, Zhenyu Wang, Shahriar Nirjon, Chau-Wai Wong, Tianlong Chen
Large Language Model Fine Tuning Domain Knowledge Parameter Efficient Fine Tuning Training Free Domain Specific Task

October 5, 2024

LoRTA: Low Rank Tensor Adaptation of Large Language Models
Ignacio Hounie, Charilaos Kanatsoulis, Arnuv Tandon, Alejandro Ribeiro
Parameter Efficient Fine Tuning Low Rank Low Rank Adaptation Low Rank Tensor Fast LoRA Low Rank Tensor Model

Parameter Efficient Fine Tuning

Papers

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation

The effect of fine-tuning on language model toxicity

MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models

LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models

LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation

Parameter-Efficient Fine-Tuning of State Space Models

QEFT: Quantization for Efficient Fine-Tuning of LLMs

Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning

Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures

SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers

Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform

PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches

LoRTA: Low Rank Tensor Adaptation of Large Language Models