Parameter-Efficient Fine-Tuning

Also known as PEFT

Overview

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

CardsList

646 papers42 papers in the last four weeks, down 51% on the four weeks before. 0.3% of all new papers.

  1. On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs

    Sep 22, 2025Rongguang Ye, Ming Tang, Edith C. H. NgaiMixed-Precision QuantizationLLM Quantization

  2. When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts

    Aug 31, 2025Anastasis Kratsios, Tin Sum Cheng, Bum Jun Kim +1Fine-TuningOOD Generalization

  3. From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning

    Aug 7, 2025Jinda Liu, Yi Chang, Yuan WuMulti-Task LearningLow-Rank Adaptation

  4. FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data

    Aug 6, 2025Thibaut Thonet, Germán Kruszewski, Jos Rozen +2LLM PersonalizationPersonalized Language Models

  5. NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding

    Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal LearningVLM Adaptation

  6. PLoRA: Efficient Concurrent LoRA Training for Large Language Models

    Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2LLM Fine-TuningEfficient Language Model Training

  7. WaRA: Wavelet Low-Rank Adaptation for Medical Image Classification

    Jun 25, 2025Moein Heidari, Yijin Huang, Yasamin Medghalchi +3Vision Foundation Model AdaptationLow-Rank Adaptation

  8. TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

    Jun 12, 2025Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang +2Continual Learning for LLMsLow-Rank Adaptation

  9. MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning

    Jun 10, 2025Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray +3TransformerTensor Decomposition

  10. COLORA: Efficient Fine-Tuning for Convolutional Models with a Study Case on Optical Coherence Tomography Image Classification

    May 23, 2025Mariano Rivera, Angello HoyosLow-Rank AdaptationConvolutional Neural Networks

  11. Freeze, Prompt, and Adapt: A Framework for Source-free Unsupervised GNN Prompting

    May 22, 2025Peyman Baghershahi, Sourav MedyaSource-Free Domain AdaptationUnsupervised Domain Adaptation

  12. Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training

    Feb 5, 2025Reza Shirkavand, Peiran Yu, Qi He +1LLM Fine-TuningZeroth-Order Optimization

  13. BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models

    Aug 8, 2024Yupeng Chang, Yi Chang, Yuan WuSocial Bias in Language ModelsLow-Rank Adaptation

  14. Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models

    Jul 28, 2024Jifeng Wang, Kaouther Messaoud, Yuejiang Liu +2ForecastingParameter-Efficient Fine-Tuning

  15. Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models

    Jan 1, 2024Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai +4LLM Fine-TuningInstruction Tuning