Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
Sep 8, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamLLM Fine-Tuning Low-Rank Adaptation
Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA
MI-PEFT: Mixture-of-Experts Integrated Parameter-Efficient Fine-Tuning Protein Language Models Improves Acidophilic Proteins Classification
Sep 7, 2026Honghan ShenMixture of Experts Protein Language Models
University of Wisconsin–Madison, Madison, WI 53706, USA
Event Interaction in Low-Rank Bottlenecks for Temporal Relation Extraction
Sep 6, 2026Wei Sun, Tingyu Qu, Jesse Davis +1Relation Extraction Feature Interaction Modeling
Department of Computer Science, KU Leuven Celestijnenlaan 200A 3001 Heverlee, Belgium
Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis
Sep 6, 2026Xiaoran Kou, Jingyi Wu, Peng Sun +2Multimodal Sentiment Analysis Prompt Tuning
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China
One Shared LoRA Weight for MRI Reconstruction across Acceleration Factors
Sep 6, 2026Zhiwei Zhao, Weikang Gong, Zhongnian Li +1Accelerated MRI Reconstruction Low-Rank Adaptation
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, Xuzhou, China
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Sep 5, 2026Ahin Lee, Sehyun Yun, Joonha Park +1Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning
Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
Sep 5, 2026Huiyi Wang, Daijiao Liu, Lina Yao +1LLM Fine-Tuning Low-Rank Adaptation
University of New South Wales
CoRe-SAM3: Conditional Semantic--Visual Reconciliation for SAM3 Crack Segmentation
Sep 5, 2026Shipeng Liu, Liang Zhao, Dengfeng ChenCrack Segmentation Domain Adaptation
School of Mechanical and Electrical Engineering, Xi’an University of Architecture and Technology, Xi’an, 710055, Shaanxi, China · School of Artificial Intelligence and Robotics, Xi’an University of Architecture and Technology, Xi’an, 710055, Shaanxi, China · School of Building Equipment Science and Engineering, Xi’an University of Architecture and Technology, Xi’an, 710055, Shaanxi, China
KanAdapter: A Kolmogorov-Arnold Network-based Plug-and-Play Module for Efficient Fine-tuning of Foundation Speech Models
Sep 4, 2026Phuong Tuan Dat, Phuong Khai Minh, Tran Huy DatAdapter Tuning Parameter-Efficient Fine-Tuning
Dept. of Electrical and Computer Engineering, National University of Singapore · School of Computing, National University of Singapore · Institute of Advanced Intelligence and Computing (IAIC), A∗STAR
Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLM
Sep 4, 2026Xinyu Li, Ruoming Jin, Jianfeng Zhu +2LLM Personalization Low-Rank Adaptation
Department of Computer Science, Kent State University · iLambda Inc.
PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspaces
Sep 4, 2026Xinyu Li, Hao Zhou, Jianfeng Zhu +4LLM Personalization Low-Rank Adaptation
Department of Computer Science Kent State University Kent, OH 44242, USA
SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution
Sep 3, 2026Federico Putamorsi, Leonardo Zini, Marcella Cornia +1Diffusion Models for Image Restoration Diffusion Transformer
University of Modena and Reggio Emilia, Italy
Choosing a PEFT Variant for Per-Patient Dysarthric ASR: A Single-Speaker Case Study on Two ASR Bases
Sep 2, 2026Bernard Muller, László Tóth, LaVonne RobertsAutomatic Speech Recognition Parameter-Efficient Fine-Tuning
The Scott-Morgan Foundation, Torquay, United Kingdom · Institute of Informatics, University of Szeged, Szeged, Hungary
LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
Sep 2, 2026Dmitrii Andriianov, Andrey Veprikov, Aleksandr BeznosikovLow-Rank Adaptation Parameter-Efficient Fine-Tuning
BRAIn Lab · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab +1
TaRA: Training-Aware Low-Rank Adaptation Initialization
Sep 2, 2026Taehyeon Kim, Eunhyeok ParkLow-Rank Adaptation Parameter-Efficient Fine-Tuning
Department of Computer Science and Engineering · Pohang University of Science and Technology (POSTECH) · Graduate School of Artificial Intelligence
RouteGraph-Mona: Confusion-Aware Routing Fine-Tuning for Mineral Image Classification
Sep 2, 2026Jierui Li, Zhiyuan Qi, Hao Zhu +7Image Classification Parameter-Efficient Fine-Tuning
Xidian University, Xi’an, China · Tsinghua University, Beijing, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), China
Compositional Spectral Prompts for LLM-based Online Time Series Forecasting
Sep 2, 2026Seungyoon Choi, Hyunchul Kim, Jae-Gil Lee +1Online Time Series Forecasting Time Series Forecasting
KAIST Daejeon, Republic of Korea
High-Value Synthetic Supervision for Parameter-Efficient Adaptation of a Compact Japanese Speech Model
Sep 2, 2026Sidi Chang, Peiying ZhuParameter-Efficient Fine-Tuning Low-Rank Adaptation
Blossom AI Labs Blossom AI Tokyo, Japan
Post-Training Science for Supervised Fine-Tuning
Sep 1, 2026Charles O'Neill, Mudith Jayasekara, Harry PartridgeSupervised Fine-Tuning Hyperparameter Optimization
1Baseten
QINA: Quantum-Inspired Nonlinear Adapters for Pretrained Vision Models
Sep 1, 2026Mostafa Mehdipour GhaziVision Foundation Model Adaptation Representation Learning
Pioneer Centre for AI University of Copenhagen
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
Sep 1, 2026Wentao Ye, Zhanming Shen, Zhiqing Xiao +3Memory-Efficient Fine-Tuning LLM Fine-Tuning
Zhejiang University · Hunyuan, Tencent
SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation
Aug 31, 2026P. Malaisree, S. Youwai, S. Janrungautai +3Promptable Image Segmentation Vision Foundation Model Adaptation
MAA Consultants Co., Ltd. · AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut´s University of Technology Thonburi (KMUTT). · School of Engineering, University of Phayao. +2
CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction
Aug 31, 2026Zhengxu Tang, Guofeng Cui, Ziyu Gong +8Autonomous Driving Benchmarks VLMs for Autonomous Driving
NVIDIA
One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
Aug 31, 2026Yunxiang Fu, Meng Lou, Yizhou YuClass-Incremental Learning Incremental Learning
School of Computing and Data Science The University of Hong Kong
Normalized Low-Rank Adaptation
Aug 31, 2026Jiale Kang, Ziyin Yue, Zheng Zhan +2Low-Rank Adaptation Parameter-Efficient Fine-Tuning
1Yuanshi Intelligence · 3The Chinese University of Hong Kong · 2Microsoft Research +1
GAFT: Geo-Anchored Fine-Tuning for Hazard Identification from Rare Failures
Aug 31, 2026Yanran Xu, Chuanhang Qiu, Yue Wang +2Robot Navigation Vision Foundation Model Adaptation
University of Southampton, Southampton, United Kingdom
RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks
Aug 31, 2026Xingran Chen, Rohit Bhagat, Ghadir Ayache +3Fine-Tuning Communication-Efficient Distributed Training
Engineering Systems and Design Pillar, Singapore University of Technology and Design, Singapore 487372 · Department of Electrical and Computer Engineering, Rutgers University, Piscataway Township, NJ 08854, USA · LinkedIn, New York, NY 10118, USA +2
HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
Aug 30, 2026Hyeonjik Bae, Minyeol Kim, Susik YoonMulti-Task Learning Prompt Tuning
Computer Science and Engineering Korea University, Seoul, Korea
When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization
Aug 29, 2026Jiayu Hou, Lei WangDomain Adaptation for LLMs Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China, Chengdu, China
Explainable Diabetic Retinopathy Classification Using Vision Foundation Models
Aug 28, 2026Abhishek Verma, Anila Krishna, Abhishek Gajanan Bankar +1Retinal Disease Classification Medical Image Classification
Department of Health Services Research, Carl von Ossietzky Universität Oldenburg Oldenburg, Germany

Sep 8, 2026J/K move · Enter open · S save

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Ashfak Yeafi, Mehedi Hasan, Md Khairul Islam

Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) to downstream tasks while updating only a small fraction of their pretrained parameters. Low-Rank Adaptation (LoRA) uses two trainable low-rank matrices, while Weight-Decomposed Low-Rank Adaptation (DoRA) further separates weight magnitude and direction but retains the dense LoRA-style factorization in its directional branch. We propose ChainDoRA, a weight-decomposed adaptation framework that constructs the directional low-rank factors from a connected Tensor-Train (TT) chain, where the adapter rank forms the boundary rank between input- and output-side TT contractions and an independent TT rank controls representation capacity and parameter cost. Under a controlled 15,119-example response-only adaptation setting with LLaMA-7B, ChainDoRA is evaluated against matched LoRA and DoRA baselines on seven commonsense reasoning benchmarks. ChainDoRA with TT rank 16 achieves a seven-task average accuracy of 72.30%, compared with 69.88% for LoRA and 69.39% for DoRA, while requiring only 5.35M trainable parameters versus 56.10M for LoRA and 56.98M for DoRA, corresponding to a 90.62% reduction relative to DoRA. Ablations over TT rank and adapter placement show controllable parameter-accuracy trade-offs, indicating that connected TT parameterization can substantially reduce the parameter cost of magnitude-direction adaptation while preserving, and in this setting improving, downstream reasoning performance.