Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
May 3, 2026Zongqian Li, Yixuan Su, Han Zhou +2Low-Rank Adaptation Bayesian LoRA
University of Cambridge
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
May 2, 2026Zhiwen Ruan, Yichao Du, Jianjie Zheng +6LLM Fine-Tuning Large Language Model-Guided Optimization
Southern University of Science and Technology · Alibaba Group · Shanghai University of Finance and Economics +2
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
May 1, 2026Zhi-Quan Feng, Ying-Jia Lin, Hung-Yu KaoLLM Fine-Tuning Low-Rank Adaptation
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Department of Computer Science, National Tsing Hua University
DIYHealth Suite: Dataset, Model, and Benchmark for Health Management at Home
May 1, 2026Changshuo Liu, Junran Wu, Zhongle Xie +9Healthcare Multimodal Large Language Models
School of Computing, National University of Singapore, Singapore
Parameter-Efficient Adaptation of Pre-Trained Vision Foundation Models for Active and Passive Seismic Data Denoising
Apr 30, 2026Jiahua Zhao, Umair bin Waheed, Jing Sun +3Test-Time Adaptation Vision Foundation Model Adaptation
Computation-based Science and Technology Research Center, The Cyprus Institute, Nicosia, Cyprus · Department of Geosciences, College of Petroleum Engineering and Geosciences, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia · Department of Intelligent Systems, Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, Netherlands +2
Post-Optimization Adaptive Rank Allocation for LoRA
Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank Adaptation Low-Rank Compression
1Indian Institute of Technology Hyderabad · 2Deakin University
BoostLoRA: Growing Effective Rank by Boosting Adapters
Apr 30, 2026Raviteja Anantha, Nick Levato, Layne C. PriceLLM Fine-Tuning Adapter Tuning
Amazon
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China
BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
Apr 27, 2026Dario Coscia, Sindy Löwe, Max WellingBayesian Neural Networks Uncertainty Quantification
mathLab, SISSA · CuspAI
Aligned Multi-View Scripts for Universal Chart-to-Code Generation
Apr 27, 2026Zhihan Zhang, Lizi LiaoCode Generation Parameter-Efficient Fine-Tuning
School of Computing and Information Systems, Singapore Management University
Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing
Apr 26, 2026Honghao Cai, Xiangyuan Wang, Yunhao Bai +8Diffusion Transformer Text-Guided Image Editing
The Chinese University of Hong Kong, Shenzhen · Xiaohongshu Inc. · Peking University +2
HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
Apr 26, 2026Francesco Dibitonto, Cigdem Beyan, Vittorio MurinoVisual Question Answering Zero-Shot Learning
Department of Computer Science, University of Verona, Verona, Italy · EVS - Embedded Vision Systems Srl, Verona, Italy · AI for Good (AIGO), Istituto Italiano di Tecnologia, Genova, Italy
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Low-Rank Adaptation Redux for Large Models
Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisAdapter Tuning Low-Rank Adaptation
Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA
GiVA: Gradient-Informed Bases for Vector-Based Adaptation
Apr 23, 2026Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
University of Illinois Urbana-Champaign · Amazon · Stanford University
Formalising the Logit Shift Induced by LoRA: A Technical Note
Apr 22, 2026Xiang Shi, Shuaizhi Cheng, Mingwei LiLow-Rank Adaptation Parameter-Efficient Fine-Tuning
Imperial College London · Harbin Institute of Technology · KigLand Learning Lab
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models
Apr 22, 2026Sachin KumarSmall Language Models Few-Shot Prompting
LexisNexis, USA
RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models
Apr 21, 2026Yusuf Çelebi, Yağız Asker, Özay Ezerceli +4LLM Fine-Tuning Low-Rank Adaptation
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Apr 21, 2026Xianming Li, Zongxi Li, Tsz-fung Andrew Lee +3Fine-Tuning Adapter Tuning
PolyU · School of Data Science, Lingnan University, Hong Kong SAR
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
Apr 21, 2026Boyan Shi, Wei Chen, Shuyuan Zhao +4Multi-Task Learning Low-Rank Adaptation
School of Computer Science and Technology, Beijing Jiaotong University, China · 2Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China · 3Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China +2
Spike-NVPT: Learning Robust Visual Prompts via Bio-Inspired Temporal Filtering and Discretization
Apr 20, 2026Qiugang Zhan, Anning Jiang, Ran Tao +4Vision Foundation Model Adaptation Visual Prompt Learning
Complex Laboratory of New Finance and Economics, Southwestern University of Finance and Economics · Engineering Research Center of Intelligent Finance, Ministry of Education · School of Computer Science and Engineering, 4Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China +3
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
Apr 20, 2026Hyeonseo Jang, Hyuk Kwon, Kibok LeeClass-Incremental Learning Vision-Language Models
Yonsei University
JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew
Apr 20, 2026Itay Razumenko, Arnon Sturm, Nir GrinbergLegal NLP LLM Personalization
Computer and Information Science Ben-Gurion University of the Negev Beer-Sheva, Israel
GraphLoRA: Structure-Aware Low-Rank Adaptation for Large Language Model Recommendation
Apr 20, 2026Lin Mu, Guoji Wang, Li Ni +4Low-Rank Adaptation Large Language Model-Based Recommendation
Anhui University · Hefei University · University of Science and Technology of China
HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
Apr 20, 2026Lixian Chen, Jianhong TanContinual Learning for LLMs Low-Rank Adaptation
Guangdong University of Technology
Representation-Guided Parameter-Efficient LLM Unlearning
Apr 19, 2026Zeguan Xiao, Lang Mo, Yun Chen +4Machine Unlearning LLM Unlearning
Shanghai University of Finance and Economics · Southern University of Science and Technology · Deepexi Technology Co. Ltd.
Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization
Apr 18, 2026Weijie Wan, Jiangjiang ZhaoContinual Learning for LLMs LLM Fine-Tuning
Shortest Path Technology · China Mobile Communications Group Co.,Ltd
A Benchmark Study of Segmentation Models and Adaptation Strategies for Landslide Detection from Satellite Imagery
Apr 17, 2026Md Kowsher, Weiwei Zhan, Chen ChenRemote Sensing Image Segmentation Vision Foundation Model Adaptation
University of Central Florida
Cross-Attentive Bayesian Low-Rank Adaptation for Multimodal Uncertainty Estimation
Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinLow-Rank Adaptation Uncertainty-Aware Multimodal Fusion
Dalhousie University, Halifax NS, Canada

May 3, 2026J/K move · Enter open · S save

Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks

Zongqian Li, Yixuan Su, Han Zhou, Zihao Fu+1

University of Cambridge

Low-Rank Adaptation Bayesian LoRA Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become essential for deploying large language models, yet their static parameter allocation remains suboptimal for inputs of varying complexity. We present Flexi-LoRA, a novel framework that dynamically adjusts LoRA ranks based on input complexity during both training and inference. Through empirical analysis across question answering, mathematical reasoning, and speech tasks, we demonstrate that maintaining consistency between training and inference dynamics is important for effective adaptation, particularly for sequential reasoning tasks. Our findings reveal that input-dependent parameter allocation achieves higher performance with fewer parameters by optimally matching rank configurations to question complexity. Furthermore, task-specific dependency on rank dynamics varies, with mathematical reasoning tasks exhibiting higher dependency than QA tasks. Successful adaptation manifests not only in correctness but also in reasoning quality and instruction adherence. Flexi-LoRA consistently outperforms static LoRA while using fewer parameters, with performance gains more pronounced on tasks requiring strict reasoning chains. Our approach realizes key benefits of mixture-of-experts frameworks through a more streamlined implementation, reducing parameter redundancy while improving model capabilities. We provide comprehensive empirical studies across diverse tasks, establishing a basis for future work in input-adaptive and efficient fine-tuning approaches.