Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
May 20, 2026Yongkang Liu, Xing Li, Mengjie Zhao +7LLM Fine-Tuning Adapter Tuning
Northeastern University, China · Shandong University, China · CIS, LMU Munich, Germany +1
FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
May 19, 2026Yequan Zhao, Ruijie Zhang, Liyan Tan +3LLM Fine-Tuning Parameter-Efficient Fine-Tuning
University of California at Santa Barbara · Amazon Lab126
AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
May 19, 2026Ziye Chen, Hongbin Lin, Chenyu Zhang +3Memory-Efficient Fine-Tuning Zeroth-Order Optimization
Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures
May 19, 2026Zeyao Liu, Zhendong Zhao, Xiaojun Chen +3Adversarial Attacks Vision Transformer
Institute of Information Engineering, Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense · School of Cyber Security, University of Chinese Academy of Sciences +1
iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models
May 19, 2026Xuezhi Cui, Dongbo Zhou, Wang Guo +8Vision-Language Models VLM Adaptation
School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · Faculty of Artifi cial Intelligence in Education in Central China Normal University, Wuhan 430079, China · School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology, Xiangtan 411201, China
Learning When to Adapt
May 18, 2026Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1LLM Fine-Tuning Low-Rank Adaptation
1Universität des Saarlandes · 2CISPA Helmholtz Center for Information Security
PERL: Parameter Efficient Reasoning in CLIP Latent Space
May 18, 2026Simone Carnemolla, Salvatore Calcagno, Daniela Giordano +2VLM Adaptation Few-Shot Learning
University of Catania
Unleashing Vision Transformer Potential In Image Quality Assessment via Global-Local Adaptive Interaction
May 18, 2026Yu Li, Puchao Zhou, Yachun Mi +3Vision Transformer Vision Foundation Model Adaptation
Harbin Institute of Technology · Meituan
TSFLora: Token-Compressed Split Fine-Tuning for Wireless Edge Networks
May 17, 2026Xianke Qiang, Zheng Chang, Li Wang +1Communication-Efficient Distributed Training Edge Computing
School of Computer Science and Engineering, UESTC, 611731 Chengdu, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China. · Center for Intelligent Networking and Communications, UESTC, 611731 Chengdu, China.
Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons
May 16, 2026Wei Tang, Jinpei Han, Kangning Cui +10Electrocardiography Electrocardiogram Foundation Models
1City University of Hong Kong · 2Imperial College London · 3Wake Forest University +3
FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
May 16, 2026Ramakrishnan SathyavageeswaranAdapter Tuning Low-Rank Adaptation
Intuit
GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining
May 16, 2026Guanghua He, Hancan Zhu, Gaohang Yu +1Medical Diagnosis Low-Rank Adaptation
Department of Mathematics, Hangzhou Dianzi University, Hangzhou, 310018, China · School of Mathematics, Physics and Information, Shaoxing2026 University, Shaoxing, 312000, China · Department of Mathematics, Zhejiang University of Science and Technology, Hangzhou, 310023, China
LoCO: Low-rank Compositional Rotation Fine-tuning
May 15, 2026An Nguyen, Jaesik Choi, Anh TongFine-Tuning LLM Fine-Tuning
1Korea University · 2KAIST · 3INEEJI
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1LLM Fine-Tuning Low-Rank Adaptation
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China · School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China +1
Parameter Efficient Multi-Class Intelligent Scheduling for Multimodal Online Distributed Industrial Anomaly Detection
May 15, 2026Heqiang Wang, Weihong Yang, Zheyuan Yang +4Industrial Anomaly Detection Multimodal Anomaly Detection
Pengcheng Laboratory, Shenzhen, 518066, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China
GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning
May 14, 2026Paolo Mandica, Michał Brzozowski, Zuzanna Dubanowska +1LLM Fine-Tuning Low-Rank Adaptation
Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland
PreFT: Prefill-only finetuning for efficient inference
May 14, 2026Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4Fine-Tuning LLM Serving
Stanford University · Tilde Research
PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts
May 13, 2026Anjir Ahmed Chowdhury, Syed Zawad, Xiaolong Ma +2LLM Fine-Tuning Multi-Task Learning
Department of Computer Science University of Houston · IBM Research · Argonne National Laboratory
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
May 13, 2026Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris +3LLM Fine-Tuning Non-IID Federated Learning
Continual Fine-Tuning of Large Language Models via Program Memory
May 13, 2026Hung Le, Svetha VenkateshContinual Learning for LLMs Memory-Augmented Language Models
Deakin Applied AI Initiative, Deakin University, Australia
Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition
May 13, 2026Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon +2ASR Evaluation Speech Recognition Benchmarks
Adalat AI, India
Output Composability of QLoRA PEFT Modules for Plug-and-Play Attribute-Controlled Text Generation
May 12, 2026Michela Lorandi, Anya BelzControllable Text Generation Parameter-Efficient Fine-Tuning
ADAPT, Dublin City University Dublin, Ireland
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
May 12, 2026Chengqian Zhang, Wei Zhu, Kyumin LeeLow-Rank Adaptation LLM Post-Training
Worcester Polytechnic Institute · University of Hong Kong
Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
May 12, 2026Arijit Sehanobish, Charles LoveringAdapter Tuning Low-Rank Adaptation
Kensho Technologies
LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection
May 12, 2026Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria +4Fine-Tuning Low-Rank Adaptation
The University of Sydney, Australia. · Microsoft India. · Indian Institute of Technology Bombay, India. +1
M $^4$ -SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection
May 12, 2026Jiyuan Liu, Jia Lin, Xiaofei Zhou +3Video Object Segmentation Vision Foundation Models
Hangzhou Dianzi University, China · Shandong University, China · Anqing Normal University, China +1
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
May 12, 2026Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang +3Audio-Visual Understanding Audio-Visual Representation Learning
AI2 Lab, KAIST
Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
May 12, 2026Yu-Hang Wu, Qin-Yuan Liu, Qiu-Yang Zhao +3Fine-Tuning Continued Pretraining
Nanhu Research Institute of China Electronic Science and Technology · School of Electronic and Electrical Engineering, Shanghai University of Engineering Science
AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
May 11, 2026Barbara Su, Fangshuo Liao, Anastasios KyrillidisLLM Fine-Tuning Adapter Tuning
Department of Computer Science Rice University

May 20, 2026J/K move · Enter open · S save

SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

Yongkang Liu, Xing Li, Mengjie Zhao, Shanru Zhang+6

Northeastern University, China · Shandong University, China · CIS, LMU Munich, Germany+1

LLM Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity. Theory suggests that LoRA fine-tuning with rank r converges toward the top r singular values of the pre-trained weight matrix. As the rank increases, more principal singular directions are preserved, which generally improves the model's performance. However, a larger rank also introduces more trainable parameters, leading to higher computational cost. To overcome this dilemma, we propose SMoA, a Spectrum Modulation Adapter that enlarges the accessible family of spectrum-aware updates under a smaller parameter budget. SMoA partitions the layer into multiple aligned spectral blocks and applies one in-block Hadamard-modulated low-rank branch to each diagonal block, yielding broader coverage of pretrained spectral directions. We provide theoretical analysis and empirical results on multiple tasks. In our experiments, SMoA improves average performance in the current lower-budget setting over LoRA and competitive LoRA-style baselines.