Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge
Mar 10, 2026Run Wang, Victor J. B. Jung, Philip Wiese +3On-Device Training Edge ML
Integrated Systems Laboratory (IIS), ETH Zurich, Switzerland · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy · Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, Italy
Grow, Don't Overwrite: Fine-tuning Without Forgetting
Mar 9, 2026Dyah Adila, Hanna Mazzawi, Benoit Dherin +1Catastrophic Forgetting Parameter-Efficient Fine-Tuning
University of Wisconsin-Madison · Google Research
IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation
Mar 9, 2026Sunghyun Baek, Jaemyung Yu, Seunghee Koh +3Test-Time Adaptation Domain Adaptation
Korea Advanced Institute of Science and Technology (KAIST) · LG Energy Solution
Gated Adaptation for Continual Learning in Human Activity Recognition
Mar 8, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Mohit Malu +4Domain-Incremental Learning Human Activity Recognition
Arizona State University, Phoenix, AZ, USA · The University of Texas at Austin, Austin, TX, USA
Adaptive Capacity Allocation for Vision Language Action Fine-tuning
Mar 8, 2026Donghoon Kim, Minji Bae, Unghui Nam +4VLA Model Training Nonlinear LoRA
Department of Electrical and Computer Engineering, Seoul National University, Seoul 08826, Republic of Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
Faster-HEAL: An Efficient and Privacy-Preserving Collaborative Perception Framework for Heterogeneous Autonomous Vehicles
Mar 7, 2026Armin Maleki, Hayder RadhaAutonomous Driving Perception Heterogeneous Collaborative Perception
Electrical and Computer Engineering, Michigan State University, East Lansing, Michigan
CLoPA: Continual Low Parameter Adaptation of Interactive Segmentation for Medical Image Annotation
Mar 6, 2026Parhom Esmaeili, Chayanin Tangwiriyasakul, Eli Gibson +2Medical Image Segmentation Interactive Medical Image Segmentation
School of Biomedical Engineering and Imaging Sciences, KCL, UK · Siemens Healthineers, Princeton NJ, USA
Self-Auditing Parameter-Efficient Fine-Tuning for Few-Shot 3D Medical Image Segmentation
Mar 6, 2026Son Thai Ly, Hien V. NguyenParameter-Efficient Fine-Tuning Few-Shot Medical Image Segmentation
University of Houston
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Mar 5, 2026Yize Wu, Ke Gao, Ling Li +1Low-Rank Adaptation LLM Fine-Tuning
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
Mar 4, 2026Zailong Tian, Yanzhe Chen, Zhuoheng Han +1Low-Rank Adaptation Adapter Tuning
School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University
Suppressing Domain-Specific Hallucination in Construction LLMs: A Knowledge Graph Foundation for GraphRAG and QLoRA on River and Sediment Control Technical Standards
Mar 3, 2026Takato YasunoQuestion Answering Graph Retrieval-Augmented Generation
Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervision
Mar 3, 2026Kirill Borodin, Kirill Kondrashov, Nikita Vasiliev +4Vision-Language Models Weakly Supervised Video Anomaly Detection
Faculty of Information Technology, Moscow Technical University of Communication and Informatics, Moscow 111024, Russia
GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASR
Mar 2, 2026Pouya Mehralian, Melissa Farasyn, Anne Breitbarth +2Automatic Speech Recognition Domain Adaptation for ASR
ESAT/PSI, KU Leuven, Belgium · ∆iaLing, Ghent University, Belgium · GLiMS & MULTPIPLES, Ghent University, Belgium
Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
Mar 2, 2026Zichen Tian, Yaoyao Liu, Qianru SunParameter-Efficient Fine-Tuning Vision Foundation Model Adaptation
Singapore Management University · University of Illinois Urbana-Champaign
Scalable Multi-Task Low-Rank Model Adaptation
Mar 2, 2026Zichen Tian, Antoine Ledent, Qianru SunLow-Rank Adaptation Multi-Task Learning
Singapore Management University
Quantum-Inspired Fine-Tuning for Few-Shot AIGC Detection via Phase-Structured Reparameterization
Mar 2, 2026Kaiyang Xing, Han Fang, Zhaoyun Chen +4Parameter-Efficient Fine-Tuning Quantum-Inspired ML
Anhui University, Hefei, China · National University of Singapore, Singapore, Singapore · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China +1
HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning
Mar 1, 2026Bartosz Trojan, Filip GębalaLow-Rank Adaptation Language Model Calibration
Upper-Secondary Schools of Communications in Cracow
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
Feb 28, 2026Jie Cao, Zhenxuan Fan, Zhuonan Wang +8Mixture-of-Experts Models Parameter-Efficient Fine-Tuning
Zhejiang University · Wechat, Tencent
TAP-SLF: Parameter-Efficient Adaptation of Vision Foundation Models for Multi-Task Ultrasound Image Analysis
Feb 28, 2026Hui Wan, Libin LanVision Foundation Model Adaptation Medical Image Analysis
College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, China
Weight Updates as Activation Shifts: A Principled Framework for Steering
Feb 28, 2026Dyah Adila, John Cooper, Alexander Yun +2Activation Steering Parameter-Efficient Fine-Tuning
Department of Computer Science, University of Wisconsin-Madison
FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning
Feb 27, 2026Jieming Bian, Lei Wang, Letian Zhang +1Federated Fine-Tuning Federated Learning
University of Florida, Gainesville, FL 32611 · Middle Tennessee State University Murfreesboro, TN 37132
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Feb 27, 2026Lingfeng He, De Cheng, Huaijie Wang +3Continual Learning Low-Rank Adaptation
State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.
Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception
Feb 26, 2026Yiding Sun, Jihua Zhu, Haozhe Cheng +4Transfer Learning Point Cloud Learning
the School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710048, China · the School of Electrical and Information Engineering, Hunan University, Changsha, 410082, China +1
pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation
Feb 26, 2026Shentong Mo, Xufang Luo, Dongsheng LiVisual Prompt Learning Vision Foundation Model Adaptation
Carnegie Mellon University · Microsoft Research
CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference
Feb 26, 2026Hung-Hsuan ChenNonlinear LoRA Adapter Tuning
Computer Science and Information Engineering National Central University Taoyuan, Taiwan
OmniOCR: Generalist OCR for Ethnic Minority Languages
Feb 24, 2026Bonan Liu, Zeyu Zhang, Bingbing Meng +5Optical Character Recognition Low-Resource Language Processing
Southwest Minzu University · AI Geeks
ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
Feb 24, 2026Xindian Ma, Rundong Kong, Peng Zhang +2LLM Fine-Tuning Low-Rank Adaptation
Tianjin University, Tianjin, China
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Feb 24, 2026Nuocheng Yang, Sihua Wang, Ouwen Huan +3Federated Fine-Tuning Federated Learning
Beijing Laboratory of Advanced Information Network, and the Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Electrical and Computer Engineering and Institute for Data Science and Computing, University of Miami, Coral Gables, FL, 33146 USA · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore
Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
Feb 23, 2026Jin Liu, Yinbin Miao, Ning Xi +1Federated Fine-Tuning Low-Rank Adaptation
School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University
Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models
Feb 22, 2026Kainan Liu, Yong Zhang, Ning Cheng +4LLM Fine-Tuning Low-Rank Adaptation
Ping An Technology (Shenzhen) Co., Ltd., China

Mar 10, 2026J/K move · Enter open · S save

TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge

Run Wang, Victor J. B. Jung, Philip Wiese, Francesco Conti+2

Integrated Systems Laboratory (IIS), ETH Zurich, Switzerland · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy · Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, Italy

On-Device Training Edge ML Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

On-device tuning of deep neural networks enables long-term adaptation at the edge while preserving data privacy. However, the high computational and memory demands of backpropagation pose significant challenges for ultra-low-power, memory-constrained extreme-edge devices. These challenges are further amplified for attention-based models due to their architectural complexity and computational scale. We present TrainDeeploy, a framework that unifies efficient inference and on-device training on heterogeneous ultra-low-power System-on-Chips (SoCs). TrainDeeploy provides the first complete on-device training pipeline for extreme-edge SoCs supporting both Convolutional Neural Networks (CNNs) and Transformer models, together with multiple training strategies such as selective layer-wise fine-tuning and Low-Rank Adaptation (LoRA). On a RISC-V-based heterogeneous SoC, we demonstrate the first end-to-end on-device fine-tuning of a Compact Convolutional Transformer (CCT), achieving up to 11 trained images per second. We show that LoRA reduces dynamic memory usage by 23%, decreases the number of trainable parameters and gradients by 15x, and reduces memory transfer volume by 1.6x compared to full backpropagation. TrainDeeploy achieves up to 4.6 FLOP/cycle on CCT (0.28M parameters, 71-126M FLOPs) and up to 13.4 FLOP/cycle on Deep-AE (0.27M parameters, 0.8M FLOPs), while expanding the scope of prior frameworks to support both CNN and Transformer models with parameter-efficient tuning on extreme-edge platforms.