Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
May 9, 2026Luca BalloreLanguage Model Pretraining Adapter Tuning
AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
May 9, 2026Ziyun Liu, Fengmiao Bian, Jian-Feng CaiLow-Rank Adaptation Adaptive Gradient Methods
Department of Mathematics The Hong Kong University of Science and Technology
S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain
May 9, 2026Baoquan Zhang, Zhehao Yu, Lisai Zhang +5Fine-Tuning Sparse Fine-Tuning
Harbin Institute of Technology, Shenzhen · Bilibili. Inc · Shenzhen University +1
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
May 8, 2026Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho +1Memory-Augmented Neural Networks Adapter Tuning
Department of Computer Science University of Texas at Austin · Department of Government University of Texas at Austin · University of Texas at Austin
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
May 8, 2026Ionut-Vlad Modoranu, Mher Safaryan, Dan AlistarhAdapter Tuning LLM Fine-Tuning
Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA
Bayesian Fine-tuning in Projected Subspaces
May 8, 2026Viktar Dubovik, Patryk Marszałek, Jacek Tabor +1Bayesian Neural Networks Fine-Tuning
Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland
SR $^2$ -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
May 8, 2026Fengqiang Wan, Yipeng Lin, Kan Lv +1Representation Drift Low-Rank Adaptation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
May 8, 2026Junjie Yu, Yue Wang, Zihan Deng +3Transfer Learning Parameter-Efficient Fine-Tuning
Department of Biomedical Engineering, Southern University of Science and Technology · Department of Psychology, The University of Hong Kong
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
CKT-WAM: Parameter-Efficient Context Knowledge Transfer Between World Action Models
May 7, 2026Yuhua Jiang, Yijun Guo, Hongbing Yang +7AI Control Robotic Manipulation
Tsinghua University · LivsynRobotics · Shanghai AI Laboratory
Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
May 7, 2026Suoxin Zhang, Run He, Di Fang +3LLM Fine-Tuning Adapter Tuning
South China University of Technology, China · Zhejiang University, China
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
May 7, 2026Jinqian Chen, Chang Liu, Jihua ZhuFederated Learning Aggregation Low-Rank Adaptation
School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000
VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts
May 7, 2026Yuhua Jiang, Junjie Lu, Xinyao Qin +4Fine-Tuning Vision-Language-Action Models
Microsoft Research Asia · Tsinghua University
One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning
May 7, 2026Xinrui Chen, Liu Yang, Ou WuSupervised Fine-Tuning LLM Fine-Tuning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · College of Intelligence and Computing, Tianjin University
TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models
May 7, 2026Duong Nguyen, Mohammed Jawhar, Nicolas ChesneauTabular Foundation Models Adapter Tuning
Ekimetrics
Prompt-Free and Efficient SAM2 Adaptation for Biomedical Semantic Segmentation via Dual Adapters
May 7, 2026Hinako Mitsuoka, Kazuhiro HottaPromptable Image Segmentation Vision Foundation Model Adaptation
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya, 468-8502, Japan
Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
May 7, 2026Bing Wang, Ximing Li, Changchun Li +3LLM Fine-Tuning Gradient Interference
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University · RIKEN Center for Advanced Intelligence Project +1
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
May 6, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia
Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
May 5, 2026Mullosharaf K. ArabovLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
Rethinking the Rank Threshold for LoRA Fine-Tuning
May 5, 2026Juneyoung ParkLow-Rank Adaptation Binary Classification
OptAI Inc.
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
May 5, 2026Bo Ye, Kai Gan, Tong Wei +1Vision Foundation Model Adaptation Parameter-Efficient Fine-Tuning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China.
Text-Guided Multi-Scale Frequency Representation Adaptation
May 5, 2026Weicai Yan, Xinhua Ma, Wang Lin +1Domain Adaptation VLM Adaptation
Zhejiang University · 2Nanyang Technological University
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
May 5, 2026Yihang Peng, Peng Jin, Jie Gong +4Commonsense Reasoning LLM Fine-Tuning
School of Computer Science and Software Engineering, Southwest Petroleum University · School of Electronic Information and Artificial Intelligence, Leshan Normal University
Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Models
May 5, 2026Zihao Guo, Yiding Sun, Jihua Zhu +33D Foundation Models Mamba
Xi’an Jiaotong University, Xi’an, China · Peking University, Beijing, China · School of Artificial Intelligence and Robotics, Hunan University, China +1
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
May 5, 2026Jingze Ge, Xue Geng, Yun Liu +6Fine-Tuning Memory-Efficient Fine-Tuning
National University of Singapore · Institute for Infocomm Research (I2R), A*STAR · Nankai University +1
RFPrompt: Prompt-Based Expert Adaptation of the Large Wireless Model for Modulation Classification
May 5, 2026Md Raihan Uddin, Tolunay Seyfi, Fatemeh AfghahWireless Foundation Models Automatic Modulation Classification
Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language Models LLM Fine-Tuning
University of Michigan
Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces
May 4, 2026Jingze Ge, Yun Liu, Xue Geng +4Model Compression Adapter Tuning
National University of Singapore · Nankai University · Institute for Infocomm Research (I2R), A*STAR
SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters
May 4, 2026Yagiz Nalcakan, Hyeongjin Ju, Incheol Park +3Cross-Modal Knowledge Distillation Multispectral Object Detection
Yonsei University, Seoul, Republic of Korea

May 11, 2026J/K move · Enter open · S save

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models

Jia Wei, Zhonghao Zhang, Ping Chen, Qianyang li+4

Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

Mixture-of-Experts Language Models Fine-Tuning Adapter Tuning Low-Rank Adaptation Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning of large language models, yet most variants target dense architectures. Mixture-of-Experts (MoE) models scale parameters at near-constant per-token compute, and their sparse activation patterns create untapped opportunities for more efficient adaptation. We propose Hot-Experts Layer-level Low-Rank Adaptation (HELLoRA), which attaches LoRA modules only to the most frequently activated experts at each layer. This simple mechanism reduces trainable parameters and adapter-induced FLOPs while improving downstream performance, an effect we attribute to a form of structured regularization that preserves pretrained expert specialization. To stress-test HELLoRA under extreme parameter budgets, we further compose it with LoRI to form HELLoRI, which freezes the up-projection and sparsifies the down-projection. Across three MoE backbones, namely OlMoE-1B-7B, Mixtral-8x7B, and DeepSeekMoE, and three task families covering mathematical reasoning, code generation, and safety alignment, HELLoRA consistently outperforms strong PEFT baselines. Relative to vanilla LoRA on OlMoE, HELLoRA uses 15.7% of the trainable parameters, reduces adapter FLOPs by 38.7%, achieves 1.9x the training throughput, and improves accuracy by 9.2%. On DeepSeekMoE, HELLoRA outperforms LoRA while using only 23.2% of its trainable parameters. These results demonstrate that activation-aware adapter placement is an effective and practical route to scaling PEFT for MoE language models.