Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

Sparse Fine-Tuning for Parameter-Efficient Adversarial Training
Oct 29, 2026Zhaoxin Wang, Zihang Ding, Handing WangAdversarial Training Adversarial Robustness
School of Artificial Intelligence, Xidian University, Xi’an, China
A note on data-efficient finetuning of vision foundation models
Oct 27, 2026Niclas Hergenröther, Antonio OrvietoVision Foundation Model Adaptation Vision Foundation Models
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection
Oct 27, 2026Igor Shalygin, Grigorii Davydenko, Andrey Veprikov +4Low-Rank Adaptation Fine-Tuning Language Models
Moscow Independent Research Institute of Artificial Intelligence (MIRAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · SB AI Lab +2
Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning
Oct 27, 2026Rong Huang, Yi Xiong, Yuan-Yuan Cheng +1LLM Fine-Tuning Parameter-Efficient Fine-Tuning
University of Science and Technology of China, Hefei, Anhui 230026, China
Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching
Oct 26, 2026Fei Zuo, Yizhou Huang, Kezhi WangLow-Rank Adaptation Parameter-Efficient Fine-Tuning
East China Normal University · Brunel University of London
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates
Oct 23, 2026Disen Liao, Yaoliang YuLow-Rank Adaptation Parameter-Efficient Fine-Tuning
University of Waterloo Vector Institute
Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir Language
Oct 22, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLow-Resource Language Processing Parameter-Efficient Fine-Tuning
Department of Data Analysis and Programming Technologies, Institute of Computational Mathematics and Information Technologies, Kazan (Volga Region) Federal University, Kazan, Russia
NormTune: Norm-Guided Parameter-Efficient Fine-Tuning for Reasoning-Intensive Tasks
Oct 21, 2026Zhangming Li, Qinghao Hu, Weifan Guan +3Fine-Tuning Language Models Parameter-Efficient Fine-Tuning
1The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3AiRiA +1
Parameter-Efficient Adaptation of Time Series Foundation Models for Electrical Load Forecasting
Oct 20, 2026Aliullah, Julian Oelhaf, Edouard Fouché +4Electricity Load Forecasting Time-Series Foundation Models
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg · Siemens AG, Smart Infrastructure · Department of Electrical Engineering, Media and Computer Science, Ostbayerische Technische Hochschule Amberg-Weiden
Temporal Evidence-Parameter Co-Alignment for LLM Reasoning over Temporal Knowledge Graphs
Oct 20, 2026Jiapu WangTemporal Reasoning in Language Models Temporal KGs
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, Jiangsu, 210094, China
MuPACE: Learning to Pace Muon for LoRA Fine-Tuning
Oct 20, 2026Jan Kobiolka, Christian Frey, Josif GrabockaLearning Rate Scheduling Adaptive Learning Rate
Department of Computer Science, University of Technology Nuremberg, Nuremberg, Germany
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
Oct 19, 2026Zhuofan Josh Ying, Guanglong Sun, Nolan Ho +6Neural Encoding Parameter-Efficient Fine-Tuning
Columbia University · Mind Lab · Tsinghua University +1
An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs
Oct 19, 2026Minglei Li, Peng Ye, Haonan He +3Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning
Fudan University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
Residual-Guided Adaptive Tucker Rank Allocation for Parameter-Efficient MCI Conversion Prediction
Oct 19, 2026Guanghua He, Hancan Zhu, Gaohang Yu +1Adaptive Rank Allocation Parameter-Efficient Fine-Tuning
Department of Mathematics, Hangzhou Dianzi University, Hangzhou, 310018, China · School of Mathematics, Physics and Information, Shaoxing University, Shaoxing, 312000, China · Department of Mathematics, Zhejiang University of Science and Technology, Hangzhou, 310023, China
PFedQFT: Personalized Federated Quantum-Inspired Fine-Tuning for Medical Large Language Models
Oct 19, 2026Qihan Huang, Lingxia Zhang, Guye Zhang +2Federated Fine-Tuning Federated Learning
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · Key Laboratory of Quantum Physics and Photonic Quantum Information, Ministry of Education, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · Sichuan Provincial Key Laboratory for Human Disease Gene Study and the Center for Medical Genetics, Department of Laboratory Medicine, Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital, University of Electronic Science and Technology of China, Chengdu, Sichuan, China +1
How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented Encoder
Oct 18, 2026Amr AbdelkhalikAutomatic Speech Recognition Speech Foundation Models
Independent Researcher, Egypt
PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs
Oct 18, 2026Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang +1Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning
University of Southern California
Dual Task Vector Merging: Leveraging Emotion and ASR Knowledge for Low-Resource Alzheimer's Disease Detection
Oct 17, 2026Yuan-Chao Li, Yin-Long Liu, Liu He +5Speech-Based Dementia Detection Alzheimer's Disease Classification
University of Cambridge, UK · University of Science and Technology of China, China
RiSE: Residual Subspace Expert for Generalizable Text-Centric Image Forgery Localization
Oct 17, 2026Kahim Wong, Kemou Li, Yiming Chen +2Image Forgery Localization Text Forgery Detection
State Key Laboratory of Internet of Things for Smart City, University of Macau · School of Information and Software Engineering, University of Electronic Science and Technology of China
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Oct 16, 2026Peter Cho-Ho Lam, Ziyi Wang, Zirui ZhouPrefix Tuning Parameter-Efficient Fine-Tuning
Huawei Technologies Canada Co., Ltd.
Looped GNNs Beyond Fixed Depth: Graph Size Generalization and Parameter-Efficient Adaptation
Oct 16, 2026Zhiyang Wang, Zhishang Luo, Yusu WangGraph Neural Networks Parameter-Efficient Fine-Tuning
Washington University in St. Louis, St. Louis, MO, USA · University of California, San Diego, La Jolla, CA, USA
ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
Oct 16, 2026Sicong Li, Qianqian Xu, Zhiyong Yang +4Parameter-Efficient Fine-Tuning
Institute of Information Engineering, CAS · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS +4
FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities
Oct 15, 2026Xin Zhang, Robby T. TanMultimodal Semantic Segmentation Multimodal Fusion
National University of Singapore
Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation
Oct 15, 2026Atai Waris Khan, Abdullah Mazhar, Raghu Kumar +1Multilingual Language Models Low-Resource MT
Indraprastha Institute of Information Technology Delhi
ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
Oct 14, 2026Fanyao Li, Yuxi Liu, Xiaohao Liu +3Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Communication University of China · National University of Singapore · Xidian University
DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression
Oct 13, 2026Yiheng Su, Junyi Jessy Li, Matthew LeaseParameter-Efficient Fine-Tuning
The University of Texas at Austin
Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters
Oct 13, 2026Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu +1Brain Tumor Segmentation Cross-Modality Medical Image Segmentation
School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland
How Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)
Oct 13, 2026Aditya Kulkarni, Davide Sartor, Carlos Soto +1LLM Fine-Tuning Fine-Tuning Language Models
Department of Physics, University of Massachusetts Amherst · Department of Mathematics and Statistics, University of Massachusetts Amherst
DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Oct 13, 2026Yangming Zhang, Ling Yi, Datong Li +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China
Confine to the Task, Anchor the Rest: Concept-Guided Retention Regularizer for Parameter-Efficient Fine-Tuning
Oct 13, 2026Ibna Kowsar, Ivan Medri, Fanyu Meng +1Fine-Tuning Parameter-Efficient Fine-Tuning
Department of Computer Science, University of California, Davis · University of Virginia

Oct 29, 2026J/K move · Enter open · S save

Sparse Fine-Tuning for Parameter-Efficient Adversarial Training

Zhaoxin Wang, Zihang Ding, Handing Wang

School of Artificial Intelligence, Xidian University, Xi’an, China

Adversarial Training Adversarial Robustness Sparse Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Deep neural networks achieve strong performance across many tasks but remain vulnerable to adversarial perturbations. Adversarial training (AT) is one of the most effective defenses, yet it suffers from high memory cost on large models and the persistent clean-robust accuracy trade-off. Recent work introduces parameter-efficient fine-tuning, such as LoRA, into AT, but this constraint induces a notable robustness gap relative to full-parameter AT. In this work, we revisit parameter-efficient adversarial fine-tuning from a parameter space perspective. Through gradient analysis, we find that adversarial optimization is highly concentrated on a small subset of parameters, which we call Robustness-Critical Parameters (RCPs), suggesting that robustness is encoded in a sparse subspace. Building on this observation, we propose Sparse Adversarial Fine-Tuning (SAFT), which identifies RCPs via adversarial gradient saliency and performs AT by updating only these parameters while freezing the rest. Across architectures and datasets, SAFT outperforms LoRA-based methods in both clean and robust accuracy with fewer trainable parameters, and approaches or surpasses full parameter AT in robustness while updating only about 5% of parameters.