Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation
Jul 26, 2026Changjing Liu, Yiming Huang, Beilei Cui +5Promptable Image Segmentation Vision Foundation Model Adaptation
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs
LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
Jul 25, 2026Pascal Ekin, Hyosun Choi, Wei JieLLM Alignment Low-Rank Adaptation
Royal Holloway, University of London, United Kingdom · University of West London, United Kingdom
REPREC: Representation Driven Parameter-Efficient Recommendation System
Jul 24, 2026Harshini Kavuru, Dwipam Katariya, Giri Iyengar +3Sequential Recommendation Large Language Model-Based Recommendation
The Ohio State University, USA · Capital One, AI Foundations, USA
\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
Jul 24, 2026Jianghui Wang, Silong Yong, Francesco Orabona +3Fine-Tuning Memory-Efficient Fine-Tuning
King Abdullah University of Science and Technology · Carnegie Mellon University
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation
Jul 24, 2026Yuqi Li, Xi Xiao, Yunbei Zhang +6Information Bottleneck Vision Foundation Model Adaptation
City College of New York, CUNY · New York, NY, USA · University of Alabama at Birmingham +11
Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach
Jul 23, 2026Yuhua Wang, Xiaodong Li, Yihao Guo +6VLM Adaptation Differentially Private Federated Learning
School of Artificial Intelligence, Beihang University Beijing, China · Center for the Applied Statistics, School of Statistics, Renmin University of China Beijing, China · School of Computer Science & Technology, Beijing Jiaotong University Beijing, China +2
Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning
Jul 23, 2026Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming FuFine-Tuning Diffusion Model Fine-Tuning
University of Science and Technology of China, China
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
Jul 22, 2026Abigail Woodring, Adrian Chan, Rana Muhammad Shahroz Khan +3Continual Learning for LLMs Low-Rank Adaptation
Department of Electrical and Computer Engineering NC State University, Raleigh, NC 27606, USA · Independent Researcher · Department of Computer Science University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
Point-Selection Fine-Tuning Framework for Robust Point Cloud Classification
Jul 22, 2026Da Li, Chang Ma, Dongfu YinPoint Cloud Classification Parameter-Efficient Fine-Tuning
Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen), Shenzhen, China · Shenzhen University, Shenzhen, China
Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding
Jul 21, 2026Junlin Chang, Longhao Zou, Rui Li3D Point Cloud Segmentation Point Cloud Classification
Beihang University, Beijing · Pengcheng Laboratory, Shenzhen
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Jul 20, 2026Valentijn Oldenburg, Floris de Kam, Bente Zuijdam +5Transformer Hyper-Connections
University of Amsterdam, Netherlands
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
Jul 20, 2026Nikhil Ghosh, Tetiana Parshakova, Robert M. GowerLLM Fine-Tuning Low-Rank Adaptation
Center for Computational Mathematics, Flatiron Institute, New York, NY
Federated Lightweight Fine-Tuning
Jul 20, 2026Radhakrishna Achanta, Will ReedFederated Learning Aggregation Federated Fine-Tuning
Cisco Systems Inc., USA
Opto-ViT-v2: Noise-Resilient On-Chip Fine-Tuning for Photonic Near-Sensor Vision Transformer Accelerators
Jul 20, 2026Xuming Chen, Deniz Najafi, Mehrdad Morsali +5Fine-Tuning Vision Transformer
Case Western Reserve University · USA · New Jersey Institute of Technology +1
Chebyshev Manifold Adaptation
Jul 19, 2026Jiawen LiParameter-Efficient Fine-Tuning
School of Computer Science and Engineering, University of New South Wales, Sydney, Australia
MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning
Jul 18, 2026Sana Tonekaboni, Viktoria Schuster, Caroline UhlerCross-Modal Representation Learning Multimodal Disentangled Representation Learning
Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Vector Institute, Toronto, Canada +1
TopoTuner: Topological Finetuning of Large Language Models
Jul 18, 2026Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad +4Fine-Tuning LLM Fine-Tuning
Department of Computer Science University of Central Florida Orlando, USA · Department of Computer Science University of Manitoba Canada · Department of Finance University of Central Florida Orlando, USA
A Better Start for Language Models: Domain-Conditional Position Offsets
Jul 17, 2026Ye QiaoDomain Adaptation Language Model Calibration
University of California, Irvine
Long-Context Fine-Tuning with Limited VRAM
Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-Tuning KV-Cache Offloading
BMW Group
Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
Jul 16, 2026Yuxuan Zhu, Rohan Alur, Daniel KangPAC-Bayesian Generalization Bounds Reinforcement Learning with Verifiable Rewards
UIUC · MIT · Bridgewater AIA Labs
XCT-SAM: Sequential Parameter-Efficient Domain Adaptation of SAM for Industrial XCT Defect Segmentation
Jul 15, 2026Md Mahedi Hasan, Md Mushfiqur Rahaman, Alan Pachkovskiy +3Domain Adaptation Vision Foundation Model Adaptation
Lane Department of Computer Science & Electrical Engineering West Virginia University, Morgantown, WV · Department of Industrial and Management Engineering West Virginia University, Morgantown, WV · School of Mathematical and Data Sciences West Virginia University, Morgantown, WV
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
Jul 15, 2026Tuomas Oikarinen, Zixiao Chen, Charlotte Siska +3LLM Fine-Tuning Text Classification
UC San Diego, Microsoft Research · Microsoft Security AI · UC San Diego +1
MBTI: A Multi-Branch Efficient Fine-Tuning Framework for Hyperspectral Image Classification with Foundation Models
Jul 14, 2026Mingzhen Xu, Haonan Guo, Di Wang +9Hyperspectral Image Classification Parameter-Efficient Fine-Tuning
School of Computer Science, Wuhan University, Wuhan, China. · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China. · China Petroleum Pipeline Engineering Corporation, Langfang, China, and also with Hebei Key Laboratory of Underground Energy Storage Technology, Langfang, China. +4
Accepted Prefixes Are Not All You Need: A Negative Result on PEFT-Based Block-Diffusion Drafting
Jul 14, 2026Abdurrahman Javat, Allan KazakovSpeculative Decoding LLM Inference Acceleration
Bahçeşehir University, Department of Artificial Intelligence
PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
Jul 13, 2026Jing Liu, Kun Yang, Yan Wang +5Multimodal Federated Learning Federated Learning Aggregation
College of Future Information Technology, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China · Department of Electrical and Computer Engineering, The University of British Columbia, BC V6T 1Z4, Canada +8
Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation
Jul 13, 2026Paul A. Bereuter, Mark D. Plumbley, Alois SontacchiFine-Tuning Domain Adaptation
Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria · Department of Informatics, King’s College London, London, United Kingdom
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
Jul 12, 2026Hao Zheng, Jinyi Huang, Tiantian Zheng +2VLM Adaptation Video Action Recognition
Department of Computer Engineering, New York University Abu Dhabi, UAE · Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand · Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE +1

Jul 26, 2026J/K move · Enter open · S save

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga+1

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA+1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning Adapter-Based Continual Learning

Open paper AskPDF arXiv

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.