Parameter-Efficient Fine-Tuning

Also known as PEFT

Ask about this topic

Overview

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

CardsList

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

  1. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangLow-Rank AdaptationLLM Fine-Tuning

    School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

  2. When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

    Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Continual LearningLow-Rank Adaptation

    Department of Computer Science, ETH Zurich, Zurich, Switzerland.

  3. TextDS: Parameter-Efficient Representation Alignment for Scene Text Detection under Distribution Shifts

    Jun 26, 2026Boyuan Chen, Zichen Dang, Chuang Yang +2Distribution Shift RobustnessDomain Generalization

    School of Electrical Engineering, Xi’an Jiaotong University · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University

  4. Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

    Jun 25, 2026Phannet Pov, Hyun Woo Park, Voneat Pen +3Low-Resource TTS SynthesisLow-Resource Language Processing

    Department of Big Data, Chungbuk National University, Cheongju, South Korea · Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology, Phnom Penh, Cambodia · Department of Management Information Systems, Chungbuk National University, Cheongju, South Korea +1

  5. Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models

    Jun 24, 2026Xi Xiao, Xingjian Li, Yunbei Zhang +7Vision TransformerVision Foundation Model Adaptation

    UAB · CMU · Tulane +3

  6. SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

    Jun 24, 2026Omanshu ThapliyalLong-Context Language ModelingReduced-Order Modeling

    Hitachi America Ltd., Santa Clara, CA, USA.

  7. Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning

    Jun 24, 2026Samuel Valland Lyngset, Tor Viljen Raanaas, Gard Sveipe +4RL for RoboticsLow-Rank Adaptation

    Department of Informatics, University of Oslo

  8. Parallel Manifold Steering: Efficient Adaptation of Large Associative Memories via Residual Energy Shaping

    Jun 23, 2026Kanishk AwadhiyaMemory-Augmented Neural NetworksTransformer Attention

    Independent Researcher

  9. Tri-Efficient Transfer Learning for Point Cloud Videos

    Jun 23, 2026Yiding Sun, Dongxu Zhang, Jihua Zhu +6Point Cloud UnderstandingSelf-Supervised Visual Representation Learning

    School of Software Engineering, Xi’an Jiaotong University · School of Information and Communication Engineering, Xi’an Jiaotong University · SIGS, Tsinghua University

  10. When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs

    Jun 23, 2026Lucky Verma, Pratik YadavDiffusion Model Fine-TuningDiffusion Language Models

    Independent Researcher · University of Maryland, Baltimore County

  11. CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift

    Jun 22, 2026Amrita Singh, Rishabh JhaVLM AdaptationElastic Weight Consolidation

    Mindriser’s Consortium, Kathmandu, Nepal · University of Victoria, BC, Canada

  12. Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

    Jun 22, 2026Nuocheng Yang, Yechen He, Sihua Wang +3Federated UnlearningCommunication-Efficient Distributed Training

    Beijing Laboratory of Advanced Information Network, and the Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore

  13. Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation

    Jun 21, 2026Junshu Sun, Wanxing Chang, Qingming Huang +1Graph Representation LearningLow-Rank Adaptation

    State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · DAMO Academy, Alibaba Group, Hangzhou, China

  14. Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

    Jun 21, 2026Edwin Kwadwo Tenagyei, Lei Wang, Ugochukwu Ejike Akpudo +2Vision TransformerParameter-Efficient Fine-Tuning

    Griffith University, Nathan QLD 4111, Australia · Data61/CSIRO, Canberra ACT 2601, Australia

  15. MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation

    Jun 20, 2026Mohammed Rawhani, Derviş Karaboğa, Özkan Ufuk Nalbantoğlu +2Unsupervised Domain AdaptationParameter-Efficient Fine-Tuning

    Computer Engineering Department Erciyes University Kayseri, Türkiye

  16. Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

    Jun 17, 2026Lin Tang, Wei Zhang, Jing Li +3Low-Rank AdaptationLanguage Model Merging

    1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China

  17. Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

    Jun 17, 2026Hassan Dbouk, Matthias Reisser, Prathamesh Mandke +2Memory-Efficient Fine-TuningLLM Fine-Tuning

    Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. · work done while at Qualcomm

  18. A Unified Framework for Efficient Remote Sensing Visual Question Answering: Adapting Dual, Hybrid, and Encoder-Decoder Architectures

    Jun 17, 2026Timothy Agboada, Shikha Chandel, Yadav Raj Ghimire +1Remote Sensing Image UnderstandingVisual Question Answering

    Computational Data Science and Engineering North Carolina A&T State University Greensboro - NC, USA · College of Science and Technology North Carolina A&T State University Greensboro - NC, USA

  19. PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentation

    Jun 17, 2026Asad Channa, Abdullah Khan, Asghar Ali Chandio +4Medical Imaging Foundation ModelsMedical Image Segmentation

    Department of Computer Science, Quaid-e-Awam University of Engineering, Sciences & Technology · Department of Artificial Intelligence, Quaid-e-Awam University of Engineering, Sciences & Technology, Pakistan · Department of Computer Science, Sindh Madressatul Islam University, City Campus, Karachi +1

  20. Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning

    Jun 17, 2026Youngwoo Cho, Seunghoon Yi, Wooil Yang +6Machine Learning Interatomic PotentialsEquivariant Neural Networks

    KAIST · Seoul National Univ. · KIAS +2

  21. Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation

    Jun 15, 2026Hongchao Shu, Roger D. Soberanis-Mukul, Hao Ding +5EndoscopyMonocular Depth Estimation

    Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA · Semaphor Surgical · Johnson & Johnson MedTech, Santa Clara, CA 95054 USA

  22. Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration

    Jun 15, 2026Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan +1Unified Multimodal ModelsModality Imbalance

    The University of Texas at Austin · Advanced Micro Devices, Inc.

  23. Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors

    Jun 15, 2026Vanshali Sharma, Andrea M. Bejar, Halil Ertugrul Aktas +43D Medical ImagingMedical VLMs

    Department of Radiology Northwestern University Chicago, USA · Department of Computer Science Aalto University, Finland · Department of Computer Science University of South Dakota South Dakota, USA

  24. Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents

    Jun 15, 2026Tianyi Zhang, Zhonghao QiAI Coding AgentsLLM Agent Skill Learning

    1The Chinese University of Hong Kong

  25. SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

    Jun 15, 2026Junghun Oh, Sungyong Baik, Kyoung Mu LeeLow-Rank AdaptationParameter-Efficient Fine-Tuning

    Dept. of ECE, ASRI, Seoul National University · Dept. of Artificial Intelligence, Dept. of Data Science, Hanyang University · Dept. of ECE, ASRI, IPAI, Seoul National University

  26. KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

    Jun 15, 2026Mao-Lin Luo, Yi-Lin Zhang, Zi-Hao Zhou +5VLM AdaptationContinual Learning for VLMs

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education, China · Huawei Noah’s Ark Lab

  27. ttda704 at SemEval-2026 Task 6: Structured Chain-of-Thought Prompting for Political Evasion Detection

    Jun 14, 2026Tai Tran Tan, An Dinh ThienCoT ReasoningPragmatic Reasoning in Language Models

    University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

  28. MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio

    Jun 14, 2026Salman Hussain Ali, Umberto Cappellazzo, Mirco RavanelliMambaSpeech Processing

    1Universit´e de Montr´eal, Canada · 2Imperial College London, UK · 3Concordia University, Canada +1

  29. Beyond LoRA: Is Sparsity-Induced Adaptation Better?

    Jun 11, 2026Elijah Cadenhead, Cristian McGee, Xin Li +2Low-Rank AdaptationParameter-Efficient Fine-Tuning

    School of Data, Mathematical and Statistical Sciences, University of Central Florida, United States · Department of Computer Science, University of Central Florida, United States · College of Computing, Mohammed VI Polytechnic University (UM6P), Morocco

  30. Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

    Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-TuningOn-Policy Distillation

    School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group