Parameter-Efficient Fine-Tuning

Also known as PEFT

Parameter-efficient fine-tuning studies how to adapt pretrained models while training only a small subset of weights or adding lightweight modules, reducing resource costs and sometimes helping preserve existing capabilities. Common approaches include low-rank updates such as LoRA and quantized QLoRA, adapters and soft prompts, selective layer or rank tuning, and sparse or routed experts. Methods are evaluated on language benchmarks such as GLUE and mathematical reasoning, as well as vision, speech, multimodal, and continual-learning tasks, often with limited data or hardware. A central tension is that fewer trainable parameters do not necessarily mean lower memory use or faster training, and accuracy, robustness, and retention can vary with the model, task, and placement of updates.

Latest papers

610 papers75 papers in the last four weeks, up 12% on the four weeks before. 0.4% of all new papers.

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Aug 28, 2026Xiaoyang Guo, Guoping Luo, Jusheng Zhang +2Vision-Language Models VLM Adaptation
Sun Yat-sen University Guangzhou, Guangdong, China · The University of British Columbia Vancouver, British Columbia, Canada · Vast Intelligence Lab Sydney, New South Wales, Australia
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
Aug 28, 2026Runze Liu, Naibin Gu, Mingxu Ai +4Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Parameter-Efficient pretrained-CT-to-MRI Transfer for Rectal Cancer Segmentation: Performance-Calibration Trade-offs
Aug 27, 2026Aneesh Rangnekar, Jorge Tapias Gomez, Joseph O Deasy +1Medical Image Segmentation 3D Medical Image Segmentation
Memorial Sloan Kettering Cancer Center, New York, USA
LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning
Aug 27, 2026Muyao Yuan, Muyan Jiao, Jiangyong Ying +5Multimodal Instruction Tuning Cross-Modal Alignment
MOEKLINNS, and School of Computer Science and Technology Xi’an Jiaotong University Xi’an, China · E-surfing Vision Technology Co., Ltd China Telecom Hangzhou, China · China Telecom Xi’an, China
Parameter Efficient Continual Learning for Sparse Event-Based Transformers
Aug 27, 2026Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon BorthakurOnline Continual Learning Class-Incremental Learning
SustainAI Lab, MFSDS&AI IIT Guwahati
Fine-Tuning of Transformer models with Frames
Aug 26, 2026Harshavardhan Adepu, Li Zhang, Sanjiv Kumar +1Parameter-Efficient Fine-Tuning LLM Fine-Tuning
University of Wisconsin-Madison · Google DeepMind
CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression
Aug 26, 2026Haobo Xiong, Shaobo Liu, Kai Liu +1Parameter-Efficient Fine-Tuning Learned Image Compression
School of Computer Science and Technology, Xidian University Xi’an, China
Parameter-Efficient Self-Supervised Adaptation for EEG-FM under Fixed Computational Budgets
Aug 25, 2026Meghal Dani, Stefanie LiebeEEG Foundation Models Parameter-Efficient Fine-Tuning
University of Tübingen, Germany · Hertie Institute for AI in Brain Health (Hertie AI), Tübingen, Germany · Dept. of Neurology and Epileptology, University Clinic Tübingen, Germany +1
Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning
Aug 25, 2026Hang Chen, Jiaying Zhu, Wenya WangMechanistic Interpretability Supervised Fine-Tuning
College of Computing and Data Science Nanyang Technological University · School of Computer Science and Engineering The Chinese University of Hong Kong
Low-Rank Ternary Adaptation for Fine-Tuning Transformers
Aug 25, 2026Alexandru-Dragos Manolache, Yunqiang Li, Jan van GemertParameter-Efficient Fine-Tuning Low-Rank Adaptation
Delft University of Technology, The Netherlands · Amazon Development Center
CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning
Aug 22, 2026Qian Zha, Jinda Liu, Yuan Wu +1LLM Fine-Tuning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
Aug 21, 2026Marko Haralović, Sounic Akkaraju, Carlo Baretta +2Vision Foundation Model Adaptation Medical Imaging Foundation Models
University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands
ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation
Aug 21, 2026Can Jin, Ying Li, Jingchen Sun +5Visual Prompt Learning Vision Foundation Model Adaptation
Rutgers University · Zhejiang University · University at Buffalo, SUNY +3
Thermo-FL: Thermal-Aware Robust Federated Fine-Tuning of Large Language Models for Edge AI
Aug 21, 2026Shiva Shrestha, Kazi Shaharair Sharif, Zongxing Xie +3Federated Fine-Tuning Federated Learning
Department of Computer Science, Kennesaw State University Marietta, GA 30060, USA · School of Data Science and Analytics, Kennesaw State University Marietta, GA 30060, USA · Department of Information Technology, Kennesaw State University Marietta, GA 30060, USA
PETA:Parameter-Efficient Test-Time Adaptation for Virtual Screening
Aug 20, 2026Jia-Qi Lin, Yinghua Yao, Chang-Dong Wang +2Virtual Screening Test-Time Adaptation
Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore 138632 · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
LoRA-GA $^2$ : Low Rank Adaptation with Multi-step Gradient Adaptive Alignment
Aug 20, 2026Haonan He, Xinyue FanLLM Fine-Tuning Low-Rank Adaptation
University of Science and Technology of China · Independent Researcher
Projector Is All You Train
Aug 20, 2026Nyx Iskandar, Saathvik Selvan, Slater VictoroffMultimodal Large Language Models VLM Adaptation
Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models
Aug 20, 2026Tenghui Huang, Jiawen Kang, Dongning Liu +5Software Vulnerability Detection Continual Learning for LLMs
School of Automation, Guangdong University of Technology, Guangzhou 510006, China · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China +4
SE-MoLoRA: Shared-Expert LoRA Adapters for Domain-Specific Photographic Assessment
Aug 18, 2026Bishwash Khanal, Anlan Zhang, Sasu Tarkoma +2VLM Adaptation Mixture of Experts
University of Jyväskylä · Adobe Research · University of Helsinki
Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation
Aug 17, 2026Suraj YadavTucker Decomposition Vision Foundation Model Adaptation
Indraprastha Institute of Information Technology Delhi
SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning
Aug 15, 2026Mohammad Aref Jafari-Raddani, Morteza Mohajjel KafshdoozParameter Sharing Parameter-Efficient Fine-Tuning
Department of Electrical and Computer Engineering, Qom University of Technology, Qom, Iran.
PhaseLoRA: Control-Regime-Conditioned Low-Rank Adaptation for Continuous-Action Vision-Language-Action Policies
Aug 15, 2026Yufei Guo, Yinan Wu, Haoran Duan +2Low-Rank Adaptation Vision-Language-Action Models
Tsinghua University
Approximate Muon with low-rank adapters
Aug 14, 2026Ben Anson, Conor Houghton, Edward MilsomMuon Optimizer Parameter-Efficient Fine-Tuning
University of Bristol School of Mathematics · University of Bristol School of Engineering Mathematics and Technology · University of Bath Department of Computer Science
Behavioral Reprogramming of Open-Weights Models: Cognitive Plasticity and Alignment Bounds
Aug 13, 2026Lucia MalíčkováLLM Alignment Language Model Steering
National Supercomputing Centre, Slovakia.
Hate Speech Classification In Roman Urdu: A Comparative Study On Parameter Efficient Fine-Tuning And Prompt Engineering
Aug 10, 2026Toneema ZubairHate Speech Detection Prompt Engineering
Department of Computer Science Information Technology University, Lahore Pakistan
Topology-Aware Parameter-Efficient Adaptation for Cross-Dataset Retinal Vessel Segmentation
Aug 10, 2026Yongsong Huang, Tomo Miyazaki, Kai Xu +3Retinal Vessel Segmentation Domain Adaptation
Tohoku University · Yale University
Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation
Aug 10, 2026Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham +1Multimodal Embedding Adapter Tuning
Eximius Labs · Wabash College · Skop Intelligence Co.
Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation
Aug 8, 2026Uri Z. Kialy, Gil Ben-ArtziFine-Tuning Vision Foundation Model Adaptation
School of Computer Science, Ariel University, Israel
AgriField-40K: Adapting Vision Models to Agriculture With Efficient Continual Pretraining
Aug 8, 2026Vasileios Tzouras, Paraskevas Pegios, Lazaros NalpantidisAgricultural Image Analysis Masked Autoencoders
Technical University of Denmark (DTU), Kongens Lyngby, Denmark · Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark
AdaDINO: Pair-Aware In-Backbone Adaptation of Frozen DINO for Efficient Remote Sensing Change Detection
Aug 8, 2026Xu Zhang, Xinqing Li, Jianpeng Xie +3Vision Foundation Model Adaptation Remote Sensing Change Detection
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · Academy for Advanced Interdisciplinary Studies, Nankai University +1

Aug 28, 2026J/K move · Enter open · S save

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang+1

Sun Yat-sen University Guangzhou, Guangdong, China · The University of British Columbia Vancouver, British Columbia, Canada · Vast Intelligence Lab Sydney, New South Wales, Australia

Vision-Language Models VLM Adaptation VLM Distillation Video Understanding Parameter-Efficient Fine-Tuning Video Token Compression

Open paper AskPDF arXiv

Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.