PEFT of VLMs

PEFT: Parameter-Efficient Fine-Tuning · VLM: Vision-Language Model

Follow this topic Ask about this topic

Latest papers

21 papers7 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation
Nov 4, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2Few-Shot VLM Adaptation VLM Adaptation
Edge Hill University, UK
From Visual Matching to Textual Conditioning: Cross-Domain Few-Shot Segmentation via Reference Inversion
Nov 2, 2026Jiahao Nie, Yun Xing, Wenbin An +4Few-Shot Semantic Segmentation Semantic Segmentation
Nanyang Technological University · Xi’an Jiaotong University · VinUniversity +1
M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models
Oct 29, 2026Shihab Aaqil Ahamed, Mohamed Ansaff, Nang Khin Myat Lay Phyu +4VLM Calibration Prompt Learning for VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · Albukhary International University, Alor Setar, Malaysia
Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks
Oct 25, 2026Szymon Mazurek, Alberto Presta, Michał ByraVision-Language Models Recurrent Transformers
Samsung AI Center Warsaw, Poland · AGH University of Krakow, Poland · ACC Cyfronet AGH, Poland +1
DROA-CLIPSeg: Prompt-Guided Thin Crack Segmentation in Low-Light Conditions
Oct 24, 2026Vinita Shukla, Partibha Jangra, Vinayak Pathak +2Crack Segmentation Image Segmentation
Indian Institute of Technology Mandi · Indian Institute of Technology Kanpur
Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation
Oct 20, 2026Hao Yan, Guanghui WangVLM Adaptation Unsupervised Domain Adaptation
Department of Computer Science, Toronto Metropolitan University, Toronto, ON, Canada
PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition
Oct 18, 2026Mingcong Ding, Babette Bühler, Tim Fütterer +2VLM Adaptation Zero-Shot Action Recognition
Technical University of Munich Munich, Germany · Munich Center for Machine Learning (MCML) Munich, Germany · University of Tübingen Tübingen, Germany
PLANE: Plug-and-Play Fine-Grained Visual Experts for Hallucination Mitigation in Large Vision-Language Models
Oct 18, 2026Jianghao Yin, Lang Yu, Qianxi Yu +4VLM Hallucination Mitigation Object Hallucination in VLMs
School of Computer Science and Technology, East China Normal University · Shanghai Qiji Zhifeng Co., Ltd.
Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning
Oct 5, 2026Suguru Onda, Matthew Bailey, Ryan FarrellVision-Language Models VLM Adaptation
Brigham Young University · Carnegie Mellon University
Fine-Tuning VLM for Enhancing AI's Spatial Intelligence: Understanding 3D and 2D Rotations
Oct 3, 2026Uttamasha Monjoree, Wei YanVisual Spatial Reasoning 3D Spatial Reasoning
Department of Architecture, Texas A&M University, College Station, USA
Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
Sep 10, 2026Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang +5Few-Shot Object Detection Few-Shot Domain Adaptation
Carnegie Mellon University · Apple
Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution
Jul 29, 2026Mingkuan Feng, Zhengqi Wen, Jianhua TaoVLM Adaptation Large Vision-Language Models
Tsinghua University · Beijing National Research Center for Information Science and Technology
Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
Apr 1, 2026Reyhaneh Ahani Manghotay, Jie LiangPEFT of VLMs Monocular Depth Estimation
Simon Fraser University, Burnaby, Canada · School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · Eastern Institute of Technology, Ningbo, China +1
The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
Mar 28, 2026Shivang Chopra, Shaunak Halbe, Chengyue Huang +2PEFT of VLMs VLM Robustness
Georgia Institute of Technology
Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation
Mar 27, 2026Yiming Ren, Yujiu Yang, Junjie WangVLM Adaptation VLM Reasoning
Tsinghua University
Balanced Thinking: Improving Chain of Thought Training in Vision Language Models
Mar 19, 2026Shaked Perek, Ben Wiesel, Avihu Dekel +2PEFT of VLMs VLM Reasoning
IBM Research
Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning
Mar 10, 2026Chun-Peng Chang, Chen-Yu Wang, Holger Caesar +1PEFT of VLMs Temporal Reasoning in VLMs
DFKI Augmented Vision · TU Delft
Tuning Qwen2.5-VL to Improve Its Web Interaction Skills
Feb 20, 2026Alexandra Yakovleva, Henrik Pärssinen, Harri Valpola +2PEFT of VLMs Web Agents
Aalto University Espoo, Finland · System 2 AI Helsinki, Finland · University of Oulu Oulu, Finland
Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
Dec 16, 2025George-Andrei Dima, Răzvan-Alexandru Smădu, Dumitru-Clementin CercelVLM Adaptation Image Captioning
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania
How to Teach Large Multimodal Models New Skills
Oct 9, 2025Zhen Zhu, Yiming Gong, Yao Xiao +2Multimodal Large Language Models Continual Learning for VLMs
University of Illinois Urbana-Champaign · Google DeepMind · Carnegie Mellon University
FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints
Sep 1, 2025Lishan Yang, Wei Emma Zhang, Nam Kha Nguygen +4Federated Fine-Tuning Federated Learning Aggregation
Adelaide University · Central China Normal University · Harbin Institute of Technology

Nov 4, 2026J/K move · Enter open · S save

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar+1

Edge Hill University, UK

Few-Shot VLM Adaptation VLM Adaptation PEFT of VLMs VLM Distillation

Open paper AskPDF arXiv

Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and cache-based adapters, typically supervise global image embeddings and therefore underuse token-level structure. Conversely, structure-aware methods exploit local evidence but often introduce additional inference-time computation. We propose SD-LoRA, a structural distillation framework that uses token-level graph reasoning only during training and transfers its effects into a lightweight LoRA-adapted CLIP model. During training, a heterogeneous graph teacher models patch-patch, global-local, and visual-text relations over high-resolution ViT tokens. Rather than retaining this graph at test time, SD-LoRA transfers its structural bias into the shared CLIP-LoRA backbone through implicit distillation and Prototype Predictive Alignment, which aligns graph-conditioned representations with support-derived class prototypes. At inference, the graph teacher is discarded; prediction requires only a single CLIP-LoRA forward pass and a cache-based classifier. Across 11 few-shot benchmarks, SD-LoRA improves over a strong CLIP-LoRA baseline by 2.4/1.8/3.0/2.4/2.0 percentage points in the 1/2/4/8/16-shot settings, respectively, without additional graph computation at test time. Results on base-to-novel generalization, cross-dataset transfer, out-of-distribution robustness, and calibration further show that training-time structural supervision improves both accuracy and transfer reliability. Code is available at https://github.com/MR-Sherif/SD-LoRA.git.