VLM Adaptation

VLM: Vision-Language Model

Overview

Vision-language model adaptation studies how to specialize pretrained models such as CLIP and instruction-tuned multimodal large language models when target data, labels, or compute are limited. Common approaches include prompt tuning, LoRA and lightweight adapters, domain-specific fine-tuning, and test-time methods that use unlabeled examples or prediction memories; newer work also adapts models for video, medical imaging, remote sensing, and robotics. Evaluation often tests few-shot transfer, base-to-novel classes, cross-dataset generalization, robustness to distribution shift, and retention of the model’s original abilities. A central tension is that stronger specialization can cause forgetting, miscalibration, or reliance on language shortcuts, while conservative updates may preserve generality but limit task gains.

Latest papers

CardsList

566 papers20 papers in the last four weeks, down 62% on the four weeks before. 0.2% of all new papers.

  1. CROMPT: Adapting CLIP via Cross-Modal Prompting for Robust Zero-Shot Generalization under Adverse Weather

    Nov 18, 2026Aniruddh Sikdar, Arya Kishor, Snehashis Dhibar +1Zero-Shot Domain AdaptationDomain Adaptation

  2. GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs

    Nov 17, 2026Maxx Richard Rahman, Wolfgang MaassVLM AdaptationTest-Time Adaptation of VLMs

  3. Practice in Text, Act in Pixels: Correcting Where Perception Fails

    Nov 17, 2026Zhongling Xu, Hairong WangGUI AgentsGUI Automation

  4. Efficient Self-Supervised Adaptation of 3D Abdominal Vision-Language Model for Institution-Specific HCC Classification via Full Fine-Tuning and PEFT

    Nov 17, 2026Febryan Putra Kartika, Cheng-Yu Ma, Ying-Jia Lin +2VLM AdaptationMedical VLMs

  5. SCAN: Site-Aware Collaborative Privacy-Preserving Adaptation of Vision-Language Models for Neurological Disease Classification

    Nov 16, 2026Nikhil J. Dhinagar, Rishab Pendam, Harmony S. Cen +7Medical Image ClassificationVLM Adaptation

  6. PRISM: Primitive-Integrated Separable Modulation with Priority Visual Memory Queue for Compositional Zero-Shot Learning

    Nov 15, 2026Jianhua Wang, Zhihang Zheng, Yang Liu +1Zero-Shot LearningTest-Time Adaptation of VLMs

  7. CARET: Clinical Assertion-Aware Residual Calibration for Medical Image-Text Alignment

    Nov 13, 2026Yifan Jin, Tao LiuRadiology VLMsMedical VLMs

  8. Context-Predictive Low-Rank Adaptation for Face Presentation Attack Detection

    Nov 12, 2026Aminu Yusuf, Shakeel Muhammad Ibrahim, Sulaimon Oyeniyi Adebayo +3Face Presentation Attack DetectionVLM Adaptation

  9. From Classification to Generation: Vision-Language Models for Roman Imperial Coin Description

    Nov 10, 2026Pedro Lourenco, Ognjen ArandjelovićVLM AdaptationImage Text Translation

  10. TMT: Adapting CLIP to Infrared Imagery via Text-Derived Modality Knowledge Transfer

    Nov 9, 2026Ruofan Wang, Puzuo Wang, Xin Chen +3VLM AdaptationCross-Modal Representation Transfer

  11. When to Trust Adaptation: Contextual Fusion for Transferable Vision-Language Prompting

    Nov 9, 2026Zhenyuan Chen, Wenshuo Tao, Zhuo Huang +3VLM EnsemblingVLM Adaptation

  12. Locating and Repairing Domain Shift in VLM Trajectory Planning

    Nov 7, 2026Zhihong Cui, Hengyu Liu, Michael Alexander Riegler +3VLM AdaptationVLMs for Autonomous Driving

  13. Permute-then-Adapt: Weak-to-Strong Contrastive Image-Text Adaptation

    Nov 7, 2026Jinhao Li, Sarah Erfani, Lei Feng +3VLM AdaptationVision-Language Alignment

  14. Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

    Nov 6, 2026Imad Eddine Marouf, Khalid Oublal, Enzo Tartaglione +1VLM AdaptationPrompt Learning for VLMs

  15. On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

    Nov 5, 2026Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1Few-Shot VLM AdaptationVLM Adaptation

  16. SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

    Nov 4, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2Few-Shot VLM AdaptationVLM Adaptation

  17. Batch-Conditioned Semantic Anchors for Robust Transductive Adaptation of Vision--Language Models

    Nov 4, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2VLM AdaptationTest-Time Adaptation of VLMs

  18. Drone-PUR: From Visual Clue Discovery to Understanding and Reasoning in UAV Scenes

    Oct 29, 2026Jiahe Wu, Bing Cao, Xingyue Guo +5Visual Spatial ReasoningVLM Evaluation

  19. LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

    Oct 26, 2026Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu +4VLM AdaptationFew-Shot Image Classification

  20. LoRA Superposition across Pretrained Models

    Oct 22, 2026Keigo Watanabe, Hiroshi Kera, Kazuhiko KawamotoLow-Rank AdaptationModel Merging

  21. FedBST: Bilateral Self-Training for Unsupervised Federated Classification

    Oct 22, 2026Aijing Yu, Kuangpu Guo, Jian Liang +1Federated LearningSelf-Training

  22. Gated Prototype Transfer for Zero-Support Medical Vision Language Model

    Oct 20, 2026Sumaiya Fazal, Rizwan Qureshi, Laleh Behjat +1VLM AdaptationMedical VLMs

  23. Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation

    Oct 20, 2026Hao Yan, Guanghui WangVLM AdaptationUnsupervised Domain Adaptation

  24. PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition

    Oct 18, 2026Mingcong Ding, Babette Bühler, Tim Fütterer +2VLM AdaptationZero-Shot Action Recognition

  25. ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation

    Oct 17, 2026Hamidreza Dastmalchi, Ali Cheraghian, Aijun An +2Video Object SegmentationReferring Video Object Segmentation

  26. LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

    Oct 15, 2026Zhipeng Ye, Feng Jiang, Qiufeng Wang +1Few-Shot LearningFew-Shot Image Classification

  27. Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

    Oct 15, 2026Madhav Iyengar, Andreas Geiger, Hilde KühneVLM CompressionVision-Language Models