VLM Adaptation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model adaptation studies how to specialize pretrained models such as CLIP and instruction-tuned multimodal large language models when target data, labels, or compute are limited. Common approaches include prompt tuning, LoRA and lightweight adapters, domain-specific fine-tuning, and test-time methods that use unlabeled examples or prediction memories; newer work also adapts models for video, medical imaging, remote sensing, and robotics. Evaluation often tests few-shot transfer, base-to-novel classes, cross-dataset generalization, robustness to distribution shift, and retention of the model’s original abilities. A central tension is that stronger specialization can cause forgetting, miscalibration, or reliance on language shortcuts, while conservative updates may preserve generality but limit task gains.

Latest papers

CardsList

546 papers41 papers in the last four weeks, down 13% on the four weeks before. 0.2% of all new papers.

  1. CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition

    Mar 30, 2026Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoit Savary +3VLM AdaptationFacial Expression Recognition

    LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal, Canada · Dept. of Computer Science, École Polytechnique, Paris, France · LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada

  2. Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation

    Mar 27, 2026Yiming Ren, Yujiu Yang, Junjie WangVLM AdaptationVLM Reasoning

    Tsinghua University

  3. SlotVTG: Object-Centric Adapter for Generalizable Video Temporal Grounding

    Mar 26, 2026Jiwook Han, Geo Ahn, Youngrae Kim +1Temporal Video GroundingVLM Adaptation

    Kyung Hee University · University of Southern California

  4. From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

    Mar 25, 2026Francesco Gentile, Nicola Dall'Asen, Francesco Tonini +3Transformer InterpretabilityVLM Interpretability

    University of Trento · University of Pisa · Fondazione Bruno Kessler

  5. Project and Mix: Task-Semantic Prototypes for Few-Shot Image Classification

    Mar 25, 2026Dipam Goswami, Simone Magistri, Gido M. van de Ven +4Prototype-Based LearningVLM Adaptation

    Department of Computer Science, Universitat Autònoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain · Media Integration and Communication Center, University of Florence, Italy +3

  6. Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

    Mar 25, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Large Vision-Language ModelsZero-Shot Image Classification

    Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France

  7. Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation

    Mar 24, 2026ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun +3Open-Vocabulary Semantic SegmentationSliding-Window Attention

    Sungkyunkwan University

  8. Adaptive Dual-Teacher Distillation with Subnetwork Rectification for Bridging Semantic Gaps in Black-Box Domain Adaptation

    Mar 24, 2026Zhe Zhang, Jing Li, Wanli Xue +4Source-Free Domain AdaptationKnowledge Distillation

    School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China. · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China

  9. CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

    Mar 22, 2026Nan Zhou, Huiqun Wang, Yaoyan Zheng +1VLM AdaptationMultimodal Large Language Models

    State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University

  10. SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

    Mar 22, 2026Zhixiang Lu, Shijie Xu, Kaicheng Yan +6Medical VLMsVLM Adaptation

    Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom

  11. Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

    Mar 20, 2026Jiajie Li, Chenhui Xu, Meihuan Liu +1VLM AdaptationMedical VLMs

    University at Buffalo Buffalo, NY, USA

  12. CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models

    Mar 19, 2026Xiang Chen, Fangfang Yang, Chunlei Meng +6Medical Image Quality AssessmentVLM Robustness

  13. Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment

    Mar 18, 2026Yaze Zhao, Yixiong Zou, Yuhua Li +1Cycle ConsistencySource-Free Cross-Domain Few-Shot Learning

    School of Computer Science and Technology, Huazhong University of Science and Technology

  14. Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

    Mar 18, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2Few-Shot Image ClassificationVLM Adaptation

    Edge Hill University, Ormskirk, Lancashire, L39 4QP

  15. MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

    Mar 18, 2026Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy +1Medical Image Anomaly DetectionMedical VLMs

    Singapore Management University · La Trobe University · Vietnam National University, Hanoi

  16. ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models

    Mar 17, 2026M. Arda Aydın, Melih B. Yilmaz, Aykut Koç +1VLM AdaptationMedical VLMs

    Bilkent University · National Magnetic Resonance Research Center (UMRAM)

  17. HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

    Mar 17, 2026Md Jahidul IslamVLM AdaptationAdapter Tuning

    Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

  18. FlowComposer: Composable Flows for Compositional Zero-Shot Learning

    Mar 17, 2026Zhenqi He, Lin Li, Long ChenZero-Shot LearningCompositional Representation Learning

    The Hong Kong University of Science and Technology

  19. Rethinking VLMs for Image Forgery Detection and Localization

    Mar 13, 2026Shaofeng Guo, Jiequan Cui, Richang HongImage Forgery DetectionImage Forgery Localization

    Hefei University of Technology Key Laboratory of Knowledge Engineering with Big Data, Ministry of Education

  20. What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

    Mar 13, 2026Sen Nie, Jie Zhang, Zhongqi Wang +3VLM RobustnessAdversarial Robustness

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China

  21. AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network

    Mar 13, 2026Yu Hu, Jianyang Gu, Hao Liu +4VLM AdaptationRemote Sensing VLMs

    The University of British Columbia, Okanagan, Kelowna, BC, Canada · The Ohio State University, Columbus, OH, USA · TerraSense Analytics, Kelowna, BC, Canada

  22. Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization

    Mar 12, 2026Ayan Banerjee, Kuntal Thakur, Sandeep GuptaDomain GeneralizationCross-Domain Generalization

    Impact Lab, Arizona State University

  23. OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

    Mar 12, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +3Remote Sensing VLMsVLM Adaptation

    INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”

  24. ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

    Mar 12, 2026Md Jahidul IslamVLM AdaptationTest-Time Adaptation of VLMs

    Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

  25. A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

    Mar 11, 2026Haihua Luo, Xuming Ran, Jiangrong Shen +4Incremental LearningContinual Learning for VLMs

    Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland · Equal contribution · National University of Singapore, Singapore, Singapore +2

  26. Ego: Embedding-Guided Personalization of Vision-Language Models

    Mar 10, 2026Soroush Seifi, Simon Gardier, Vaggelis Dorovatas +2VLM AdaptationMemory-Augmented VLMs

    Toyota Motor Europe

  27. Guided Prompt Evolution for Vision-Language Models Adaptation

    Mar 10, 2026Enming Zhang, Jiayang Li, Yanlong Wang +3VLM AdaptationFew-Shot Learning

    Tsinghua Shenzhen International Graduate School, Tsinghua University · Sun Yat-sen University · Chinese University of Hong Kong, Shenzhen

  28. BiCLIP: Domain Canonicalization via Structured Geometric Transformation

    Mar 9, 2026Pranav Mantini, Shishir K. ShahVLM AdaptationDomain Adaptation

    University of Houston, Department of Computer Science · The University of Oklahoma, School of Computer Science

  29. Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

    Mar 9, 2026Junyuan Mao, Qiankun Li, Linghao Meng +5Large Vision-Language ModelsFine-Grained Visual Perception

    National University of Singapore · Nanyang Technological University

  30. FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

    Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5Attention-Guided DistillationVLM Adaptation

    University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China