VLM Adaptation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model adaptation studies how to specialize pretrained models such as CLIP and instruction-tuned multimodal large language models when target data, labels, or compute are limited. Common approaches include prompt tuning, LoRA and lightweight adapters, domain-specific fine-tuning, and test-time methods that use unlabeled examples or prediction memories; newer work also adapts models for video, medical imaging, remote sensing, and robotics. Evaluation often tests few-shot transfer, base-to-novel classes, cross-dataset generalization, robustness to distribution shift, and retention of the model’s original abilities. A central tension is that stronger specialization can cause forgetting, miscalibration, or reliance on language shortcuts, while conservative updates may preserve generality but limit task gains.

Latest papers

CardsList

546 papers41 papers in the last four weeks, down 13% on the four weeks before. 0.2% of all new papers.

  1. AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning

    Apr 16, 2026Peifeng Zhang, Zice Qiu, Donghua Yu +4Visual Question AnsweringVision-Language Models

    Sun Yat-Sen University · National Supercomputing Center in Shenzhen · Tsinghua University

  2. BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation

    Apr 15, 2026Baoyou Chen, Hanchen Xia, Peng Tu +5Vision-Language ModelsDiffusion VLMs

    Shanghai Academy of AI for Science · Fudan University · Shanghai Innovation Institute +1

  3. MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models

    Apr 14, 2026Ruoxiang Huang, Zhen YuanVision-Language ModelsVLM Adaptation

    Peking University, China

  4. Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning

    Apr 14, 2026Jungwon Choi, Eunwoo KimVLM RobustnessVLM Adaptation

    School of Computer Science and Engineering, Chung-Ang University

  5. Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localization

    Apr 14, 2026Zanyi Wang, Fan Li, Dengyang Jiang +4Temporal Video GroundingVLM Adaptation

    SGIT AI Lab, State Grid Corporation of China · University of HongKong · Zhejiang University of Technology

  6. Anthropogenic Regional Adaptation in Multimodal Vision-Language Model

    Apr 13, 2026Samuel Cahyawijaya, Peerat Limkonchotiwat, Tack Hwa Wong +45VLM AdaptationVision-Language Alignment

  7. Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models

    Apr 13, 2026Zihang Fu, Haonan Wang, Jian Kang +2Video-Language ModelsVLM Adaptation

    National University of Singapore · MBZUAI

  8. A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinations

    Apr 12, 2026Xiaoda Yang, Shuai Yang, Can Wang +9VLM AdaptationVLM Hallucination

    The Hong Kong University of Science and Technology · The Chinese University of Hong Kong

  9. Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

    Apr 10, 2026Zibin Geng, Xuefeng Jiang, Jia Li +6VLM RobustnessVisual Prompt Learning

    Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · PCALab, VCIP, College of Computer Science, Nankai University, Tianjin, China +1

  10. VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

    Apr 10, 2026Guanyu Zhou, Yida Yin, Wenhao Chai +3Visual Question AnsweringSynthetic Data Augmentation

    Princeton University · New York University

  11. Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

    Apr 10, 2026Lingfeng Huang, Huizhong Guo, Tianjun Wei +2User SimulationVLM Adaptation

    Singapore University of Technology and Design, Singapore · Zhejiang University, Hangzhou, China · Nanyang Technological University, Singapore

  12. Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

    Apr 10, 2026Harshith Kethavath, Weiming HuVLM AdaptationRemote Sensing VLMs

    University of Georgia, USA

  13. T-Gated Adapter: A Lightweight Temporal Adapter for Vision-Language Medical Segmentation

    Apr 9, 2026Pranjal KhadkaMedical Image Segmentation3D Medical Image Segmentation

    Independent Researcher

  14. Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labels

    Apr 8, 2026Yaqi Zhao, Haoliang Sun, Yating Wang +2Prompt LearningVLM Adaptation

    School of Software, Shandong University, Jinan, China

  15. Lightweight Multimodal Adaptation of Vision Language Models for Species Recognition and Habitat Context Interpretation in Drone Thermal Imagery

    Apr 7, 2026Hao Chen, Fang Qiu, Fangchao Dong +3VLM AdaptationWildlife Monitoring

    Geospatial Information Science, The University of Texas at Dallas, Richardson, TX 75080, USA · Department of Landscape Architecture, University of Florida, Gainesville, FL 32611, USA · College of Forestry, Wildlife and Environment, Auburn University, Auburn, AL 36849, USA +1

  16. WRF4CIR: Weight-Regularized Fine-Tuning Network for Composed Image Retrieval

    Apr 7, 2026Yizhuo Xu, Chaojian Yu, Yuanjie Shao +3Composed Image RetrievalVLM Adaptation

    Huazhong University of Science and Technology · University of Sydney

  17. The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Models

    Apr 6, 2026Runhao Mao, Hanshi Wang, Yixiang Yang +3VLMs for Autonomous DrivingVLM Adaptation

    AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University

  18. Scale-Aware Vision-Language Adaptation for Extreme Far-Distance Video Person Re-identification

    Apr 5, 2026Ashwat Rajbhandari, Bharatesh ChakravarthiPerson Re-IdentificationVLM Adaptation

    Arizona State University

  19. Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics

    Apr 5, 2026Minglei Chen, Weilong Wang, Jiang Duan +1Prompt Learning for VLMsPrompt Learning

    All authors are with Southwestern University of Finance and Economics, Chengdu, Sichuan, China.

  20. Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

    Apr 3, 2026Hai Nguyen-Truong, Alper Balbay, Tunga BayrakReferring Image SegmentationVisual Reasoning

    Freya

  21. Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation

    Apr 1, 2026Yunbei Zhang, Chengyi Cai, Feng Liu +1VLM AdaptationVision Foundation Model Adaptation

    Tulane University · University of Melbourne

  22. Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation

    Apr 1, 2026Reyhaneh Ahani Manghotay, Jie LiangPEFT of VLMsMonocular Depth Estimation

    Simon Fraser University, Burnaby, Canada · School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · Eastern Institute of Technology, Ningbo, China +1

  23. LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

    Apr 1, 2026Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar +2VLM AdaptationVLM Distillation

    Mohamed bin Zayed University of Artificial Intelligence

  24. Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP

    Apr 1, 2026Ruize XiaVLM AdaptationLow-Rank Adaptation

    Independent Researcher

  25. CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks

    Apr 1, 2026Wish Suharitdamrong, Tony Alex, Muhammad Awais +1Cross-Modal LearningVLM Adaptation

    Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK

  26. The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

    Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal AlignmentModality Gap in VLMs

    University of Electronic Science and Technology of China · University of Bristol

  27. GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection

    Mar 31, 2026Yaning Zhang, Linlin Shen, Zitong Yu +2Deepfake DetectionDigital Forensics

    Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250014, China · Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China · National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University +5

  28. Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

    Mar 31, 2026Jingqi XuVLM AdaptationVLM Reasoning

    University of Southern California, Los Angeles, CA, USA

  29. Domain-Invariant Prompt Learning for Vision-Language Models

    Mar 30, 2026Arsham Gholamzadeh Khoee, Yinan Yu, Robert FeldtVLM AdaptationVisual Prompt Learning

    Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden

  30. Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model

    Mar 30, 2026Athos GeorgiouVision-Language ModelsVisual Document Retrieval

    Independent Researcher