Prompt Learning for VLMs

VLM: Vision-Language Model

Latest papers

38 papers10 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models
Nov 6, 2026Imad Eddine Marouf, Khalid Oublal, Enzo Tartaglione +1VLM Adaptation Prompt Learning for VLMs
Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CNR-ISASI, Lecce, Italy +1
FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning
Nov 4, 2026Xuanjing Zhang, Qi Guo, Yanan Qiao +2Prompt Learning Prompt Learning for VLMs
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China · Air Traffic Control and Navigation School, Air Force Engineering University, Xi’an, 710051, Shaanxi, China
Vision-language classification of Torsades de Pointes risk from pseudo-coloured ECG images: do T-wave morphology and colour context help or hinder?
Nov 1, 2026Dario Labrador Alonso, Alaa AlahmadiElectrocardiogram Classification Visual Evidence Sensitivity in VLMs
School of Computing, Newcastle University, Newcastle upon Tyne, NE4 5TG, United Kingdom
Beyond CLIP: Calibrated (Few-Shot) Defect Detection With System One Decision Models
Oct 30, 2026Fadel M. Megahed, Ying-Ju ChenPrompt Learning for VLMs Few-Shot Anomaly Detection
Farmer School of Business, Miami University, Oxford, OH 4050, USA · College of Arts and Sciences, University of Dayton, Dayton, OH 45469, USA
M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models
Oct 29, 2026Shihab Aaqil Ahamed, Mohamed Ansaff, Nang Khin Myat Lay Phyu +4VLM Calibration Prompt Learning for VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · Albukhary International University, Alor Setar, Malaysia
Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe
Oct 29, 2026Jiawei ChenVLM Evaluation Visual Evidence Sensitivity in VLMs
Netflix
Grounding on Request: Learning Modality Control in Vision-Language Models
Oct 24, 2026Maxime Dassen, Alexander Martin, Andrew Yates +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
University of Amsterdam · Johns Hopkins University · Human Language Technology Center of Excellence, Johns Hopkins University
Zero-Shot Domain Generalization via Domain Prompt Learning
Oct 22, 2026Shiming Chen, Shuhuang Chen, Salman Khan +1Domain Generalization Multimodal Domain Generalization
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia
Knowledge-Guided Distributional Prompt Learning with Visual Hierarchy for Zero-Shot Anomaly Detection
Oct 16, 2026Jiyul Ham, Seongho Yoon, Yonggon Jung +2Zero-Shot Anomaly Detection Prompt Learning
Department of Industrial and Management Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul, 02841, Republic of Korea
ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers
Oct 16, 2026Kun Zhai, Teng Li, Yunhao Feng +1Backdoor Attacks in Federated Learning Backdoor Attacks on VLMs
Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China
Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
Oct 12, 2026Haichen He, Weibin Liu, Weiwei XingPrompt Learning for VLMs Vision-Language Models
School of Computer Science and Technology, Beijing Jiaotong University · Visual Intelligence +X International Cooperation Joint Laboratory of MOE · School of Software Engineering, Beijing Jiaotong University
Anchor and Adapt: Asymmetric Prompt Adaptation for Few-Shot Industrial Anomaly Detection
Oct 4, 2026Mengyang Zhao, Teng Fu, Haiyang Yu +3Prompt Learning for VLMs Few-Shot Anomaly Detection
Shanghai Key Lab of Intelligent Information Processing and College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China
Hierarchical Prompt Learning for Hyperbolic Vision-Language Models
Sep 21, 2026Andro Erdelez, Pascal Mettes, Behzad BozorgtabarVLM Adaptation Vision-Language Models
Aarhus University · University of Amsterdam
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Sep 20, 2026Weihan Cai, Hao Tan, Xinping Gao +2Zero-Shot Learning VLM Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Purple Mountain Laboratories, Nanjing, China · Beijing University of Posts and Telecommunications, Beijing, China
Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
Sep 10, 2026Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang +5Few-Shot Object Detection Few-Shot Domain Adaptation
Carnegie Mellon University · Apple
Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases
Sep 8, 2026Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1Vision-Language Models Prompt Learning
Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara, CA 93106, USA
What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
Aug 25, 2026Ryo Kamiya, Hiroshi Kera, Kazuhiko KawamotoPrompt Learning Prompt Learning for VLMs
Chiba University, Chiba, Japan
Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
Aug 18, 2026Yang Chen, Zhan Zhuang, Yanbin Wei +3Prompt Learning for VLMs Prompt Tuning
Southern University of Science and Technology Shenzhen, China · City University of Hong Kong Hong Kong, China · Hong Kong University of Science and Technology Hong Kong, China
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
Aug 14, 2026Liang Wang, Haoyang Li, Chao Wang +3VLM Adaptation Prompt Learning for VLMs
Shanghai University, Shanghai, China · University of Technology Sydney, Sydney, Australia
ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
Aug 8, 2026Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1VLM Adaptation Zeroth-Order Optimization
Electrical and Computer Engineering Department, UC Santa Barbara · Computer Science Department, UC Santa Barbara
Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
Jul 27, 2026Haoyue Liu, Xiaoyu Ma, Ye Chen +2Visual Question Answering VLM Adaptation
The Chinese University of Hong Kong, Shenzhen, China · Xi’an Jiaotong University, China · Peking University, China +1
AdaBoosting Text Prompts for Vision-Language Models
Jul 1, 2026Seokhee Jin, Changhwan Sung, Sunung Mun +2Vision-Language Models Ensemble Learning
KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea
Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
May 25, 2026Xiao Liu, Jiaxiang Liu, Boci Peng +6VLM Robustness Prompt Learning
University of Macau · Guangdong Institute of Intelligence Science and Technology · Peking University +4
Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangVision-Language Models VLM Adaptation
Beijing Jiaotong University, Beijing, China.
When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
May 10, 2026Yunxuan Fang, Ziwei Zhang, Xinhe WangPrompt Learning VLM Adaptation
Beihang University · Beijing, China
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
May 7, 2026Junhui Yin, Nan Pu, Xinyu Zhang +4Prompt Learning VLM Adaptation
University of Science and Technology Beijing, China. · Hefei University of Technology, China. · University of Auckland, New Zealand. +3
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Apr 19, 2026Mainak Singha, Tanisha Gupta, Ankit Jha +3Cross-Modal Learning Prompt Learning
University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India +3
Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics
Apr 5, 2026Minglei Chen, Weilong Wang, Jiang Duan +1Prompt Learning for VLMs Prompt Learning
All authors are with Southwestern University of Finance and Economics, Chengdu, Sichuan, China.
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5Attention-Guided Distillation VLM Adaptation
University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China

Nov 6, 2026J/K move · Enter open · S save

Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

Imad Eddine Marouf, Khalid Oublal, Enzo Tartaglione, Stéphane Lathuilière

Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CNR-ISASI, Lecce, Italy+1

VLM Adaptation Prompt Learning for VLMs Vision-Language Models

Open paper AskPDF arXiv

Prompt learning adapts vision-language models such as CLIP by optimizing a short learnable prompt. The cross-entropy objective pulls this prompt toward base-class specialization, whereas generalization to unseen classes benefits from staying close to CLIP's zero-shot feature space. Because both objectives act on the same prompt, their gradients conflict at convergence, and in our experiments no single-prompt loss design clearly improves on the cosine baseline. We propose Decoupled Mode Connectivity (DMC), which gives each objective its own prompt. Starting from a base-trained prompt, DMC trains a specialization prompt and a generalization prompt, and merges their text features with a fixed coefficient at test time, at the inference cost of a single prompt. A linear mode connectivity (LMC) corridor in text-feature space keeps the classification loss low for every classifier interpolated between the two endpoints, and a Visual Anchor KL term preserves CLIP's zero-shot class-similarity distribution of each image during specialization. We further argue that such regularizers should satisfy class-permutation invariance (CPI), and show via Fano's inequality, in an idealized model, that violating CPI lower-bounds the drop in novel accuracy by a term proportional to the mutual information between the prompt and the base-class labels. Across 11 datasets, DMC improves the average harmonic mean of base and novel accuracy over MergeTune for both CoOp and KgCoOp, and its interpolation curves pass beyond the single-prompt operating point, shifting the base-novel frontier rather than trading along it. Code is available at https://github.com/IemProg/DMC.