Prompt Learning for VLMs

VLM: Vision-Language Model

Latest papers

23 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia
Knowledge-Guided Distributional Prompt Learning with Visual Hierarchy for Zero-Shot Anomaly Detection
Oct 16, 2026Jiyul Ham, Seongho Yoon, Yonggon Jung +2Zero-Shot Anomaly Detection Prompt Learning
Department of Industrial and Management Engineering, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul, 02841, Republic of Korea
Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
Oct 12, 2026Haichen He, Weibin Liu, Weiwei XingPrompt Learning for VLMs Vision-Language Models
School of Computer Science and Technology, Beijing Jiaotong University · Visual Intelligence +X International Cooperation Joint Laboratory of MOE · School of Software Engineering, Beijing Jiaotong University
Anchor and Adapt: Asymmetric Prompt Adaptation for Few-Shot Industrial Anomaly Detection
Oct 4, 2026Mengyang Zhao, Teng Fu, Haiyang Yu +3Prompt Learning for VLMs Few-Shot Anomaly Detection
Shanghai Key Lab of Intelligent Information Processing and College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China
Hierarchical Prompt Learning for Hyperbolic Vision-Language Models
Sep 21, 2026Andro Erdelez, Pascal Mettes, Behzad BozorgtabarVision-Language Models Prompt Learning
Aarhus University · University of Amsterdam
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Sep 20, 2026Weihan Cai, Hao Tan, Xinping Gao +2Zero-Shot Learning VLM Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Purple Mountain Laboratories, Nanjing, China · Beijing University of Posts and Telecommunications, Beijing, China
Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
Sep 10, 2026Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang +5Few-Shot Object Detection Few-Shot Domain Adaptation
Carnegie Mellon University · Apple
Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases
Sep 8, 2026Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1Vision-Language Models Prompt Learning
Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara, CA 93106, USA
What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
Aug 25, 2026Ryo Kamiya, Hiroshi Kera, Kazuhiko KawamotoPrompt Learning Prompt Learning for VLMs
Chiba University, Chiba, Japan
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
Aug 14, 2026Liang Wang, Haoyang Li, Chao Wang +3VLM Adaptation Prompt Learning for VLMs
Shanghai University, Shanghai, China · University of Technology Sydney, Sydney, Australia
ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
Aug 8, 2026Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1VLM Adaptation Zeroth-Order Optimization
Electrical and Computer Engineering Department, UC Santa Barbara1 · Computer Science Department, UC Santa Barbara2
Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
Jul 27, 2026Haoyue Liu, Xiaoyu Ma, Ye Chen +2Visual Question Answering VLM Adaptation
The Chinese University of Hong Kong, Shenzhen, China · Xi’an Jiaotong University, China · Peking University, China +1
AdaBoosting Text Prompts for Vision-Language Models
Jul 1, 2026Seokhee Jin, Changhwan Sung, Sunung Mun +2Vision-Language Models Ensemble Learning
KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea
Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
May 25, 2026Xiao Liu, Jiaxiang Liu, Boci Peng +6VLM Robustness Prompt Learning
University of Macau · Guangdong Institute of Intelligence Science and Technology · Peking University +4
Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangVision-Language Models VLM Adaptation
Beijing Jiaotong University, Beijing, China.
When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
May 10, 2026Yunxuan Fang, Ziwei Zhang, Xinhe WangPrompt Learning VLM Adaptation
Beihang University · Beijing, China
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
May 7, 2026Junhui Yin, Nan Pu, Xinyu Zhang +4Prompt Learning VLM Adaptation
University of Science and Technology Beijing, China. · Hefei University of Technology, China. · University of Auckland, New Zealand. +3
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Apr 19, 2026Mainak Singha, Tanisha Gupta, Ankit Jha +3Cross-Modal Learning Prompt Learning
University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India +3
Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics
Apr 5, 2026Minglei Chen, Weilong Wang, Jiang Duan +1Prompt Learning for VLMs Prompt Learning
All authors are with Southwestern University of Finance and Economics, Chengdu, Sichuan, China.
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5Attention-Guided Distillation VLM Adaptation
University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China
WMoE-CLIP: Wavelet-Enhanced Mixture-of-Experts Prompt Learning for Zero-Shot Anomaly Detection
Mar 6, 2026Peng Chen, Chao HuangPrompt Learning for VLMs Zero-Shot Anomaly Detection
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen
MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Feb 24, 2026Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1Multimodal Foundation Models Prompt Learning
University of California, Santa Barbara · University of California, Los Angeles
Towards Calibrating Prompt Tuning of Vision-Language Models
Feb 22, 2026Ashshak Sharifdeen, Fahad Shamshad, Muhammad Akhtar Munir +6VLM Calibration Prompt Learning for VLMs
Mohamed bin Zayed University of AI · University of Colombo · Michigan State University

Oct 16, 2026J/K move · Enter open · S save

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali+1

Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia

Prompt Learning for VLMs Test-Time Prompt Tuning Test-Time Adaptation of VLMs Federated Learning Aggregation Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data. However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions. At test time, source-free federated prompt tuning remains largely unexplored. Clients may receive unlabeled target samples with heterogeneous, often non-IID distributions that differ substantially across clients and from the source training distribution, causing locally adapted prompts to drift in different directions and reducing the effectiveness of federated aggregation. In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data. Our study reveals that conventional federated aggregation methods struggle under heterogeneous client distributions, as locally adapted prompts become increasingly specialized and difficult to combine effectively. To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation. FedRAPP selectively incorporates information from reliable and compatible clients while preserving local specialization, enabling more robust adaptation under distribution shift. Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.