Prompt Learning

Follow this topic Ask about this topic

Prompt learning studies how to steer pretrained models by changing the instructions or prompt representations they receive, rather than retraining all their weights. Approaches range from hand-written hard prompts to trainable soft prompts, including methods such as CoOp and CoCoOp, as well as prompt pools, routing, and low-rank prompt tuning. These methods support efficient adaptation in settings such as few-shot CLIP classification, open-vocabulary recognition, out-of-distribution detection, and continual learning, where data or compute may be limited. A central challenge is balancing specialization to a task or domain against generalization to new classes and changing inputs, while keeping prompt selection reliable and adaptations interpretable.

Latest papers

78 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing
Aug 5, 2026Simon Lösche, Barış Büyüktaş, Mathis Adler +3Remote Sensing Image Understanding Prompt Learning
BIFOLD - Berlin Institute for the Foundations of Learning and Data, 10587 Berlin, Germany · Technische Universität Berlin, 10623 Berlin, Germany · Orion Lab, School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens, 15772 Athens, Greece +2
Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior
Jul 21, 2026Garrett Baker, Vinayak Pathak, Daniel Murfet +1Prompt Learning Transformer
Timaeus · CORAL · Monash University
Model Agnostic Graph Prompt Learning for Crystal Property Prediction
Jul 9, 2026Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri +2Graph Neural Networks Prompt Learning
School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science, Kolkata, India · Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India
Learning Probabilistic Prompt for Continual Learning
Jul 6, 2026Hyekang Park, Sanghoon Lee, Geon Lee +2Class-Incremental Learning Prompt Learning
1Yonsei University, Seoul, South Korea · 2Samsung Electronics, Suwon, South Korea
ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models
Jun 30, 2026Asif Hanif, Mohammad YaqubZero-Shot Learning Prompt Learning
Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care
Jun 30, 2026Shreyas Rajesh, Kartik Sharma, Tonmoy Monsoor +8Prompt Learning Selective Prediction
Electrical and Computer Engineering, University of California, Los Angeles · Makerere University · St. Francis Hospital Nsambya +1
Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities
Jun 29, 2026Taixi Chen, Nancy GuoMissing-Modality Learning Prompt Learning
School of Computing State University of New York at Binghamton
Robust Zero-shot Anomaly Detection under Limited Auxiliary Anomaly Priors
Jun 28, 2026Guanyu Lu, Fang Zhou, Cheqing JinPrompt Learning Domain Generalization
East China Normal University, Shanghai, China
Automatic Generation of Highlights for Academic Paper Via Prompt-based Learning
Jun 24, 2026Yi Xiang, Chengzhi Zhang, Heng ZhangPrompt Learning Text Summarization
Department of Information Management, Nanjing University of Science and Technology, Nanjing, China
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
Jun 23, 2026Ahmad Pouramini, Hesham FailiLanguage Model Pretraining Fine-Tuning
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, North Kargar Street, Tehran, 515-14395, Tehran, Iran.
Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation
Jun 21, 2026Na Sang, Ding Ma, Rui Sang +1Prompt Learning VLM Adaptation
University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher
Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring
Jun 18, 2026Po-Chin Chang, Nicholas Hogan, Aske Plaat +1Prompt Learning Intelligent Tutoring Systems
Leiden University · FutureWhiz
Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning
Jun 1, 2026Wenhang Shi, Yiren Chen, Shuqing Bian +5Continual Learning for LLMs Prompt Learning
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China · Tencent, Beijing, China
Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
May 25, 2026Xiao Liu, Jiaxiang Liu, Boci Peng +6VLM Robustness Prompt Learning
University of Macau · Guangdong Institute of Intelligence Science and Technology · Peking University +4
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1VLM Robustness Prompt Learning
Wenzhou University · Wenzhou Business College
Learning the Interaction Prior for Protein-Protein Interaction Prediction: A Model-Agnostic Approach
May 11, 2026Ziqi Gao, Chenyi Zi, Zijing Liu +3Prompt Learning Graph Representation Learning
Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) · IDEA Research +1
When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
May 10, 2026Yunxuan Fang, Ziwei Zhang, Xinhe WangPrompt Learning VLM Adaptation
Beihang University · Beijing, China
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
May 7, 2026Junhui Yin, Nan Pu, Xinyu Zhang +4Prompt Learning VLM Adaptation
University of Science and Technology Beijing, China. · Hefei University of Technology, China. · University of Auckland, New Zealand. +3
Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection
May 7, 2026Letian Bai, Xuanming Cao, Juan Du +1Prompt Learning Cross-Modal Alignment
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · College of Mechanical and Vehicle Engineering, Hunan University
HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning
May 7, 2026Yu Feng, Zhen Tian, Haoran Luo +8Prompt Learning Domain-Incremental Learning
China Mobile Research Institute, China · Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore +1
Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning
May 6, 2026Yating Wang, Yaqi Zhao, Yongshun Gong +2Prompt Learning VLM Adaptation
School of Software, Shandong University, Jinan, 250101, China
Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation
Apr 28, 2026Tianshui Chen, Yujie Zhu, Jianman Lin +4Prompt Learning VLM Adaptation
Guangdong University of Technology · South China University of Technology · Peking University +1
State Transfer Reveals Reuse in Controlled Routing
Apr 20, 2026Yanzhen Lu, Zhicheng Qian, Muchen Jiang +1Prompt Learning Transfer Learning
Shanghai Qizhi Institute · Southeast University · Nanjing Whale Cloud +1
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Apr 19, 2026Mainak Singha, Tanisha Gupta, Ankit Jha +3Cross-Modal Learning Prompt Learning
University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India +3
Generalizable Face Forgery Detection via Separable Prompt Learning
Apr 19, 2026Enrui Yang, Yuezun LiPrompt Learning Image Forgery Detection
School of Computer Science and Technology, Ocean University of China, Qingdao, China
TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models
Apr 17, 2026Jinlun Ye, Jiang Liao, Runhe Lai +4Prompt Learning Test-Time Adaptation
Sun Yat-sen University · Peng Cheng Laboratory · Key Laboratory of Machine Intelligence and Advanced Computing, MOE +2
Unified Graph Prompt Learning via Low-Rank Graph Message Prompting
Apr 13, 2026Beibei Wang, Bo Jiang, Ziyan Zhang +1Graph Prompt Learning Prompt Learning
School of Computer Science and Technology, Anhui University, Hefei 230601, China. · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, School of Computer Science and Technology, Anhui University, Hefei 230601, China.
LDEPrompt: Layer-importance guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning
Apr 13, 2026Linjie Li, Zhenyu Wu, Huiyu Xiao +1Class-Incremental Learning Prompt Learning
School of Information and Communication Engineering, BUPT. · Engineering Research Center for Information Network, Ministry of Education
Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labels
Apr 8, 2026Yaqi Zhao, Haoliang Sun, Yating Wang +2Prompt Learning VLM Adaptation
School of Software, Shandong University, Jinan, China
Is Prompt Selection Necessary for Task-Free Online Continual Learning?
Apr 6, 2026Seoyoung Park, Haemin Lee, Hankook LeeTask-Free Continual Learning Continual Learning
Sungkyunkwan University

Aug 5, 2026J/K move · Enter open · S save

On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing

Simon Lösche, Barış Büyüktaş, Mathis Adler, Angelos Zavras+2

BIFOLD - Berlin Institute for the Foundations of Learning and Data, 10587 Berlin, Germany · Technische Universität Berlin, 10623 Berlin, Germany · Orion Lab, School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens, 15772 Athens, Greece+2

Remote Sensing Image Understanding Prompt Learning VLM Adaptation Non-IID Federated Learning Low-Rank Adaptation Federated Fine-Tuning

Open paper AskPDF arXiv

Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization. This paradigm is particularly relevant in remote sensing (RS), where legal regulations, privacy concerns, and bandwidth constraints restrict data sharing. However, the presence of training data heterogeneity across clients (known as non-IID data) can impede convergence and limit the generalization capability of the aggregated global model. To mitigate the adverse effects of training data heterogeneity, vision-language models (VLMs) can be leveraged in FL due to their transferable representations, which have demonstrated robustness under distribution shifts. However, their large parameter size may substantially increase communication overhead and local computational complexity in federated settings. Therefore, it is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints. To address this issue, in this paper, we present the first comparative study of VLM adaptation strategies for FL in the context of RS image classification. We investigate full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA) tuning, and analyze them with respect to three criteria: 1) generalization capability under non-IID data, 2) communication overhead, and 3) local computational complexity. Experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet reveal distinct trade-offs between task specialization, cross-domain generalization, and efficiency. Based on our findings, we derive a guideline for the selection of an appropriate VLM adaptation strategy in FL for RS image classification under different operational constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/FL-RS-VLM.