cs.LGJun 21, 2026

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

Authors: Na Sang, Ding Ma, Rui Sang, Yuxuan Liu

Abstract

Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.

Explore similar work

CardsList
Sep 8, 2026cs.CV

Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases

Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense prompt matrix P∈Rm×d\mathbf{P}\in\mathbb{R}^{m\times d} trained from only a few examples per class. We study whether this matrix is over-parameterized by factorizing it as P=BA\mathbf{P}=\mathbf{B}\mathbf{A}, which cuts the trainable prompt parameters from mdmd to r(m+d)r(m+d), and to rdrd once the token-side factor B\mathbf{B} is fixed. Across seven few-shot benchmarks and two CLIP backbones, low-rank prompts match or improve dense CoOp at far fewer parameters, with the clearest gains on low-shot base-to-new generalization. We then find that the token-side factor need not be learned at all: fixing B\mathbf{B} to a Gaussian, orthogonal, SVD-derived, or even random basis and training only the embedding-side factor A\mathbf{A} stays on par with the fully trainable factorization, and a source-trained B\mathbf{B} offers no advantage over a random one. A prompt-factor asymmetry and a local update-space dimension gap show why fixing B\mathbf{B} is far less restrictive than fixing A\mathbf{A}, and a smoothness-only guarantee certifies that optimizing A\mathbf{A} over a fixed B\mathbf{B} converges. In the CLIP prompt setting, the embedding-side coefficients carry the adaptation while the token basis can simply be fixed.
Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1
Contrastive Language-Image Pre-TrainingSoft Prompt TuningPrompt OptimizationAsymmetryCoefficients
AskPDF
May 12, 2026cs.CV

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
Yaze Zhao, Yicong Liu, Yixiong Zou +2
Few-Shot LearningVision-Language Model AdaptationContrastive Language-Image Pre-TrainingContrastive LossModality Alignment
AskPDF
Jun 5, 2026cs.CV

GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection

We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection. While CLIP achieves strong zero-shot recognition, it yields deterministic similarity scores and offers limited uncertainty information, which is critical under distribution shift and data scarcity. GP-Adapter constructs modality-specific, class-wise one-class GPs on top of frozen CLIP embeddings using an RBF kernel for image features and a linear kernel for text prompts and fuses their predictive statistics to produce a variance-aware confidence score for OOD detection. The method requires no fine-tuning of the CLIP backbone and relies only on a small KK-shot cache and lightweight hyperparameter selection, with memory cost scaling as O(CK2)O(CK^2) for CC classes and KK shots. Experiments on ImageNet and multiple OOD benchmarks show that GP-Adapter provides competitive few-shot performance and consistently improves OOD detection when combined with prompt-learning baselines, highlighting the complementarity between GP-based uncertainty modeling and prompt learning. Overall, our results suggest that integrating probabilistic inference with large pre-trained vision-language models can improve reliability in low-data and distribution-shifted settings. Code is available at https://github.com/tms-byte/GP-Adapter
Taisei Saito, Koretaka Ogata, Takafumi Hiroi
Out-Of-Distribution Detection PerformanceContrastive Language-Image Pre-TrainingOut-Of-DistributionAdaptersGaussian Process
AskPDF