cs.CVAug 25, 2026

What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

Authors: Ryo Kamiya, Hiroshi Kera, Kazuhiko Kawamoto

Organizations: Chiba University, Chiba, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Prompt learning adapts vision-language models such as CLIP by optimizing continuous prompt vectors, but the learned prompts are difficult to interpret in natural language. We present PromptSpLiCE, a post-hoc method that expresses each class-conditioned text embedding as a sparse combination of concepts from a fixed natural-language dictionary. Using the same dictionary before and after prompt learning allows us to compare changes in their concept profiles. We evaluate PromptSpLiCE on CoOp, a representative prompt-learning method, across 11 image-classification datasets. The concept profiles change substantially: on average, only 1.6 of the initial top-10 concepts remain in the top 10 after learning. Across datasets, profile change is positively associated with accuracy gain. We also derive a local gradient expression that provides geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.

Figures & tables

Figure 1: Illustration of fitted concept-coefficient changes and the local geometric intuition considered in this work. A dictionary direction parallel to the current prompt has a small tangent component, whereas an image-aligned tangent component can have greater local loss sensitivity.
Figure 2: Improvement in classification accuracy obtained by CoOp relative to the initial CLIP prompt.
Figure 3: Cosine similarity between original prompt embeddings and their full PromptSpLiCE reconstructions, averaged over 11 datasets. Sparsity is controlled by λ∈{0.5,0.4,0.3,0.2,0.1,0.05,0.01}\lambda\in\{0.5,0.4,0.3,0.2,0.1,0.05,0.01\} .
Figure 4: Top-10 dictionary terms and their fitted coefficients before and after prompt learning. The top row shows EuroSAT’s “Annual Crop Land” class, and the bottom row shows StanfordCars’ “2012 FIAT 500 Convertible” class.
DatasetHigh →\rightarrow HighHigh →\rightarrow LowLow →\rightarrow High
EuroSAT0.39.29.4
DTD0.88.38.5
FGVCAircraft1.17.37.2
UCF1011.27.87.8
Caltech1011.57.17.2
Food1011.66.36.5
Table 1: Average number of dictionary terms in each rank-transition group.
Figure 5: Changes in fitted decomposition coefficients for representative classes. Each point is a dictionary term; the horizontal and vertical axes show its coefficient before (CLIP) and after (CoOp) prompt learning, respectively. Selected High →\rightarrow High, High →\rightarrow Low, and Low →\rightarrow High terms are labeled.
Figure 6: Complementary decomposition-coefficient changes for Flowers102 and SUN397. The axes and colors follow Fig. 5 .
Figure 7: Mean fitted coefficient-distribution change versus classification-accuracy improvement across the 11 datasets (Pearson’s r=0.64r=0.64 ). The dashed line is a linear regression fit. The 11-point association is exploratory.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Additional decomposition-coefficient changes for DTD, FGVCAircraft, UCF101, and Caltech101. Colors denote High →\rightarrow High, High →\rightarrow Low, and Low →\rightarrow High transitions.
Figure 9: Additional decomposition-coefficient changes for Food101 and ImageNet.
Figure 10: Additional decomposition-coefficient changes for the OxfordPets class “sphynx.”
More in this topic

Prompt Learning

Prompt learning adapts pretrained models by designing or tuning input prompts, often without changing most model weights.

See all 77 papers

Explore similar work

CardsList

  1. Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

    Jun 21, 2026Na Sang, Ding Ma, Rui Sang +1Prompt LearningVLM Adaptation

    University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher

  2. Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning

    May 6, 2026Yating Wang, Yaqi Zhao, Yongshun Gong +2Prompt LearningVLM Adaptation

    School of Software, Shandong University, Jinan, 250101, China

  3. ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

    Sep 24, 2026Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud AbdarVLM AdaptationVisual Prompt Learning

    University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia