Prompt Tuning for CLIP on the Pretrained Manifold

Xi Yang; Yuanrong Xu; Weigang Zhang; Guangming Lu; David Zhang; Jie Wen

cs.CVFeb 22, 2026

Prompt Tuning for CLIP on the Pretrained Manifold

Authors: Xi Yang, Yuanrong Xu, Weigang Zhang, Guangming Lu, David Zhang, Jie Wen

Organizations: Guizhou University, Guiyang, China · Harbin Institute of Technology, China · The Chinese University of Hong Kong, Shenzhen, China

Abstract

Prompt tuning introduces learnable prompt vectors that adapt pretrained vision-language models to downstream tasks in a parameter-efficient manner. However, under limited supervision, prompt tuning alters pretrained representations and drives downstream features away from the pretrained manifold toward directions that are unfavorable for transfer. This drift degrades generalization. To address this limitation, we propose ManiPT, a framework that performs prompt tuning on the pretrained manifold. ManiPT introduces cosine consistency constraints in both the text and image modalities to confine the learned representations within the pretrained geometric neighborhood. Furthermore, we introduce a structural bias that enforces incremental corrections, guiding the adaptation along transferable directions to mitigate reliance on shortcut learning. From a theoretical perspective, ManiPT alleviates overfitting tendencies under limited data. Our experiments cover four downstream settings: unseen-class generalization, few-shot classification, cross-dataset transfer, and domain generalization. Across these settings, ManiPT achieves higher average performance than baseline methods. Notably, ManiPT provides an explicit perspective on how prompt tuning overfits under limited supervision.

Figures & tables

Figure 1: Manifold drift and manifold preservation on the CLIP feature space. (a) Prompt tuning. Under limited supervision, prompt tuning drives adapted representations away from the pretrained manifold. (b) ManiPT. The proposed method constrains adapted representations to stay close to the pretrained manifold.

Figure 2: Overview of ManiPT. The framework enriches class descriptions with an LLM, constructs a text feature bank as semantic prototypes, and applies cosine consistency constraints together with a structural bias to keep prompt tuning near the pretrained manifold.

Method	(a) Average			(b) ImageNet			(c) Caltech101			(d) OxfordPets
Method	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP	69.34	74.22	71.70	72.43	68.14	70.22	96.84	94.00	95.40	91.17	97.26	94.12
CoOp	82.69	63.22	71.66	76.47	67.88	71.92	98.00	89.81	93.73	93.67	95.29	94.47
CoCoOp	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
MaPLe	82.28	75.14	78.55	76.66	70.54	73.47	97.74	94.36	96.02	95.43	97.76	96.58
LLaMP	85.16	77.71	81.27	77.99	71.27	74.48	98.45	95.85	97.13	96.31	97.74	97.02

Table 1: Base-to-novel generalization results on 11 datasets. We report accuracy on base and novel classes and their harmonic mean (HM). ManiPT achieves the best average harmonic mean across datasets, indicating more balanced generalization between base and novel classes.

	Source	Target
	ImageNet	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Average
CoOp	71.51	93.70	89.14	64.51	68.71	85.30	18.47	64.15	41.92	46.39	66.55	63.88
CoCoOp	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21	65.74
MaPLe	70.72	95.53	90.49	65.57	72.20	86.20	24.74	67.01	46.49	48.06	68.69	66.50
CoPrompt	70.80	94.50	90.73	65.67	72.30	86.43	24.00	67.57	47.07	51.90	69.73	66.99
TAC	72.77	94.53	90.67	65.30	72.20	85.83	23.53	67.63	47.57	48.07	70.00	66.53

Table 2: Cross-dataset transfer results from ImageNet to ten downstream datasets. Models are trained on ImageNet and evaluated in a zero-shot manner on each target dataset. ManiPT attains the highest average accuracy over all target datasets.

Figure 3: Few-shot classification performance averaged over 11 datasets under 1, 2, 4, 8, and 16 shot settings. ManiPT consistently outperforms baseline methods across all shot numbers, with especially clear gains in the 1-shot and 2-shot regimes.

	Source	Target
	ImageNet	-V2	-S	-A	-R	Avg.
CoOp	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp	71.02	64.07	48.75	50.63	76.18	59.91
MaPLe	70.72	64.07	49.15	50.90	76.98	60.27
CoPrompt	70.80	64.25	49.43	50.50	77.51	60.42
TAC	72.77	65.97	50.30	51.73	78.50	61.63

Table 3: Domain generalization results on ImageNet variants. Models trained on ImageNet are tested on distribution-shifted versions such as ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. ManiPT achieves the best average performance and maintains strong accuracy across all domains.

Structural Bias	Cos. Consistency	LLM Enrich.	Base	New	HM
$\times$	$\times$	$\times$	82.91	64.15	72.82
✓	$\times$	$\times$	84.85	68.32	75.71
$\times$	✓	$\times$	82.85	76.50	79.54
✓	✓	$\times$	85.65	77.58	81.42
✓	✓	✓	85.82	78.67	82.09

Table 4: Ablation study of ManiPT components. We report performance for the baseline, variants with a single component removed, and the full ManiPT model. Each component brings improvements over the baseline, and combining cosine consistency, structural bias, and LLM-based enrichment yields the best overall trade-off.

Figure 4: Quantitative analysis of manifold drift using PCA. We compare the distance between prompt-adapted features and pretrained features for different methods.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain / Task	Classes	Train Size	Test Size
Standard Benchmarks
ImageNet	General Objects	1,000	1,281,167	50,000
Caltech101	General Objects	100	4,128	2,465
OxfordPets	Fine-grained (Pets)	37	2,944	3,669
StanfordCars	Fine-grained (Cars)	196	6,509	8,041
Flowers102	Fine-grained (Flowers)	102	4,093	2,463

Appendix

Table 5: Statistics of all 15 datasets used in our experiments, including standard classification benchmarks and ImageNet variants for domain generalization.

	Mean $\bigl\langle\mathbf{f}^{\mathrm{vis}}_{\mathbf{x}},\mathbf{z}^{\mathrm{vis}}_{\mathbf{x}}\bigr\rangle$		Drift $\Delta$
Dataset	$\lambda=12$	$\lambda=0$	$\lambda=12$	$\lambda=0$
ImageNet	0.9952	0.8741	0.0045	0.1452
Caltech101	0.9978	0.8950	-0.0021	0.1050
OxfordPets	0.9965	0.8620	0.0012	0.1523
StanfordCars	0.9910	0.8150	0.0078	0.2105
Flowers102	0.9930	0.8245	-0.0048	0.1950

Appendix

Table 6: Full ablation results on the effect of consistency loss weights across all 11 datasets. We compare the default setting (

\lambda=12

) with the unconstrained setting (

\lambda=0

). The results consistently show that removing the constraint leads to a significant drop in cosine alignment and a sharp increase in manifold drift

\Delta

, confirming the necessity of geometric regularization.

Dataset	Method	1 shot	2 shots	4 shots	8 shots	16 shots
Average	Linear probe CLIP	45.83	57.98	68.01	74.47	78.79
	CoOp	67.56	70.65	74.02	76.98	79.89
	CoCoOp	66.79	67.65	71.21	72.96	74.90
	MaPLe	69.27	72.58	75.37	78.89	81.79
	PromptSRC	72.32	75.29	78.35	80.69	82.87
	ManiPT	74.75	76.77	80.01	82.12	84.64

Appendix

Table 7: Comparison of ManiPT performance with various methods for each dataset in few-shot setting. Best and second-best results are highlighted in bold and underline, respectively.

Strategy	Base	New	HM
Direct Tuning	82.85	76.50	79.54
Gate	85.35	78.40	81.73
Additive	85.82	78.67	82.09

Appendix

Table 8: Comparison of different structural bias strategies. Our additive structure ensures that the adapted features are geometrically anchored by the pretrained manifold, achieving the best trade-off between base retention and novel generalization.

Anchor Source	Base	New	HM
a photo of a {class}	85.65	77.58	81.42
Handcrafted Templates	85.70	77.80	81.56
LLM Descriptions	85.82	78.67	82.09

Appendix

Table 9: Ablation study on different text anchor sources. While handcrafted templates offer a marginal improvement over ”a photo of a {class}” template, LLM descriptions provide rich semantic knowledge, significantly boosting generalization on novel classes.

$\mathcal{L}^{\mathrm{img}}$	$\mathcal{L}^{\mathrm{txt}}$	Base	New	HM
$\times$	$\times$	84.85	68.32	75.71
✓	$\times$	85.45	73.15	78.82
$\times$	✓	85.15	74.80	79.64
✓	✓	85.82	78.67	82.09

Appendix

Table 10: Ablation study on the complementary roles of visual and textual consistency constraints. The results show that

\mathcal{L}^{\mathrm{img}}

and

\mathcal{L}^{\mathrm{txt}}

are complementary. While each single constraint improves over the baseline, combining them yields the best performance across all metrics, confirming their synergistic effect.

Regularization Method	Base	New	HM
No Constraint	84.85	68.32	75.71
L1	83.50	72.45	77.58
L2	78.95	76.10	77.50
Cosine	85.82	78.67	82.09

Appendix

Table 11: Comparison with standard regularization methods. L1 and L2 denote constraining the distance between prompt-adapted features and pretrained features via L1 and L2 distances, respectively.

Variant	Base	New	HM
ManiPT (Full)	85.82	78.67	82.09
w/o Frozen Text ( $\mathbf{f}^{\mathrm{txt,clip}}$ )	83.45	77.25	80.23
w/o Frozen Visual ( $\mathbf{f}^{\mathrm{vis,clip}}$ )	85.40	78.25	81.67

Appendix

Table 12: Ablation study on the components of the structural bias.

Constraint Target	Base	New	HM
Prompt Features (prompt-branch $\mathbf{f}^{\mathrm{vis,prompt}}$ , $\mathbf{f}^{\mathrm{txt,prompt}}$ )	84.65	77.45	80.89
Final Features (fused $\mathbf{f}^{\mathrm{vis,final}}$ , $\mathbf{f}^{\mathrm{txt,final}}$ )	85.82	78.67	82.09

Appendix

Table 13: Ablation study on the target of consistency constraints. We compare applying the constraints to the final fused representations versus applying them only to the intermediate prompt features. The results show that constraining the final representation yields better performance.

Initialization	Base	New	HM
Kaiming Uniform	85.45	78.10	81.61
Normal (std=0.02)	85.82	78.67	82.09

Appendix

Table 14: Ablation study on initialization strategies. We compare the Normal initialization with Kaiming Uniform initialization. The results indicate that ManiPT is relatively robust to initialization strategies, though Normal initialization yields slightly better performance by avoiding excessive initial variance.

Method	Base	New	HM
Txt-only Deep Prompting	85.45	69.40	76.60
Vis-only Deep Prompting	84.20	74.85	79.25
ManiPT	85.82	78.67	82.09

Appendix

Table 15: Ablation study on modality-specific deep prompting. We compare our dual-branch design with single-branch deep prompting variants namely Visual-only and Textual-only. Vis-only and Txt-only denote applying deep prompting, consistency constraints, and structural bias exclusively to the image or text encoder respectively.

Figure 5: Sensitivity analysis of the consistency constraint weight.

Figure 6: Sensitivity analysis of the context length. Unlike standard prompt tuning where increasing parameters often leads to overfitting, ManiPT effectively utilizes the increased capacity to boost both Base and Novel performance.

Method	Training Time	Peak Memory	Infer. Latency	Params
	(s/epoch)	(GB)	(ms/img)	(M)
Zero-shot CLIP	N/A	N/A	3.42	0.00
CoOp	29.12	8.33	3.50	0.01
CoCoOp	836.08	8.25	39.97	0.04
MaPLe	241.51	8.30	3.83	3.56
PromptSRC	248.49	8.63	3.87	0.05

Appendix

Table 16: Training and inference efficiency comparison. OOM denotes out of memory.

	MaPLe				ManiPT
Dataset	$d^{\mathrm{pca}}=32$	$d^{\mathrm{pca}}=64$	$d^{\mathrm{pca}}=128$	$d^{\mathrm{pca}}=256$	$d^{\mathrm{pca}}=32$	$d^{\mathrm{pca}}=64$	$d^{\mathrm{pca}}=128$	$d^{\mathrm{pca}}=256$
ImageNet	0.1320	0.1250	0.1085	0.0750	0.0038	0.0045	0.0051	0.0055
Caltech101	0.0980	0.0910	0.0750	0.0480	-0.0025	-0.0021	-0.0010	0.0005
OxfordPets	0.0890	0.0840	0.0680	0.0420	0.0008	0.0012	0.0015	0.0020
StanfordCars	0.1585	0.1520	0.1350	0.0950	0.0072	0.0078	0.0082	0.0085
Flowers102	0.1450	0.1380	0.1120	0.0750	-0.0055	-0.0048	-0.0030	-0.0010

Appendix

Table 17: The manifold drift metric

\Delta

with respect to the PCA rank

d^{\mathrm{pca}}

across all 11 datasets.

Symbol	Description
$\mathcal{X}$ , $\mathcal{Y}$	Input image space and label space
$\mathcal{C}=\{1,\dots,C\}$	Class index set, $C$ denotes the number of classes
$\mathcal{D}$	Data distribution over $\mathcal{X}\times\mathcal{Y}$
$\mathcal{D}^{\mathrm{train}}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}$	Training dataset, $N$ denotes the number of samples
$T^{\mathrm{all}},\,T^{\star}_{c}$	Sets of all and class-specific token sequences for the text encoder
$d$	Embedding dimension of CLIP features

Appendix

Table 18: Notation for core sets, indices, and scalars.

Symbol	Description
$\mathcal{T},\,\mathcal{V}$	Frozen CLIP text and image encoders (prompts are provided as additional inputs during tuning)
$\mathbf{z}^{\mathrm{txt}}_{c}$	Frozen CLIP text feature for class $c$
$\mathbf{z}^{\mathrm{vis}}_{\mathbf{x}}$	Frozen CLIP visual feature for image $\mathbf{x}$
$\mathbf{h}^{\mathrm{txt}}_{c}$	Prompt-branch text feature for class $c$
$\mathbf{h}^{\mathrm{vis}}_{\mathbf{x}}$	Prompt-branch visual feature for image $\mathbf{x}$
$\mathbf{f}^{\mathrm{txt}}_{c}$	Final fused text feature for class $c$

Appendix

Table 19: Notation for features, prompts, and manifolds.

Explore similar work

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation
Jun 21, 2026Na Sang, Ding Ma, Rui Sang +1Prompt Learning VLM Adaptation
University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
Aug 14, 2026Liang Wang, Haoyang Li, Chao Wang +3VLM Adaptation VLM Robustness
Shanghai University, Shanghai, China · University of Technology Sydney, Sydney, Australia
Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
May 12, 2026Boyang Guo, Liang Li, Lin Peng +3Long-Tail Learning VLM Adaptation
Hangzhou Dianzi University, Hangzhou, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China +2

Jun 21, 2026J/K move · Enter open · S save

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

Na Sang, Ding Ma, Rui Sang, Yuxuan Liu

University of California, San Diego, La Jolla, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA · Independent Researcher

Prompt Learning VLM Adaptation Few-Shot Prompting

Open paper AskPDF arXiv

Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.

Abstract

Explore similar work

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning

Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models