ProTPS: Prototype-Guided Text Prompt Selection for Continual Learning

Jie Mei; Li-Leng Peng; Keith Fuller; Jenq-Neng Hwang

cs.CVApr 1, 2026

ProTPS: Prototype-Guided Text Prompt Selection for Continual Learning

Authors: Jie Mei, Li-Leng Peng, Keith Fuller, Jenq-Neng Hwang

Organizations: University of Washington, Seattle, WA, USA · Alaska Pacific University, Anchorage, Alaska, USA

Abstract

For continual learning, text-prompt-based methods leverage text encoders and learnable prompts to encode semantic features for sequentially arrived classes over time. A common challenge encountered by existing works is how to learn unique text prompts, which implicitly carry semantic information of new classes, so that the semantic features of newly arrived classes do not overlap with those of trained classes, thereby mitigating the catastrophic forgetting problem. To address this challenge, we propose a novel approach Prototype-guided Text Prompt Selection (ProTPS)'' to intentionally increase the training flexibility thus encouraging the learning of unique text prompts. Specifically, our ProTPS learns class-specific vision prototypes and text prompts. Vision prototypes guide the selection and learning of text prompts for each class. We first evaluate our ProTPS in both class incremental (CI) setting and cross-datasets continual (CDC) learning setting. Because our ProTPS achieves performance close to the upper bounds, we further collect a real-world dataset with 112 marine species collected over a span of six years, named Marine112, to bring new challenges to the community. Marine112 is authentically suited for the class and domain incremental (CDI) learning setting and is under natural long-tail distribution. The results under three settings show that our ProTPS performs favorably against the recent state-of-the-art methods. The implementation code and Marine112 dataset will be released upon the acceptance of our paper.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of learned text prompts via Grad-CAM [ 38 ] . Highlighted image areas are the attended regions of the learned text prompt of each class. Each pair of adjacent columns, moving from left to right, represents the same class: “african hunting dog”, “chihuahua”, “borzoi”, “little blue heron”, “american coot”.

Figure 2: Prior work [ 65 ] proposes learnable prompts to replace predefined prompt templates. Our work introduces a novel text prompt selection approach that enables semantic prompts of each class to interact with all other class names. Note that three big gray circles in the feature space represent all possible semantic features for each class.

Figure 3: ProTPS framework. In Task-1, we use the prototype $I_{i}$ of class $i$ to initialize the weights of the linear $classifier_{1}$ , which is referred to as ProTPS’s “vision classifier”. We finetune these vision prototypes for refinement. Each class prototype $I_{i}$ is paired with a learnable text prompt $P_{i}$ . For a single input image, we select a text prompt to concatenate with all class names in Task-1 to create $classifier_{2}$ , which is referred to as the “text classifier” of ProTPS. In Task-2, prototypes and paired prompts are expanded. We freeze prototypes $I_{1},...,I_{n}$ and prompts $P_{1},...,P_{n}$ trained in the previous task. We propose a sampling method to create $sampled$ $classifier_{2}$ to help new text prompts capture exclusive unique features for new classes. For the later tasks, the expansion rule is the same as Task-2.

Prompt	Method	T-1	T-2	T-3	T-4	T-5	T-6	T-7	T-8	T-9	T-10
$visual$	L2P [ 54 ]	60.9	53.5	49.8	48.6	47.2	48.1	49.5	49.7	48.1	48.3
	DualPrompt [ 53 ]	72.5	60.8	55.7	54.3	53.2	53.6	53.8	53.3	52.8	52.6
	CODA-P [ 41 ]	91.1	86.5	82.9	80.3	79.8	78.2	77.1	77.2	75.6	75.2
	HiDe-Prompt [ 49 ]	92.3	90.1	89.8	87.5	85.5	85.9	84.5	85.9	83.5	82.9
$text$	CoOp [ 65 ] (1000)	89.2	83.2	76.7	79.8	79.9	82.3	79.7	80.1	80.3	79.3
	Continual-CLIP [ 43 ]	93.3	87.6	83.1	81.7	80.5	80.2	79.3	78.5	76.9	75.4

Table 1: Accuracy of different continual learning methods on ImageNet100 [ 2 ] under CI setting. Second-best results are underlined. T-i denotes Task-i.

Prompt	Method	T-1	T-2	T-3	T-4	T-5	T-6	T-7	T-8	T-9	T-10
$visual$	L2P [ 54 ]	60.1	47.8	45.9	47.1	48.9	48.9	49.4	49.0	50.0	49.4
	DualPrompt [ 53 ]	73.9	60.2	56.4	55.3	54.5	53.3	53.4	53.1	53.5	54.0
	CODA-P [ 41 ]	94.3	87.3	83.7	82.7	81.3	78.2	76.7	74.0	74.1	72.3
	HiDe-Prompt [ 49 ]	95.4	93.6	90.7	89.5	87.9	85.5	84.0	82.0	82.2	81.0
$text$	CoOp [ 65 ] (1000)	95.8	90.7	85.2	83.4	80.8	75.8	74.7	71.7	71.3	67.6
	Continual-CLIP [ 43 ]	95.3	92.3	82.1	79.2	78.5	77.9	75.7	75.2	74.8	73.4

Table 2: Accuracy of different continual learning methods on CIFAR100 [ 17 ] under CI setting. Second-best results are underlined. T-i denotes Task-i.

Figure 4: Accuracy evolution of ProTPS’s different classifiers on ImageNet100 [ 2 ] under CI setting.

Prompt	Method	I2C	FT $\uparrow$
$visual$	L2P-1 [ 54 ]	45.7	-3.7
	DualPrompt-1 [ 53 ]	49.5	-4.5
	CODA-P-1 [ 41 ]	66.9	-5.4
	HiDe-Prompt-1 [ 49 ]	77.2	-3.8
$text$	CoOp-1 [ 65 ] (1000)	61.1	-6.5
	Continual-CLIP [ 43 ]	73.4	0

Table 3: Last accuracy of different methods on CIFAR100 under CDC setting. The models are continually fine-tuned on CIFAR100 after being trained on ImageNet100 (I2C).

Prompt	Method	I+C
$visual$	L2P-2 [ 54 ]	41.6
	DualPrompt-2 [ 53 ]	45.8
	CODA-P-2 [ 41 ]	62.7
	HiDe-Prompt-2 [ 49 ]	75.3
$text$	CoOp-2 [ 65 ] (1000)	55.4
	Continual-CLIP [ 43 ]	65.9

Table 4: Last accuracy of different methods when continually trained on ImageNet100 and CIFAR100 (I+C) in sequence under CDC setting.

Dataset	Authentic CL	Setting	Distribution	#Classes	Images	Fine-grained
Rotation MNIST [ 24 ]	$\times$	DI	balanced	10	synthetic	$\times$
Permutation MNIST [ 16 ]	$\times$	DI	balanced	10	synthetic	$\times$
NS-MiniImageNet [ 25 ]	$\times$	DI	balanced	100	synthetic	$\times$
CLEAR [ 21 ]	$\times$	DI	balanced	100	real	$\times$
DS-ImageNet-R [ 41 ]	$\times$	CDI 1 1 1 DS-ImageNet-R introduces domain shifts that occur between classes, rather than within each class, as reflected in our training data.	balanced	200	real	$\times$
CORe50-NIC [ 23 ]	$\times$	CDI 2 2 2 CORe50-NIC aims to evaluate the model’s generalization to unseen domains, rather than its performance on domains learned incrementally, as demonstrated by our test protocol.	balanced	50	real	$\times$

Table 5: Comparison of Marine112 with public datasets with domain shift. “CL”: Continual learning. “DI”: Domain incremental learning. “CDI”: Class and domain incremental learning.

Figure 5: Marine112 captures both ecological dynamics (incremental domains and classes) and natural longtail distribution.

Prompt	Method	2015	2016	2019	2020	2021	2022
$visual$	CODA-P [ 41 ]	71.2	42.1	33.6	13.4	12.7	13.2
$visual$	HiDe-Prompt [ 49 ]	73.4	50.8	45.2	22.2	22.3	22.5
$text$	Continual-CLIP [ 43 ]	12.7	11.8	10.2	9.6	10.1	9.5
	Zero-Shot CLIP [ 35 ]	15.4	13.1	11.6	10.3	10.5	9.6
	CoOp [ 65 ] (1000)	65.3	30.2	15.1	10.2	11.4	10.3
	AttriCLIP [ 50 ]	76.1	48.3	34.8	11.9	13.6	13.9

Table 6: Accuracy on real-world Marine112 dataset under CDI setting.

PL	Prompt-Selection	STC	$Loss_{pp}$	Last acc.
refined	weight-top-1	✓	✓	83.28
refined	weight-top-1	$\times$	✓	80.95 (-2.33)
refined	weight-top-1	✓	$\times$	80.87 (-2.41)
frozen	weight-top-1	✓	✓	80.65 (-2.63)
refined	$\times$	✓	✓	77.62 (-5.66)
scratch	weight-top-1	✓	✓	58.42 (-24.86)

Table 7: Ablation study on CIFAR100.“PL”: Prototype Learning strategy. “STC”: Sampled Text Classifier strategy.

Figure 6: Comparison of t-SNE [ 46 ] visualization on text prompts for 24 dog breeds from ImageNet100 [ 2 ] .

Figure 7: Similarity matrix of prototypes on CIFAR100. Left: Initialized prototypes via the frozen CLIP image encoder. Right: ProTPS’s refined vision prototypes.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Marine112. Data distribution in each year.

Figure 9: Marine112. Number of kept test data from each year.

Figure 10: Comparison of different (a) loss factor $\lambda_{pp}$ values and different (b) prompt length $M$ values on CIFAR100 [ 17 ] .

Aggregation	Last acc.
Average	93.92
Max	93.86 (-0.06)

Appendix

Table 8: Different aggregation methods on ImageNet100.

Prompt	Method	CIFAR100	ImageNet100
-	SLCA [ 62 ]	79.1	56.3
$visual$	L2P [ 54 ]	49.4	48.3
	DualPrompt [ 53 ]	54.0	52.6
	APG [ 42 ]	54.2	53.1
	CODA-P [ 41 ]	72.3	75.2
	HiDe-Prompt [ 49 ]	81.0	82.9

Appendix

Table 9: Last accuracy on CIFAR100 [ 17 ] and ImageNet100 [ 2 ] under CI setting.

Prompt	Method	C	I2C	FT $\uparrow$
$v+t$	S-liPrompts [ 52 ]	58.9	53.3	-5.6
$visual$	L2P-1 [ 54 ]	49.4	45.7	-3.7
	DualPrompt-1 [ 53 ]	54.0	49.5	-4.5
	CODA-P-1 [ 41 ]	72.3	66.9	-5.4
	HiDe-Prompt-1 [ 49 ]	81.0	77.2	-3.8
$text$	CoOp-1 [ 65 ] (1000)	67.6	61.1	-6.5

Appendix

Table 10: Last accuracy of different methods on CIFAR100 [ 17 ] under CDC setting. The models are either trained on CIFAR100 only (C), or continually fine-tuned on CIFAR100 after being trained on ImageNet100 (I2C).

Prompt	Method	I+C
$v+t$	S-liPrompts [ 52 ]	39.6
$visual$	L2P-2 [ 54 ]	41.6
	DualPrompt-2 [ 53 ]	45.8
	CODA-P-2 [ 41 ]	62.7
	HiDe-Prompt-2 [ 49 ]	75.3
$text$	CoOp-2 [ 65 ] (1000)	55.4

Appendix

Table 11: Last accuracy of different methods on ImageNet100 + CIFAR100 (I+C) where each model is continually trained on ImageNet100 and CIFAR100 in sequence under CDC setting.

Loss function	Last acc.
$Loss_{pp}$ (ours)	83.28
SupCon Loss [ 13 ]	82.57 (-0.71)

Appendix

Table 12: Different loss functions on CIFAR100 [ 17 ] .

Method	CIFAR100	ImageNet100
Linear Probe CLIP	86.69	95.52
ProTPS (ours)	87.19	95.86

Appendix

Table 13: Upper bound accuracy of ‘Linear Probe CLIP’ [ 35 ] and our ProTPS on CIFAR100 [ 17 ] and ImageNet100 [ 2 ] .

Figure 11: Ablation study of ProTPS’s on ImageNet100 [ 2 ] under CI setting. The “vision classifier” and “text classifier” of ProTPS are well-aligned when using the proposed prototype learning strategy.

Method	Classifier	T-1	T-2	T-3	T-4	T-5	T-6	T-7	T-8	T-9	T-10
Continual-CLIP [ 43 ]	text	94.7	88.5	76.8	73.6	72.4	72.8	69.6	68.5	68.1	66.7
Zero-Shot CLIP [ 35 ]	text	95.6	89.2	77.7	74.6	73.9	74.1	70.6	70.1	69.6	68.3
Prototypes-CLIP	vision	95.6	90.7	81.5	77.1	75.9	75.7	72.8	72.3	71.4	69.9
Continual-CLIP [ 43 ]	text	95.3	92.3	82.1	79.2	78.5	77.9	75.7	75.2	74.8	73.4
Zero-Shot CLIP [ 35 ]	text	95.7	92.9	85.2	82.9	82.3	82.1	79.6	79.6	79.2	78.3
Prototypes-CLIP	vision	97.6	92.8	88.1	85.3	84.3	84.3	81.9	81.9	81.2	80.0

Appendix

Table 14: Pilot Study. Comparison of different CLIP-based methods in CI setting on CIFAR100 [ 17 ] with ViT-B/16 [ 3 ] (first 3 rows) or ViT-L/14 [ 3 ] (second 3 rows) backbone. After learning each task

T-i

, the accuracy on exposed classes from all the trained tasks (i.e.,

T-1,2,...,t

) is reported. The upper-bound method is ‘Linear Probe CLIP’ [ 35 ] using all tasks’ training data at the same time to optimize a linear classifier. The column ‘Classifier’ denotes the encoder from which the classifier weights are derived.

Panel of Figure 12 — Figure 12: 1st and 4th column: CLIP. 2nd and 5th column: Image. 3rd and 6th column: Ours. Text prompts visualized via Grad-CAM [ 38 ] on randomly selected classes of ImageNet100.

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 77 papers

Information-Theoretic Decoupled Prompt Tuning for Continual Learning
Aug 13, 2026Yunfei Zhang, Wen Wen, Tieliang Gong +1Continual Learning for VLMs Continual Learning for LLMs
National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Learning Probabilistic Prompt for Continual Learning
Jul 6, 2026Hyekang Park, Sanghoon Lee, Geon Lee +2Class-Incremental Learning Prompt Learning
1Yonsei University, Seoul, South Korea · 2Samsung Electronics, Suwon, South Korea
Is Prompt Selection Necessary for Task-Free Online Continual Learning?
Apr 6, 2026Seoyoung Park, Haemin Lee, Hankook LeeTask-Free Continual Learning Continual Learning
Sungkyunkwan University

Aug 13, 2026J/K move · Enter open · S save

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Yunfei Zhang, Wen Wen, Tieliang Gong, Weizhan Zhang

National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Continual Learning for VLMs Continual Learning for LLMs VLM Adaptation

Open paper AskPDF arXiv

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Abstract

Continual Learning for VLMs

Explore similar work

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Learning Probabilistic Prompt for Continual Learning

Is Prompt Selection Necessary for Task-Free Online Continual Learning?