Is Prompt Selection Necessary for Task-Free Online Continual Learning?

Seoyoung Park; Haemin Lee; Hankook Lee

cs.LGApr 6, 2026

Is Prompt Selection Necessary for Task-Free Online Continual Learning?

Authors: Seoyoung Park, Haemin Lee, Hankook Lee

Organizations: Sungkyunkwan University

Abstract

Task-free online continual learning has recently emerged as a realistic paradigm for addressing continual learning in dynamic, real-world environments, where data arrive in a non-stationary stream without clear task boundaries and can only be observed once. To consider such challenging scenarios, many recent approaches have employed prompt selection, an adaptive strategy that selects prompts from a pool based on input signals. However, we observe that such selection strategies often fail to select appropriate prompts, yielding suboptimal results despite additional training of key parameters. Motivated by this observation, we propose a simple yet effective SinglePrompt that eliminates the need for prompt selection and focuses on classifier optimization. Specifically, we simply (i) inject a single prompt into each self-attention block, (ii) employ a cosine similarity-based logit design to alleviate the forgetting effect inherent in the classifier weights, and (iii) mask logits for unexposed classes in the current minibatch. With this simple task-free design, our framework achieves state-of-the-art performance across various online continual learning benchmarks. Source code is available at https://github.com/efficient-learning-lab/SinglePrompt.

Figures & tables

Figure 1: An overview of the proposed SinglePrompt . When minibatch $\mathcal{B}_{t}=\{(\mathbf{x}_{t}^{(i)},y_{t}^{(i)})\}_{i=1}^{B_{t}}$ is provided, it passes though a pretrained Vision Transformer encoder. At the $i$ -th self-attention block $f_{i}$ , the input sequence $\mathbf{h}_{i-1}$ is given, and during the attention operation the learnable prompts $\mathbf{p}_{i}^{k}$ and $\mathbf{p}_{i}^{v}$ are prepended to the key and value, respectively. Only the class token from the encoder’s output sequence is used as the final representation. The cosine similarity between this representation and the class prototypes is computed and used as the logit values for prediction. The logit values corresponding to labels not exposed in the current minibatch are masked out before computing the cross-entropy loss.

Panel of Figure 2 — Figure 2: Histograms of prompt selection counts per class on task-free continual learning using CIFAR100 [ 13 ] . In all histograms, the x-axis represents 100 class IDs and the y-axis denotes the number of times each prompt is selected. (a) L2P [ 22 ] is evaluated on the 10-task class-incremental setting and the model selects the top-5 prompts per input from a prompt pool. (b) MVP [ 18 ] and (c) MISA [ 10 ] are evaluated on the 5-task Si-Blurry setting. Note that both methods select only one prompt per input image unlike L2P.

Panel of Figure 3 — Figure 3: Prompt selection failures in task-based methods on CIFAR100 [ 13 ] . (a) Task identification accuracy (%) via prompt selection. The x-axis represents the 100 class IDs, sorted in ascending order of their task identification accuracy, which is shown on the y-axis as the proportion of correct prompt selection. (b) The average and standard variation of cosine similarities computed from the 7-th layer. The x-axis represents the task ID of input sample and the y-axis indicates the average cosine similarity between each task’s samples and their assigned keys. Ideally, all value should be close to 1, but most are near 0. Note that consistent results are observed across other layers as well.

	# of Prompts	Selection	Params	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)
(a)	10	Similarity-based	560K	79.89 $\pm$ 2.14	82.12 $\pm$ 1.66
(b)	10	Random	553K	79.78 $\pm$ 2.03	81.85 $\pm$ 1.78
(c)	1	N/A	138K	81.05 $\pm$ 2.18	82.38 $\pm$ 1.95

Table 1: Ablation study on the prompt selection. Note that (a) is same as MISA [ 10 ] without pretrained prompts.

		CIFAR100		Tiny-ImageNet		ImageNet-R
Buffer	Method	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)
0	Finetuning	19.71 $\pm$ 3.39	10.42 $\pm$ 4.92	15.50 $\pm$ 0.74	10.42 $\pm$ 4.92	7.51 $\pm$ 3.94	2.29 $\pm$ 0.85
	Linear Probe	49.69 $\pm$ 6.09	23.07 $\pm$ 7.33	42.15 $\pm$ 2.79	21.97 $\pm$ 6.43	29.24 $\pm$ 1.26	16.87 $\pm$ 3.14
	EWC [ 11 ]	49.51 $\pm$ 0.52	52.83 $\pm$ 2.31	51.70 $\pm$ 2.89	31.04 $\pm$ 3.12	31.58 $\pm$ 1.04	20.72 $\pm$ 1.11
	LwF [ 17 ]	55.51 $\pm$ 3.49	36.53 $\pm$ 10.96	49.00 $\pm$ 1.52	27.47 $\pm$ 7.59	31.61 $\pm$ 1.53	20.62 $\pm$ 3.67
	L2P [ 22 ]	57.08 $\pm$ 4.43	41.63 $\pm$ 12.73	52.09 $\pm$ 1.92	35.05 $\pm$ 5.73	29.65 $\pm$ 1.63	19.55 $\pm$ 4.78

Table 2: Accuracy (%) on CIFAR100 [ 13 ] , Tiny-Imagenet [ 14 ] , Imagenet-R [ 7 ] . We average the performance over 5 random seeds (

\text{Mean}\pm\text{Std}

) for ours. Others reported numbers borrow from MISA [ 10 ] . Bold indicates the best performance, while underlined entries denote the second-best results.

Figure 4: Visualization of the L2 norms of the weights for each class of the linear classifier, with blurry classes excluded. The x-axis shows class IDs in the order the model observes them.

PEFT Adapter	Logit type	Batch Masking	K	$\#\text{Params}$	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)	$F_{\text{last}}$ (↓)
Prompt tuning	cosine similarity	✓	N/A	92K	81.68 $\pm$ 1.34	84.37 $\pm$ 0.38	5.68 $\pm$ 0.92
LoRA	cosine similarity	✓	N/A	138K	82.21 $\pm$ 1.74	81.25 $\pm$ 1.64	11.45 $\pm$ 0.76
AdaptMLP	cosine similarity	✓	N/A	203K	83.74 $\pm$ 1.97	84.34 $\pm$ 0.42	9.07 $\pm$ 0.66
Prefix tuning	cosine similarity	✓	5	230K	85.58 $\pm$ 1.46	87.53 $\pm$ 0.45	5.89 $\pm$ 0.99
Prefix tuning	linear classifier	✓	5	230K	81.63 $\pm$ 1.98	82.37 $\pm$ 1.75	10.82 $\pm$ 1.57
Prefix tuning	cosine similarity	✓	5	230K	85.58 $\pm$ 1.46	87.53 $\pm$ 0.45	5.89 $\pm$ 0.99

Table 3: Ablation studies on different PEFT Adapters, logit types, batch masking and K values conducted on CIFAR100 [ 13 ] . The prompt length is fixed at 20 in all experiments involving prompts. We average the performance over 5 random seeds (

\text{Mean}\pm\text{Std}

). Bold indicates the best result within each ablation setting.

Disjoint class ratio (%)	Method	$A_{\text{auc}}$ (↑)	$A_{\text{last}}$ (↑)
0	MISA [ 10 ]	77.91 $\pm$ 1.63	82.26 $\pm$ 0.92
0	SinglePrompt	83.26 $\pm$ 0.89	88.15 $\pm$ 0.18
25	MISA [ 10 ]	79.13 $\pm$ 2.55	81.36 $\pm$ 1.01
25	SinglePrompt	84.65 $\pm$ 2.29	87.67 $\pm$ 0.52
50	MISA [ 10 ]	80.55 $\pm$ 2.17	80.98 $\pm$ 1.08
50	SinglePrompt	85.58 $\pm$ 1.46	87.53 $\pm$ 0.45

Table 4: Comparison study on disjoint class ratios conducted on CIFAR100 [ 13 ] . We average the performance over 5 random seeds (

\text{Mean}\pm\text{Std}

). Bold indicates the best result.

Figure 5: Ablation study on Prompt length. The x-axis denotes the value of $M$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Anytime inference accuracy curves of SinglePrompt and MISA [ 10 ] on CIFAR100 [ 13 ] . Accuracy is measured every 1,000 training samples, and the curves visualize the mean and standard deviation over five random seeds.

Figure 7: Visualization of the Si-blurry scenario. The dataset for the $k$ -th task, $\mathcal{D}_{k}$ , consists of the disjoint dataset $T_{k}^{D}$ and the blurry dataset $T_{k}^{B}$ .

Panel of Figure 8 — Figure 8: Prompt selection failures on additional datasets, showing that the observed failure patterns generalize beyond CIFAR-100.

Panel of Figure 9 — Figure 9: Failure of ConvPrompt [ 20 ] selection in task-based continual learning on CIFAR100 [ 13 ] . The x-axis represents the task ID of input sample and the y-axis indicates the average cosine similarity between each task’s samples and their assigned keys. (a) Result in the offline continual learning setting, where class information of upcoming tasks is available. Task similarity is computed using class descriptors, which determines the number of prompts allocated to each task (22 prompts in total). (b) Result in the online continual learning setting, where future task classes are unknown. Therefore, the number of prompts assigned to each task is fixed to 5, resulting in 50 prompts in total. In both cases, the cosine similarity is expected to be high for all tasks but remains close to zero. Results are shown for the 7-th layer. Note that consistent results are observed across other layers as well.

Method	#Params	FLOPs	Total time (s)
MVP [ 18 ]	$562K$	$34.21G$	2731.43
MISA [ 10 ]	$576K$	$35.67G$	3405.24
Online-LoRA [ 23 ]	$524K$	$17.64G$	1764.47
SinglePrompt	$230K$	$17.61G$	1589.74

Appendix

Table 5: Computational statistics for ours on CIFAR100 [ 13 ] without buffer. We compare our method with MISA [ 10 ] , MVP [ 18 ] and online-LoRA [ 23 ] . FLOPs refer to forward pass computations only, and total time is averaged over five runs with different random seeds. #Params refers to the number of learnable parameters. All methods use the same ViT-B/16 architecture to ensure a fair comparison.

Task-Free Continual Learning

See all 19 papers

Information-Theoretic Decoupled Prompt Tuning for Continual Learning
Aug 13, 2026Yunfei Zhang, Wen Wen, Tieliang Gong +1Continual Learning for VLMs Continual Learning for LLMs
National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Learning Probabilistic Prompt for Continual Learning
Jul 6, 2026Hyekang Park, Sanghoon Lee, Geon Lee +2Class-Incremental Learning Prompt Learning
1Yonsei University, Seoul, South Korea · 2Samsung Electronics, Suwon, South Korea
LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models
Jun 8, 2026Mingqi Yuan, Xiaoquan Sun, Shihao Luo +1Distribution Shift Concept Drift
Agentic Intelligence Lab | HKU · Qicore Tech

Aug 13, 2026J/K move · Enter open · S save

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Yunfei Zhang, Wen Wen, Tieliang Gong, Weizhan Zhang

National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Continual Learning for VLMs Continual Learning for LLMs VLM Adaptation

Open paper AskPDF arXiv

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Abstract

Task-Free Continual Learning

Explore similar work

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Learning Probabilistic Prompt for Continual Learning

LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models