cs.LGApr 6, 2026

Is Prompt Selection Necessary for Task-Free Online Continual Learning?

Authors: Seoyoung Park, Haemin Lee, Hankook Lee

Organizations: Sungkyunkwan University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Task-free online continual learning has recently emerged as a realistic paradigm for addressing continual learning in dynamic, real-world environments, where data arrive in a non-stationary stream without clear task boundaries and can only be observed once. To consider such challenging scenarios, many recent approaches have employed prompt selection, an adaptive strategy that selects prompts from a pool based on input signals. However, we observe that such selection strategies often fail to select appropriate prompts, yielding suboptimal results despite additional training of key parameters. Motivated by this observation, we propose a simple yet effective SinglePrompt that eliminates the need for prompt selection and focuses on classifier optimization. Specifically, we simply (i) inject a single prompt into each self-attention block, (ii) employ a cosine similarity-based logit design to alleviate the forgetting effect inherent in the classifier weights, and (iii) mask logits for unexposed classes in the current minibatch. With this simple task-free design, our framework achieves state-of-the-art performance across various online continual learning benchmarks. Source code is available at https://github.com/efficient-learning-lab/SinglePrompt.

Figures & tables

Figure 1: An overview of the proposed SinglePrompt . When minibatch Bt={(xt(i),yt(i))}i=1Bt\mathcal{B}_{t}=\{(\mathbf{x}_{t}^{(i)},y_{t}^{(i)})\}_{i=1}^{B_{t}} is provided, it passes though a pretrained Vision Transformer encoder. At the ii -th self-attention block fif_{i} , the input sequence hi−1\mathbf{h}_{i-1} is given, and during the attention operation the learnable prompts pik\mathbf{p}_{i}^{k} and piv\mathbf{p}_{i}^{v} are prepended to the key and value, respectively. Only the class token from the encoder’s output sequence is used as the final representation. The cosine similarity between this representation and the class prototypes is computed and used as the logit values for prediction. The logit values corresponding to labels not exposed in the current minibatch are masked out before computing the cross-entropy loss.
Figure 2: Histograms of prompt selection counts per class on task-free continual learning using CIFAR100 [ 13 ] . In all histograms, the x-axis represents 100 class IDs and the y-axis denotes the number of times each prompt is selected. (a) L2P [ 22 ] is evaluated on the 10-task class-incremental setting and the model selects the top-5 prompts per input from a prompt pool. (b) MVP [ 18 ] and (c) MISA [ 10 ] are evaluated on the 5-task Si-Blurry setting. Note that both methods select only one prompt per input image unlike L2P.
Figure 3: Prompt selection failures in task-based methods on CIFAR100 [ 13 ] . (a) Task identification accuracy (%) via prompt selection. The x-axis represents the 100 class IDs, sorted in ascending order of their task identification accuracy, which is shown on the y-axis as the proportion of correct prompt selection. (b) The average and standard variation of cosine similarities computed from the 7-th layer. The x-axis represents the task ID of input sample and the y-axis indicates the average cosine similarity between each task’s samples and their assigned keys. Ideally, all value should be close to 1, but most are near 0. Note that consistent results are observed across other layers as well.
 # of PromptsSelectionParamsAaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)
(a)10Similarity-based560K79.89 ±\pm 2.1482.12 ±\pm 1.66
(b)10Random553K79.78 ±\pm 2.0381.85 ±\pm 1.78
(c)1N/A138K81.05 ±\pm 2.1882.38 ±\pm 1.95
Table 1: Ablation study on the prompt selection. Note that (a) is same as MISA [ 10 ] without pretrained prompts.
  CIFAR100Tiny-ImageNetImageNet-R
BufferMethodAaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)AaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)AaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)
0Finetuning19.71 ±\pm 3.3910.42 ±\pm 4.9215.50 ±\pm 0.7410.42 ±\pm 4.927.51 ±\pm 3.942.29 ±\pm 0.85
Linear Probe49.69 ±\pm 6.0923.07 ±\pm 7.3342.15 ±\pm 2.7921.97 ±\pm 6.4329.24 ±\pm 1.2616.87 ±\pm 3.14
EWC [ 11 ]49.51 ±\pm 0.5252.83 ±\pm 2.3151.70 ±\pm 2.8931.04 ±\pm 3.1231.58 ±\pm 1.0420.72 ±\pm 1.11
LwF [ 17 ]55.51 ±\pm 3.4936.53 ±\pm 10.9649.00 ±\pm 1.5227.47 ±\pm 7.5931.61 ±\pm 1.5320.62 ±\pm 3.67
L2P [ 22 ]57.08 ±\pm 4.4341.63 ±\pm 12.7352.09 ±\pm 1.9235.05 ±\pm 5.7329.65 ±\pm 1.6319.55 ±\pm 4.78
Table 2: Accuracy (%) on CIFAR100 [ 13 ] , Tiny-Imagenet [ 14 ] , Imagenet-R [ 7 ] . We average the performance over 5 random seeds ( Mean±Std\text{Mean}\pm\text{Std} ) for ours. Others reported numbers borrow from MISA [ 10 ] . Bold indicates the best performance, while underlined entries denote the second-best results.
Figure 4: Visualization of the L2 norms of the weights for each class of the linear classifier, with blurry classes excluded. The x-axis shows class IDs in the order the model observes them.
PEFT AdapterLogit typeBatch MaskingK#Params\#\text{Params}AaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)FlastF_{\text{last}} (↓)
Prompt tuningcosine similarity✓N/A92K81.68 ±\pm 1.3484.37 ±\pm 0.385.68 ±\pm 0.92
LoRAcosine similarity✓N/A138K82.21 ±\pm 1.7481.25 ±\pm 1.6411.45 ±\pm 0.76
AdaptMLPcosine similarity✓N/A203K83.74 ±\pm 1.9784.34 ±\pm 0.429.07 ±\pm 0.66
Prefix tuningcosine similarity✓5230K85.58 ±\pm 1.4687.53 ±\pm 0.455.89 ±\pm 0.99
Prefix tuninglinear classifier✓5230K81.63 ±\pm 1.9882.37 ±\pm 1.7510.82 ±\pm 1.57
Prefix tuningcosine similarity✓5230K85.58 ±\pm 1.4687.53 ±\pm 0.455.89 ±\pm 0.99
Table 3: Ablation studies on different PEFT Adapters, logit types, batch masking and K values conducted on CIFAR100 [ 13 ] . The prompt length is fixed at 20 in all experiments involving prompts. We average the performance over 5 random seeds ( Mean±Std\text{Mean}\pm\text{Std} ). Bold indicates the best result within each ablation setting.
Disjoint class ratio (%)MethodAaucA_{\text{auc}} (↑)AlastA_{\text{last}} (↑)
0MISA [ 10 ]77.91 ±\pm 1.6382.26 ±\pm 0.92
SinglePrompt83.26 ±\pm 0.8988.15 ±\pm 0.18
25MISA [ 10 ]79.13 ±\pm 2.5581.36 ±\pm 1.01
SinglePrompt84.65 ±\pm 2.2987.67 ±\pm 0.52
50MISA [ 10 ]80.55 ±\pm 2.1780.98 ±\pm 1.08
SinglePrompt85.58 ±\pm 1.4687.53 ±\pm 0.45
Table 4: Comparison study on disjoint class ratios conducted on CIFAR100 [ 13 ] . We average the performance over 5 random seeds ( Mean±Std\text{Mean}\pm\text{Std} ). Bold indicates the best result.
Figure 5: Ablation study on Prompt length. The x-axis denotes the value of MM .
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Anytime inference accuracy curves of SinglePrompt and MISA [ 10 ] on CIFAR100 [ 13 ] . Accuracy is measured every 1,000 training samples, and the curves visualize the mean and standard deviation over five random seeds.
Figure 7: Visualization of the Si-blurry scenario. The dataset for the kk -th task, Dk\mathcal{D}_{k} , consists of the disjoint dataset TkDT_{k}^{D} and the blurry dataset TkBT_{k}^{B} .
Figure 8: Prompt selection failures on additional datasets, showing that the observed failure patterns generalize beyond CIFAR-100.
Figure 9: Failure of ConvPrompt [ 20 ] selection in task-based continual learning on CIFAR100 [ 13 ] . The x-axis represents the task ID of input sample and the y-axis indicates the average cosine similarity between each task’s samples and their assigned keys. (a) Result in the offline continual learning setting, where class information of upcoming tasks is available. Task similarity is computed using class descriptors, which determines the number of prompts allocated to each task (22 prompts in total). (b) Result in the online continual learning setting, where future task classes are unknown. Therefore, the number of prompts assigned to each task is fixed to 5, resulting in 50 prompts in total. In both cases, the cosine similarity is expected to be high for all tasks but remains close to zero. Results are shown for the 7-th layer. Note that consistent results are observed across other layers as well.
Method#ParamsFLOPsTotal time (s)
MVP [ 18 ]562K562K34.21G34.21G2731.43
MISA [ 10 ]576K576K35.67G35.67G3405.24
Online-LoRA [ 23 ]524K524K17.64G17.64G1764.47
SinglePrompt230K230K17.61G17.61G1589.74
Appendix
Table 5: Computational statistics for ours on CIFAR100 [ 13 ] without buffer. We compare our method with MISA [ 10 ] , MVP [ 18 ] and online-LoRA [ 23 ] . FLOPs refer to forward pass computations only, and total time is averaged over five runs with different random seeds. #Params refers to the number of learnable parameters. All methods use the same ViT-B/16 architecture to ensure a fair comparison.
More in this topic

Task-Free Continual Learning

See all 19 papers

Explore similar work

CardsList
Aug 13, 2026

Information-Theoretic Decoupled Prompt Tuning for Continual Learning

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Yunfei Zhang, Wen Wen, Tieliang Gong +1

National Engineering Lab for Big Data Analytics, School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Continual Learning for VLMsContinual Learning for LLMsVLM Adaptation
AskPDF
Jul 6, 2026

Learning Probabilistic Prompt for Continual Learning

Continual learning aims to progressively learn from a sequence of tasks, each containing a disjoint subset of classes, while preserving previously learned knowledge. Prompt-based continual learning methods propose to learn a small set of parameters, i.e., prompts, by associating them with a query feature of an input image. These methods optimize the prompts, attempting to represent diverse patterns of images. However, we have observed that existing prompt-based methods suffer from a prompt collapse problem, that is, the prompts tend to be highly similar to each other, thereby failing to capture the diverse data distributions in continual learning scenarios. To address this issue, we propose in this paper a novel prompt-based continual learning framework that captures diverse patterns of images across a sequence of tasks. To this end, we model each prompt as a probabilistic distribution and construct a mixture of these distributions, from which we sample diverse prompts. This enables our model to effectively capture highly diverse image distributions in the continual learning process. We also present a distribution regularization loss to prevent abrupt changes in the prompt distributions throughout the training process. We show extensive experimental results for continual learning on standard benchmarks, including ImageNet-R, CIFAR-100, and CUB-200, demonstrating the effectiveness of our framework.

Hyekang Park, Sanghoon Lee, Geon Lee +2

1Yonsei University, Seoul, South Korea · 2Samsung Electronics, Suwon, South Korea

Class-Incremental LearningPrompt LearningContinual LearningVisual Prompt Learning
AskPDF
Jun 8, 2026

LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models

Online task-free continual learning (TFCL) requires intelligent agents to sequentially accumulate knowledge from an unbounded, non-stationary data stream under strict single-pass constraints and without any explicit task identifiers. Existing online TFCL paradigms primarily rely on parameter-efficient prompt tuning or dynamic structure expansion driven by training-coupled optimization dynamics, such as empirical loss fluctuations or evolving latent distances. As a result, these training-coupled solvers remain agnostic to the structural origins of distribution drift, mechanically enforcing a fixed strategy across fundamentally distinct streaming variations. To address this gap, we propose LargeMonitor, a framework that leverages large pretrained foundation models to autonomously orchestrate task-free continuous adaptation. Specifically, LargeMonitor introduces a decoupled detection module utilizing the frozen, stable representation space of large vision models (LVMs) to achieve robust, zero-shot drift detection without training-dependent interference or brittle threshold tuning. Upon a confirmed drift, the framework activates a context-aware diagnostic module driven by large multimodal models (LMMs) to interpret the precise semantic etiologies of the stream variation (e.g., novel class emergence vs. environmental domain shift). This dual-stage capability empowers the continuous learner to dynamically deploy adaptive and shift-specific optimization strategies. Extensive experiments across multiple TFCL settings and benchmarks demonstrate that LargeMonitor achieves precise, robust detection and diagnosis of complex data streams while consistently improving the performance of existing online TFCL algorithms.

Mingqi Yuan, Xiaoquan Sun, Shihao Luo +1

Agentic Intelligence Lab | HKU · Qicore Tech

Distribution ShiftConcept DriftContinual LearningOnline Continual LearningTask-Free Continual Learning
AskPDF