Routing without Forgetting

Continual learning (CL) aims to incrementally acquire knowledge from sequential data while avoiding catastrophic forgetting. Recently, prompt tuning has attracted increasing attention as an efficient approach for adapting pre-trained models to CL tasks. However, existing prompt design paradigms commonly suffer from retrieval dependence and classifier bias, which make model adaptation sensitive to prompt selection and bias predictions toward newly arrived classes. To address these challenges, we propose Decoupled Prompt Tuning for Continual Learning (DPT4CL), which decouples the CLIP textual prompt into a task-shared prompt distribution and class-specific prompts. The task-shared prompt distribution is derived by optimizing an Information Bottleneck objective to facilitate cross-task knowledge transfer and alleviate classifier bias, while class-specific prompts enhance inter-class separability without relying on explicit prompt retrieval. Furthermore, we establish a unified excess risk bound from an information-theoretic perspective, providing theoretical support for the robust generalization and forgetting mitigation of the proposed framework. Extensive experiments on standard CL benchmarks demonstrate that DPT4CL achieves state-of-the-art performance. The source code is available at https://github.com/Cloudfly-Z/DPT4CL

Task-free online continual learning has recently emerged as a realistic paradigm for addressing continual learning in dynamic, real-world environments, where data arrive in a non-stationary stream without clear task boundaries and can only be observed once. To consider such challenging scenarios, many recent approaches have employed prompt selection, an adaptive strategy that selects prompts from a pool based on input signals. However, we observe that such selection strategies often fail to select appropriate prompts, yielding suboptimal results despite additional training of key parameters. Motivated by this observation, we propose a simple yet effective SinglePrompt that eliminates the need for prompt selection and focuses on classifier optimization. Specifically, we simply (i) inject a single prompt into each self-attention block, (ii) employ a cosine similarity-based logit design to alleviate the forgetting effect inherent in the classifier weights, and (iii) mask logits for unexposed classes in the current minibatch. With this simple task-free design, our framework achieves state-of-the-art performance across various online continual learning benchmarks. Source code is available at https://github.com/efficient-learning-lab/SinglePrompt.

Identifying and exploiting common features across domains is at the heart of the human ability to make analogies, and is believed to be crucial for the ability to continually learn. To do this successfully, general and flexible computational strategies must be developed. While the extent to which Transformer neural network models can perform compositional reasoning has been the subject of intensive recent investigation, little work has been done to systematically understand how well these models can leverage their representations to learn new, related experiences. To address this gap, we expand the previously developed Learning Equality and Group Operations (LEGO) framework to a continual learning (CL) setting ("continual LEGO"). Using this continual LEGO experimental paradigm, we study the capability of feedforward and recurrent Transformer models to perform CL. We find that BERT, a canonical feedforward Transformer model, learns shortcut solutions that limits its ability to generalize and prevents strong forward transfer to new experiences. In contrast, we find evidence supporting the hypothesis that ALBERT, a recurrent version of BERT, learns a For loop-esque solution, which leads to better CL performance. When applying BERT and ALBERT models to a CL setting that requires composition across experiences, we find that both model families fail. Our investigation suggests that ALBERT models can have their performance drop rescued by use of training strategies that combine data across experiences, but this is not true for BERT models, where a detrimental shortcut solution becomes entrenched with initial training. Our results demonstrate that the recurrent ALBERT model may have an inductive bias better suited for CL and motivate future investigation of the interplay between Transformer architecture and computational solutions that emerge in modern models and tasks.