Machine learning models often need to adapt to new data after deployment due to structured or unstructured real-world dynamics. The Continual Learning (CL) framework enables continuous model adaptation, but most existing approaches either assume each task contains sufficiently many data samples or that the learning tasks are non-overlapping. In this paper, we address the more general setting where each task may have a limited dataset, and tasks may overlap in an arbitrary manner without a priori knowledge. This general setting is substantially more challenging for two reasons. On the one hand, data scarcity necessitates effective contextualization of general knowledge and efficient knowledge transfer across tasks. On the other hand, unstructured task overlapping can easily result in negative knowledge transfer. To address the above challenges, we propose an adaptive mixture-of-experts (MoE) framework over pre-trained models that progressively establishes similarity awareness among tasks. Our design contains two innovative algorithmic components: incremental global pooling and instance-wise prompt masking. The former mitigates prompt association noise through gradual prompt introduction over time. The latter decomposes incoming task samples into those aligning with current prompts (in-distribution) and those requiring new prompts (out-of-distribution). Together, our design strategically leverages potential task overlaps while actively preventing negative mutual interference in the presence of per-task data scarcity. Experiments across varying data volumes and inter-task similarity show that our method enhances sample efficiency and is broadly applicable.
Figures & tables
Figure 1. As tasks of interest evolve over time, the learner may encounter similar or recurring concepts. For example, Task 1 and Task t+1 may both involve the high-level category of birds.
Figure 2. Comparison of Prompt Routing Strategies. (Left) When training on task t , typical methods select prompt models in one of two ways: either by querying an unstructured global pool or by rigidly training a prompt designated for task t . (Right) We employ an out-of-distribution (OOD) detector to dynamically determine which prompts may be queried. Inputs detected as novel (OOD) are routed to a task-specific prompt to learn new concepts, whereas for familiar samples, we query the set of previous task prompts to facilitate knowledge reuse.
Figure 3. Final Average Accuracy on ImageNet-R across varying training set sizes (expressed as a fraction of the full dataset). While global pooling enables effective transfer in scarce-data regimes, it suffers from interference as data availability increases. Incremental expansion mitigates this by reducing routing errors in early learning stage, simultaneously enabling the refinement of consolidated knowledge and the allocation of fresh capacity for novel knowledge.
ImageNet-R
5-Dataset
Learnable Key ( ( Wang et al., 2022c ; Wang et al., 2022b ) )
.602
.749
Task Centroids ( ( Wang et al., 2022a ) )
.633
.698
Task Classifier ( ( Le et al., 2025 ) )
.749
.791
Online RMD (Ours)
.877
.819
Table 1. Task Separability Analysis. Average AUC scores for discriminating between seen and unseen tasks. We compare our proposed method against popular query score mechanisms (Learnable Keys, Task Centroids, Task Classifiers) on the ImageNet-R and 5-Dataset benchmarks.
Figure 4. Impact of training data availability on routing method performance on ImageNet-R.
Method
5-Datasets
5-task
10-task
15-task
L2P
81.84
78.45
75.22
DualPrompt
77.91
74.33
71.15
S-Prompt++
86.06
82.71
79.58
CODA-Prompt
64.18
61.52
58.83
HiDe-Prompt
93.83
90.25
87.12
Table 2. Performance on 5-Datasets. We present FAA ( ↑ ) across different task scales (5, 10, and 15 tasks).
Method
DomainNet
5-task
10-task
15-task
L2P
69.58
66.82
64.15
DualPrompt
70.73
68.11
65.48
S-Prompt++
72.35
69.54
66.88
CODA-Prompt
73.24
70.62
68.03
HiDe-Prompt
78.92
75.68
72.85
Table 3. Performance on DomainNet. We present FAA ( ↑ ) across different task scales (5, 10, and 15 tasks).
Method
M-EMNIST
M-CifarCeleb
FAA
CA
FM
FAA
CA
FM
L2P
93.12
94.50
2.11
72.44
76.88
6.40
DualPrompt
95.88
96.92
1.85
78.10
81.55
5.11
CODA-Prompt
95.50
97.10
1.90
77.65
82.40
5.45
S-Prompt++
96.12
97.25
1.40
76.90
80.22
4.92
HiDe-Prompt
98.45
98.90
0.75
86.55
88.92
2.85
Table 4. Evaluation on task sequences with a mixture of similar and dissimilar tasks.
Method
Split CIFAR-100
Split ImageNet-R
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
L2P
83.06 ±0.17
88.25 ±0.01
6.58 ±0.40
63.65 ±0.12
67.25 ±0.02
7.51 ±0.17
DualPrompt
86.60 ±0.19
90.64 ±0.01
4.45 ±0.16
68.79 ±0.31
71.96 ±0.04
4.49 ±0.14
S-Prompt++
88.81 ±0.18
92.25 ±0.03
3.87 ±0.05
69.68 ±0.12
72.50 ±0.04
3.29 ±0.05
CODA-Prompt
86.94 ±0.63
91.57 ±0.75
4.04 ±0.18
70.03 ±0.47
74.26 ±0.24
5.17 ±0.22
HiDe-Prompt
92.61 ±0.28
94.03 ±0.01
3.16 ±0.10
75.06 ±0.12
76.60 ± 0.01
2.17 ±0.19
Table 5. Overall performance of continual learning. We present the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM) with ± standard deviation, over three runs of different random seeds and task splits.
Figure 5. Comparison of prompt allocation during training (top) and inference time (bottom) on ImageNet-R (10-Tasks).
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. SETA shares an overlapping parameter block only when the new task's update passes a subspace-agreement test and a sign test against its prior expert; shared knowledge is protected by adaptive elastic anchoring, and independently gated routers retrieve the relevant experts at inference without task identifiers. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, improving overall performance over the strongest baseline by 1.1 to 1.7 points on TRACE and by up to 5.5 points on the O-LoRA benchmark, with backward transfer between -2.2 and -1.3 on TRACE, against 4.6 to 6.3 points of forgetting for MoE-CL and 8 to 15 points for the other LoRA baselines.
Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1
Iowa State University, Ames, USA · Argonne National Laboratory, USA
Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
Yunxiang Fu, Meng Lou, Zicheng Liao +1
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.
Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1
Iowa State University, Ames, USA · Argonne National Laboratory, USA