Memory-efficient Continual Learning with Prototypical Exemplar Condensation

Minh-Duong Nguyen; Thien-Thanh Dao; Le-Tuan Nguyen; Dung D. Le; Kok-Seng Wong

cs.LGMar 14, 2026

Memory-efficient Continual Learning with Prototypical Exemplar Condensation

Authors: Minh-Duong Nguyen, Thien-Thanh Dao, Le-Tuan Nguyen, Dung D. Le, Kok-Seng Wong

Organizations: College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center for AI Research, VinUniversity, Hanoi, Vietnam · Faculty of Information Systems, Phenikaa University, Hanoi, Vietnam · Center for Environmental Intelligence, VinUniversity, Hanoi, Vietnam

Abstract

Rehearsal-based continual learning (CL) mitigates catastrophic forgetting by maintaining a subset of samples from previous tasks for replay. Existing studies primarily focus on optimizing memory storage through coreset selection strategies. While these methods are effective, they typically require storing a substantial number of samples per class (SPC), often exceeding 20, to maintain satisfactory performance. In this work, we propose to further compress the memory footprint by synthesizing and storing prototypical exemplars, which can form representative prototypes when passed through the feature extractor. Owing to their representative nature, these exemplars enable the model to retain previous knowledge using only a small number of samples while preserving privacy. Moreover, we introduce a perturbation-based augmentation mechanism that generates synthetic variants of previous data during training, thereby enhancing CL performance. Extensive evaluations on widely used benchmark datasets and settings demonstrate that the proposed algorithm achieves superior performance compared to existing baselines, particularly in scenarios involving large-scale datasets and a high number of tasks.

Figures & tables

Figure 1: Illustration of ProtoCore architecture.

Figure 2: Last accuracy on tasks observed so far in the test set of S-CIFAR-100 (10, 50 tasks), S-TinyImageNet (20 tasks), and S-ImageNet-1K (100 tasks).

Method	S-CIFAR-100 (T=10)	S-TinyImageNet (T=20)
Method	Acc $\uparrow$	Acc $\uparrow$
Prototypical	29.51 $\pm$ 0.64	15.67 $\pm$ 0.44
Contrastive	31.17 $\pm$ 0.83	17.31 $\pm$ 0.37

Table 2: Ablation studies of using different losses for

{\mathcal{L}}_{\text{pro\_cur}}

and

{\mathcal{L}}_{\text{pro\_pre}}

. The contrastive loss is our proposed baselines, while prototypical loss is the loss using by [ 38 ] .

Method	CIFAR-10	CIFAR-100
Method	Acc $\uparrow$ (Forget $\downarrow$ )	Acc $\uparrow$ (Forget $\downarrow$ )
(1)	43.12 $\pm$ 0.65(12.5 $\pm$ 0.1)	22.43 $\pm$ 0.40(18.2 $\pm$ 0.5)
(1) + (2)	45.45 $\pm$ 0.83(11.8 $\pm$ 0.3)	25.10 $\pm$ 0.18(16.7 $\pm$ 0.4)
(1) + (2) + (3)	47.60 $\pm$ 0.10(10.9 $\pm$ 0.3)	27.85 $\pm$ 0.86(15.5 $\pm$ 0.8)
(4)	44.20 $\pm$ 0.32(12.2 $\pm$ 0.6)	23.50 $\pm$ 0.39(17.8 $\pm$ 0.4)
(4) + (5)	46.10 $\pm$ 0.41(11.4 $\pm$ 0.3)	26.30 $\pm$ 0.77(16.2 $\pm$ 0.9)

Table 3: Ablation studies on CIFAR-10 and CIFAR-100. ProtoCore denotes the algorithms with all losses. We denote (1) as

{\mathcal{L}}_{\text{cur\_syn}}

, (2) as

{\mathcal{L}}_{\text{pre\_syn}}

, and (3) as

{\mathcal{L}}_{\text{shift}}

for synthetic exemplar generation. We denote (4) as

{\mathcal{L}}_{\text{pro\_cur}}

, (5) as

{\mathcal{L}}_{\text{pro\_pre}}

, and (6) as

{\mathcal{L}}_{\text{task\_cur}}

, (7) as

{\mathcal{L}}_{\text{task\_pre}}

Table 4: Last accuracy of ProtoCore on S-CIFAR-100 with

T{=}10

and

T{=}50

across different replay memory sizes.

Panel of Figure 3 — Figure 3: t-SNE visualizations of synthetic features generated by ProtoCore on the CIFAR-100 test set. The synthetic data generation process converges within less than 40 epochs, yielding exemplars that closely approximate the real data distribution. $\bigstar$ represents the real prototypes, $\bullet$ represents the embeddings generated by the synthetic exemplars.

Method	Comp. Time	GPU Mem.	Max HDD Mem.
	(Hours)	(GB)	(MB)
CSReL	$5.97$	$1.23$	$6.20$
BCSR	$8.16$	$3.80$	$6.20$
OCS	$7.72$	$3.42$	$6.20$
PBCS	$9.79$	$2.22$	$6.20$
OnPro	$8.35$	$1.75$	$6.20$

Table 5: Comparison on CIFAR-100 in terms of computation (Hours), GPU memory (GB) and memory usage (MB).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$T$	Total number of tasks.
$\|{\mathcal{T}}_{t}\|$	Number of samples per task $t$ .
${\mathcal{T}}_{t}$	Dataset of task $t$ with $\|{\mathcal{T}}_{t}\|$ samples.
${\mathcal{C}}_{t}$	Set of available class for task $t$ .
${\mathcal{C}}$	Set of available class.
$y_{i}^{t}$	Class label of sample $x_{i}^{t}$ .

Appendix

Table 6: Notations used in the proposed algorithms.

Type	Hyperparameters	S-CIFAR-100 [ 15 ]	S-CIFAR-100 [ 15 ]	S-TinyImageNet [ 25 ]	S-ImageNet-1K [ 15 ]
Proto Network	Optimizer	Adam	Adam	Adam	Adam
	LR	$0.05$	$0.05$	$0.05$	$0.05$
	Temperature	$0.1$	$0.1$	$0.1$	$0.1$
	Alignment Coeff.	$0.1$	$0.1$	$0.1$	$0.1$
	Proto Coeff.	$0.95$	$0.95$	$0.95$	$0.95$
Synthetic Generator	Iterations	$50$	$50$	$50$	$50$

Appendix

Table 7: Hyperparameter settings for different datasets.

Method	Comp. Time	GPU Mem.	HDD Mem.
	(Hours)	(GB)	(G)
CSReL	214.61	10.04	12.08
BCSR	287.84	24.89	12.10
OCS	313.44	27.55	12.94
PBCS	391.63	18.89	12.14
OnPro	316.21	11.70	12.92

Appendix

Table 9: Comparison on S-ImageNet-1K (100 tasks) in terms of computation (Hours), GPU and memory usage (GB).

Method	S-CIFAR-100 (T=10)	S-ImageNet-1K (T=100)
Method	Seconds/task	Seconds/task
PBCS	$269.28$	$11112.84$
CSReL	$151.60$	$4740.84$
BCSR	$218.16$	$7368.84$
OCS	$235.31$	$8304.84$
ProtoCore	$161.85$	$6108.84$

Appendix

Table 10: Analysis on time complexity of the synthetic example generation at the end of every task.

Replay-Based Continual Learning

Replay-based continual learning preserves prior-task knowledge by revisiting stored or generated experience while adapting models to new data.

See all 131 papers

Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning
Mar 24, 2026Connor Mclaughlin, Nigel Lee, Lili SuContinual Learning Data-Efficient Learning
Northeastern University, Boston, Massachusetts, USA · The Charles Stark Draper Laboratory, Inc., Cambridge, Massachusetts, USA
Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
Sep 25, 2025Seohyeon Cha, Huancheng Chen, Haris VikaloFederated Continual Learning Gradient Projection
Department of Electrical and Computer Engineering The University of Texas at Austin
Are Current Continual Learning Methods Truly Agnostic? Introducing OPRE, a Step Toward Agnostic Continual Learning
Nov 11, 2025Raphaël Bayle, Martial Mermillod, Robert M. FrenchContinual Learning Coreset Selection
Univ. Grenoble Alpes, Univ. Savoie Mont Blanc, CNRS, LPNC, Grenoble, France · LEAD-CNRS UMR 5022, U. de Bourgogne Europe, Dijon, France

Mar 24, 2026J/K move · Enter open · S save

Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning

Connor Mclaughlin, Nigel Lee, Lili Su

Northeastern University, Boston, Massachusetts, USA · The Charles Stark Draper Laboratory, Inc., Cambridge, Massachusetts, USA

Continual Learning Data-Efficient Learning Mixture-of-Experts Models

Open paper AskPDF arXiv

Machine learning models often need to adapt to new data after deployment due to structured or unstructured real-world dynamics. The Continual Learning (CL) framework enables continuous model adaptation, but most existing approaches either assume each task contains sufficiently many data samples or that the learning tasks are non-overlapping. In this paper, we address the more general setting where each task may have a limited dataset, and tasks may overlap in an arbitrary manner without a priori knowledge. This general setting is substantially more challenging for two reasons. On the one hand, data scarcity necessitates effective contextualization of general knowledge and efficient knowledge transfer across tasks. On the other hand, unstructured task overlapping can easily result in negative knowledge transfer. To address the above challenges, we propose an adaptive mixture-of-experts (MoE) framework over pre-trained models that progressively establishes similarity awareness among tasks. Our design contains two innovative algorithmic components: incremental global pooling and instance-wise prompt masking. The former mitigates prompt association noise through gradual prompt introduction over time. The latter decomposes incoming task samples into those aligning with current prompts (in-distribution) and those requiring new prompts (out-of-distribution). Together, our design strategically leverages potential task overlaps while actively preventing negative mutual interference in the presence of per-task data scarcity. Experiments across varying data volumes and inter-task similarity show that our method enhances sample efficiency and is broadly applicable.

Abstract

Replay-Based Continual Learning

Explore similar work

Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning

Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection

Are Current Continual Learning Methods Truly Agnostic? Introducing OPRE, a Step Toward Agnostic Continual Learning