CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

Arundhati S. Shanbhag; Brian B. Moser; Federico Raue; Tobias C. Nauen; Andreas Dengel

cs.LGOct 23, 2026

CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

Authors: Arundhati S. Shanbhag, Brian B. Moser, Federico Raue, Tobias C. Nauen, Andreas Dengel

Organizations: University of Kaiserslautern-Landau, Germany · German Research Center for Artificial Intelligence, Germany

Abstract

Replay-based Continual Learning mitigates catastrophic forgetting by keeping a buffer of past samples, making buffer composition critical under a fixed memory budget. Recent coreset-based replay methods improve buffer quality using model-derived gradients, losses, or auxiliary optimization, but incur substantial computational overhead as memory is repeatedly updated. We propose CARES, a continual model-optimization-free approach for buffer construction that instead exploits the geometry of a frozen pretrained feature space using class-aware, density-weighted coverage. We evaluate CARES across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings. On Tiny-ImageNet, CARES is 2.3 to 8.8 times faster than existing coreset-based selectors. Although CARES achieves this speedup using no training signal from the continual learner, it ranks among the state of the art across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings, outperforming the strongest baseline by up to 7.58 percentage points under class imbalance while also attaining higher average accuracy on Tiny-ImageNet. Code will be released upon publication

Figures & tables

Method	Balanced		Imbalanced		Label Noise
Method	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$
K-means Embedding	$59.77{\pm}0.24$	$0.061{\pm}0.001$	$43.91{\pm}0.15$	$0.044{\pm}0.001$	$57.92{\pm}1.25$	$0.091{\pm}0.016$
Uniform	$58.99{\pm}0.54$	$0.074{\pm}0.004$	$44.73{\pm}0.11$	$0.033{\pm}0.007$	$58.76{\pm}1.07$	$0.087{\pm}0.006$
iCaRL	$60.74{\pm}0.09$	$\mathbf{0.044{\pm}0.026}$	$44.25{\pm}2.04$	$0.042{\pm}0.019$	$59.70{\pm}0.70$	$0.071{\pm}0.010$
Greedy Coreset	$59.39{\pm}0.16$	$0.066{\pm}0.017$	$43.80{\pm}0.01$	$0.039{\pm}0.007$	$58.22{\pm}0.16$	$0.066{\pm}0.001$
GCR	$58.73{\pm}0.43$	$0.073{\pm}0.013$	$44.48{\pm}0.05$	$0.035{\pm}0.005$	$58.72{\pm}0.63$	$0.081{\pm}0.005$

Table 1: Results on Split-CIFAR-100. CARES matches the strongest baseline on the balanced stream and achieves the highest final average accuracy under imbalance and label noise settings.

Method	Balanced		Imbalanced		Label Noise
Method	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$
K-means Embedding	$41.48{\pm}1.21$	$0.129{\pm}0.007$	$36.29{\pm}0.23$	$0.085{\pm}0.003$	$36.01{\pm}1.51$	$0.083{\pm}0.005$
Uniform	$42.11{\pm}0.52$	$0.129{\pm}0.002$	$37.07{\pm}0.53$	$0.083{\pm}0.009$	$37.14{\pm}1.05$	$0.099{\pm}0.003$
iCaRL	$43.84{\pm}0.09$	$0.114{\pm}0.004$	$37.65{\pm}0.84$	$0.058{\pm}0.003$	$38.52{\pm}0.25$	$0.063{\pm}0.006$
Greedy Coreset	$41.02{\pm}0.33$	$0.119{\pm}0.017$	$33.43{\pm}0.86$	$0.103{\pm}0.002$	$36.37{\pm}0.16$	$0.079{\pm}0.006$
GCR	$41.45{\pm}0.35$	$0.125{\pm}0.008$	$36.08{\pm}0.62$	$0.072{\pm}0.017$	$37.46{\pm}0.40$	$0.115{\pm}0.011$

Table 2: Experimental results on Split-Tiny-ImageNet. CARES achieves the highest average accuracy (

A_{T}

) across all three stream conditions, with particularly large gains under label noise.

Figure 1: Accuracy–runtime trade-off on Split-Tiny-ImageNet under the balanced setting in Protocol 1 at

K=200

. CARES achieves substantially higher final accuracy at a runtime comparable to lightweight selection methods.

Method	Split-CIFAR-10		Split-CIFAR-100		Split-TinyImageNet
Method	$K=200$	$K=2000$	$K=200$	$K=2000$	$K=200$	$K=2000$
ER	$32.63{\pm}5.11$	$49.31{\pm}7.45$	$11.72{\pm}1.25$	$20.78{\pm}1.48$	$4.36{\pm}1.02$	$11.30{\pm}1.45$
GEM	$16.81{\pm}1.17$	$20.35{\pm}2.44$	$10.42{\pm}1.66$	$12.38{\pm}3.81$	$5.15{\pm}0.71$	$9.56{\pm}2.63$
GSS	$33.83{\pm}3.95$	$41.70{\pm}4.56$	$12.31{\pm}0.57$	$16.24{\pm}0.80$	$5.82{\pm}0.38$	$8.44{\pm}0.48$
iCaRL	$38.66{\pm}2.03$	$39.56{\pm}1.53$	$11.23{\pm}0.31$	$12.28{\pm}0.39$	$4.39{\pm}0.19$	$6.66{\pm}0.30$
MIR	$21.50{\pm}0.70$	$46.40{\pm}1.81$	$10.30{\pm}0.30$	$16.02{\pm}0.50$	$4.90{\pm}0.17$	$7.05{\pm}0.30$

Table 3: Online CIL average accuracy (15 seeds) under small (

K=200

) and large (

K=2000

) replay budgets. CARES is competitive under tight memory and becomes increasingly effective as capacity grows, achieving the highest mean accuracy on Split-CIFAR-10 and Split-CIFAR-100 at

K=2000

$\gamma$	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
$A_{T}$	59.8	59.0	61.3	59.5	60.0	58.7	60.6	59.8	60.8	60.1
$P/K$	1	2	3	4	5	6	7	8	9	10	20
$A_{T}$	59.8	60.2	59.4	60.7	58.7	59.7	57.3	58.5	58.3	58.8	60.7

Table 4: Hyperparameter sensitivity of CARES on balanced Split CIFAR-100. Final task-incremental accuracy (%) when varying the coverage parameter

\gamma

and the staging-pool size

P

, expressed as a multiple of the replay capacity

K

. All other hyperparameters are held fixed.

Selection encoder	$K=200$	$K=500$	$K=2000$
InceptionV3	15.0	19.1	23.5
ResNet-50	14.4	19.6	23.7
CLIP ViT-B/32	14.3	19.1	23.3

Table 5: Sensitivity to the frozen encoder. Final class-incremental accuracy (%) of CARES on Split-CIFAR-100 under Protocol 2 with different pretrained encoders used for selection. CARES remains consistent across encoders, indicating limited sensitivity to the choice of representation used for coreset selection.

Variant	Class-aware	Coverage	Density	Balanced	Imbalanced	Label Noise
Uniform	$\times$	$\times$	$\times$	$58.99{\pm}0.54$	$44.73{\pm}0.11$	$58.76{\pm}1.07$
CB-Random	$\checkmark$	$\times$	$\times$	$59.50{\pm}1.63$	$52.89{\pm}0.41$	$59.37{\pm}0.72$
Global SubZeroCore	$\times$	$\checkmark$	$\checkmark$	$59.77{\pm}1.94$	$46.72{\pm}1.24$	$60.71{\pm}1.50$
Global Facility	$\times$	$\checkmark$	$\times$	$59.83{\pm}1.50$	$46.29{\pm}2.07$	$61.36{\pm}1.09$
Class-wise Facility	$\checkmark$	$\checkmark$	$\times$	$60.04{\pm}1.61$	$\mathbf{55.12{\pm}0.37}$	$61.37{\pm}1.41$
CARES	$\checkmark$	$\checkmark$	$\checkmark$	$\mathbf{61.52{\pm}1.48}$	$54.88{\pm}0.33$	$\mathbf{62.12{\pm}1.14}$

Table 6: Component ablation of CARES on Split-CIFAR-100. Class-aware allocation provides the dominant gain under class imbalance, substantially outperforming global geometric selection, while density weighting provides additional gains on balanced and label-noisy streams.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Balanced		Imbalanced		Label Noise
Method	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$	$\mathrm{A_{T}}\uparrow$	$\mathrm{FGT_{T}}\downarrow$
K-means Features	$57.82{\pm}0.69$	$0.070{\pm}0.003$	$45.44{\pm}0.76$	$0.037{\pm}0.002$	$57.38{\pm}1.26$	$0.098{\pm}0.003$
K-means Embedding	$59.77{\pm}0.24$	$0.061{\pm}0.001$	$43.91{\pm}0.15$	$0.044{\pm}0.001$	$57.92{\pm}1.25$	$0.091{\pm}0.016$
Uniform	$58.99{\pm}0.54$	$0.074{\pm}0.004$	$44.73{\pm}0.11$	$0.033{\pm}0.007$	$58.76{\pm}1.07$	$0.087{\pm}0.006$
iCaRL	$60.74{\pm}0.09$	$\mathbf{0.044{\pm}0.026}$	$44.25{\pm}2.04$	$0.042{\pm}0.019$	$59.70{\pm}0.70$	$0.071{\pm}0.010$
Grad Matching	$59.17{\pm}0.38$	$0.067{\pm}0.003$	$45.44{\pm}0.64$	$0.038{\pm}0.001$	$59.58{\pm}0.28$	$0.073{\pm}0.008$

Appendix

Table 7: Results on Split-CIFAR-100 under balanced, imbalanced, and label-noise settings. Higher

\mathrm{A_{T}}

is better, while lower

\mathrm{FGT_{T}}

is better. Results are reported as mean

\pm

standard deviation. Best results are shown in bold .

Method	Balanced		Imbalanced		Label Noise
Method	AVG ACC $\uparrow$	FGT $\downarrow$	AVG ACC $\uparrow$	FGT $\downarrow$	AVG ACC $\uparrow$	FGT $\downarrow$
K-means Embedding	$41.48{\pm}1.21$	$0.129{\pm}0.007$	$36.29{\pm}0.23$	$0.085{\pm}0.003$	$36.01{\pm}1.51$	$0.083{\pm}0.005$
Uniform	$42.11{\pm}0.52$	$0.129{\pm}0.002$	$37.07{\pm}0.53$	$0.083{\pm}0.009$	$37.14{\pm}1.05$	$0.099{\pm}0.003$
iCaRL	$43.84{\pm}0.09$	$0.114{\pm}0.004$	$37.65{\pm}0.84$	$0.058{\pm}0.003$	$38.52{\pm}0.25$	$0.063{\pm}0.006$
Greedy Coreset	$41.02{\pm}0.33$	$0.119{\pm}0.017$	$33.43{\pm}0.86$	$0.103{\pm}0.002$	$36.37{\pm}0.16$	$0.079{\pm}0.006$
GCR	$41.45{\pm}0.35$	$0.125{\pm}0.008$	$36.08{\pm}0.62$	$0.072{\pm}0.017$	$37.46{\pm}0.40$	$0.115{\pm}0.011$

Appendix

Table 8: Results on Split-Tiny-ImageNet under balanced, imbalanced, and label-noise settings. Higher average accuracy is better, while lower forgetting is better. Results are reported as mean

\pm

standard deviation.

Method	S-CIFAR-10			S-CIFAR-100			S-TinyImageNet
Method	$K=200$	$K=500$	$K=2000$	$K=200$	$K=500$	$K=2000$	$K=200$	$K=500$	$K=2000$
ER	$32.63{\pm}5.11$	$43.32{\pm}4.75$	$49.31{\pm}7.45$	$11.72{\pm}1.25$	$14.94{\pm}1.69$	$20.78{\pm}1.48$	$4.36{\pm}1.02$	$6.69{\pm}0.53$	$11.3{\pm}1.45$
GEM	$16.81{\pm}1.17$	$17.73{\pm}0.98$	$20.35{\pm}2.44$	$10.42{\pm}1.66$	$9.18{\pm}1.8$	$12.38{\pm}3.81$	$5.15{\pm}0.71$	$6.14{\pm}1.01$	$9.56{\pm}2.63$
GSS	$33.83{\pm}3.95$	$40.59{\pm}4.20$	$41.7{\pm}4.56$	$12.31{\pm}0.57$	$13.84{\pm}0.57$	$16.24{\pm}0.80$	$5.82{\pm}0.38$	$6.78{\pm}0.41$	$8.44{\pm}0.48$
iCaRL	$38.66{\pm}2.03$	$33.53{\pm}3.17$	$39.56{\pm}1.53$	$11.23{\pm}0.31$	$11.81{\pm}0.29$	$12.28{\pm}0.39$	$4.39{\pm}0.19$	$5.36{\pm}0.27$	$6.66{\pm}0.30$
MIR	$21.5{\pm}0.7$	$30.52{\pm}1.22$	$46.4{\pm}1.81$	$10.3{\pm}0.3$	$11.4{\pm}0.38$	$16.02{\pm}0.5$	$4.9{\pm}0.17$	$5.1{\pm}0.33$	$7.05{\pm}0.30$

Appendix

Table 9: Online CIL average accuracy (15 seeds). Best results are shown in bold.

Hyperparameter	Split CIFAR-100			Split Tiny-ImageNet
Hyperparameter	Bal.	Imb.	Noise	Bal.	Imb.	Noise
Number of tasks	20			20
Classes per task	5			10
Replay capacity $K$	100			200
Incoming stream batch size	50			100
Training batch size	10			20

Appendix

Table 10: Hyperparameter settings for Protocol 1. Settings shared across stream conditions are reported once per dataset; learning rate, weight decay, coverage, and staging-pool size are selected separately for the balanced, imbalanced, and label-noise settings.

Hyperparameter	Split CIFAR-10	Split CIFAR-100	Split Tiny-ImageNet
Number of tasks	5	5	10
Replay capacity $K$	{200, 500, 2000}	{200, 500, 2000}	{200, 500, 2000}
Training batch size	32	32	32
Incoming stream batch size	6250	6250	6250
Training epochs	1	1	1
Learning rate	0.05	0.05	0.05

Appendix

Table 11: Hyperparameter settings for Protocol 2.

$K$	$P$	CIL Acc. $\uparrow$	TIL Acc. $\uparrow$	Update Overhead (s) $\downarrow$
200	200	$9.83{\pm}0.42$	$63.60{\pm}0.70$	$6.8{\pm}1.0$
	500	$9.78{\pm}0.63$	$64.16{\pm}1.90$	$12.4{\pm}2.1$
	1000	$10.47{\pm}0.00$	$62.46{\pm}0.00$	$17.6{\pm}0.0$
500	200	$16.06{\pm}0.90$	$69.26{\pm}0.56$	$11.2{\pm}1.9$
	500	$16.24{\pm}1.33$	$69.37{\pm}1.02$	$15.1{\pm}7.3$
	1000	$16.72{\pm}1.50$	$70.12{\pm}0.97$	$23.5{\pm}13.4$

Appendix

Table 12: Scaling of CARES with replay capacity

K

and staging-pool capacity

P

on Split CIFAR-100 under Protocol 1. We report final class-incremental (CIL) and task-incremental (TIL) accuracy and the cumulative memory-update overhead. All other hyperparameters are held fixed. Results are reported as mean

\pm

standard deviation over

n

completed runs.

Figure 2: Joint sensitivity of CARES to the coverage parameter $\gamma$ and staging-pool capacity $P$ on balanced Split CIFAR-100. Each cell represents final task-incremental accuracy, with $P$ expressed relative to replay capacity $K=100$ . CARES exhibits a broad region of strong performance, while increasing the staging pool beyond moderate sizes provides no consistent benefit.

Replay-Based Continual Learning

Replay-based continual learning preserves prior-task knowledge by revisiting stored or generated experience while adapting models to new data.

See all 131 papers

Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers
Apr 9, 2026Danit Yanowsky, Daphna WeinshallExperience Replay Replay-Based Continual Learning
School of Computer Science and Engineering The Hebrew University of Jerusalem
CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental Learning Continual Learning
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
IDER: IDempotent Experience Replay for Reliable Continual Learning
Feb 28, 2026Zhanwang Liu, Yuting Li, Haoyuan Gao +4Continual Learning Catastrophic Forgetting
School of Computer Science, Shanghai Jiao Tong University · State Key Laboratory of General Artificial Intelligence, BIGAI · Lehigh University +1

Apr 9, 2026J/K move · Enter open · S save

Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers

Danit Yanowsky, Daphna Weinshall

School of Computer Science and Engineering The Hebrew University of Jerusalem

Experience Replay Replay-Based Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy for the replay buffer. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, Multiple Embedding Replay Selection, MERS, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over SOTA selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.

Abstract

Replay-Based Continual Learning

Explore similar work

Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

IDER: IDempotent Experience Replay for Reliable Continual Learning