Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities

JinLi He; Liang Bai; Xian Yang

cs.LGFeb 24, 2026

Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities

Authors: JinLi He, Liang Bai, Xian Yang

Organizations: Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK

Abstract

Rehearsal is one of the key techniques for mitigating catastrophic forgetting and has been widely adopted in continual learning algorithms due to its simplicity and practicality. However, the theoretical understanding of how rehearsal scale influences learning dynamics remains limited. To address this gap, we formulate rehearsal-based continual learning as a multidimensional effectiveness-driven iterative optimization problem, providing a unified characterization across diverse performance metrics. Within this framework, we derive a closed-form analysis of adaptability, memorability, and generalization from the perspective of rehearsal scale. Our results uncover several intriguing and counterintuitive findings. First, rehearsal can impair model's adaptability, in sharp contrast to its traditionally recognized benefits. Second, increasing the rehearsal scale does not necessarily improve memory retention. When tasks are similar and noise levels are low, the memory error exhibits a diminishing lower bound. Finally, we validate these insights through numerical simulations and extended analyses on deep neural networks across multiple real-world datasets, revealing statistical patterns of rehearsal mechanisms in continual learning.

Figures & tables

Fig. 1 : An ideal continual learning system should strike a delicate balance among the adaptation of newly acquired knowledge, the memorization of previously learned knowledge, and the generalization of unseen data distributions across diverse scenarios.

Panel of Figure 2 — Fig. 2 : Adaptation error of rehearsal-based methods under different setups, where $T=8$ , $n=1000$ and $\|\boldsymbol{w}_{t}^{*}\|^{2}=1$ for all $t\in T$ . Subfigure settings: (a) $s=500$ ; (b) $\sigma=0.02$ ; (c) $s=500$ , $\sigma=0.02$ . Discrete points denote averages across runs.

Panel of Figure 3 — Fig. 3 : Memory error of rehearsal-based methods under different setups, where $T=8$ , $n=1000$ and $\|\boldsymbol{w}_{t}^{*}\|^{2}=1$ for all $t\in T$ . Subfigure settings: (a) $s=500$ ; (b) $\sigma=0.02$ ; (c) $s=500$ , $\sigma=0.02$ . Discrete points denote averages over simulation runs.

Panel of Figure 4 — Fig. 4 : Generalization error w.r.t. the number of model parameters or rehearsal samples, with $T=8$ , $n=1000$ and $\|\boldsymbol{w}_{t}^{*}\|^{2}=1$ . Subfigure settings: (a) $s=500$ ; (b) $\sigma=0.02$ ; (c) $s=500$ , $\sigma=0.02$ . Discrete points denote averages across runs.

Num of Tasks	T3	T6	T9	T12	T16	T18	T20
0%	$1.12\pm 0.03$	$0.96\pm 0.03$	$1.17\pm 0.02$	$0.89\pm 0.02$	$0.99\pm 0.02$	$1.04\pm 0.01$	$1.01\pm 0.01$
5%	$1.18\pm 0.02$	$1.02\pm 0.03$	$1.20\pm 0.03$	$0.91\pm 0.02$	$1.03\pm 0.00$	$1.08\pm 0.01$	$1.05\pm 0.02$
10%	$1.26\pm 0.04$	$1.06\pm 0.02$	$1.27\pm 0.04$	$0.94\pm 0.02$	$1.10\pm 0.02$	$1.13\pm 0.02$	$1.12\pm 0.01$
50%	$1.45\pm 0.03$	$1.22\pm 0.05$	$1.42\pm 0.08$	$1.09\pm 0.07$	$1.21\pm 0.04$	$1.31\pm 0.11$	$1.27\pm 0.02$

TABLE I : The adaptation errors on the Tiny-ImageNet dataset across 20 training tasks with varying buffer sizes.

Fig. 5 : Adaptation error on Tiny-ImageNet with increasing training classes, with the legend showing varying buffer sizes.

Panel of Figure 7 — Fig. 6 : Impact of rehearsal size on adaptation, memory, and generalization errors in deep neural networks trained on MNIST, CIFAR-10, and CIFAR-100. Subfigures (a)-(b) show how accumulated classes affect adaptation error under different buffer sizes; (c)-(d) illustrate the impact of inter-task relatedness on generalization; (e)-(f) show how rehearsal size influences memory error under varying similarity levels.

Training Classes	2	4	6	8	10
Average Adaptation Error	0.127 $\pm$ 0.004	0.398 $\pm$ 0.003	0.279 $\pm$ 0.001	0.105 $\pm$ 0.003	0.159 $\pm$ 0.002
Average Memory Error	–	1.544 $\pm$ 0.031	3.896 $\pm$ 0.054	4.238 $\pm$ 0.163	4.655 $\pm$ 0.065
Average Generalization Error	0.127 $\pm$ 0.004	1.035 $\pm$ 0.013	2.865 $\pm$ 0.038	3.406 $\pm$ 0.121	3.938 $\pm$ 0.053
Forgetting Ratio	–	7.300 $\pm$ 0.628	22.350 $\pm$ 0.536	27.233 $\pm$ 0.427	38.375 $\pm$ 0.135
Average Accuracy	95.050 $\pm$ 0.260	65.775 $\pm$ 0.335	39.200 $\pm$ 0.414	35.775 $\pm$ 0.179	32.060 $\pm$ 0.334

TABLE II : Comparison of three types of model errors with traditional metrics as training classes increase on CIFAR-10.

Methods	2000	4000	6000	8000	10000
EWC	86.20	84.60	83.17	81.91	80.11
LwF	73.35	73.86	72.92	73.90	74.76
iCaRL	84.59	82.98	81.59	80.89	80.19
DER	73.87	73.57	72.40	72.86	72.40
FOSTER	79.48	84.71	84.35	83.95	83.10
MEMO	68.10	66.78	66.01	65.56	65.16

TABLE III : Comparison of current accuracy across continual learning benchmarks as rehearsal increases on CIFAR-100.

Fig. 7 : The memory error as rehearsal size increases across different levels of classes interrelation on Tiny-ImageNet.

TABLE IV : The average error comparing different sampling strategies and deeper network architectures on CIFAR-10.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Information-Theoretic Generalization Bounds Replay-Based Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China
Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning
Feb 24, 2026JinLi He, Liang Bai, Xian YangContinual Learning Catastrophic Forgetting
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Sep 30, 2026Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3Continual Learning for LLMs Replay-Based Continual Learning
University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +3

Aug 12, 2026J/K move · Enter open · S save

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

Tieliang Gong, Zhongbo Zhang, Wen Wen, Yong-Jin Liu

Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China

Information-Theoretic Generalization Bounds Replay-Based Continual Learning Representation Drift Catastrophic Forgetting Generalization Bounds

Open paper AskPDF arXiv

Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting. Yet its generalization behavior is shaped by two coupled effects that existing analyses fold into a single hypothesis-level quantity: finite memory replaces each past distribution with an empirical proxy, and repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. We develop a layer-wise information-theoretic framework that separates these effects at every depth. Our main result decomposes the expected generalization gap into a replay-induced representation drift and an optimization-dependence term, the latter further resolved into stability, plasticity, interaction, and residual-coupling components. Two refinements make the framework operational. A Wasserstein relaxation of the drift term, valid under support mismatch, yields a depth-dependent drift--sensitivity trade-off whose minimizer identifies which interior layer to stabilize. An SGLD instantiation of the optimization term reduces it to a trajectory-level log-determinant budget, exposing a curvature-aware gradient-alignment statistic that serves as an online diagnostic of task-wise forgetting. Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.

Abstract

Continual Learning

Explore similar work

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning

Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining