Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

Renjie Cao; Yifei Cheng; Li Shen

cs.LGOct 10, 2026

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

Authors: Renjie Cao, Yifei Cheng, Li Shen

Organizations: Boston College · Sun Yat-sen University

Abstract

Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.

Figures & tables

$d$	Order	$c_{1,j}$ , $q<1$	$g_{1,j}$ , $\rho>1$
1	$(C_{1},C_{1},C_{2},C_{3},C_{4},C_{5})$	$q^{4}$	$\rho^{5}-\rho+\rho^{4}$
2	$(C_{1},C_{2},C_{1},C_{3},C_{4},C_{5})$	$q^{3}$	$\rho^{5}-\rho^{2}+\rho^{3}$
3	$(C_{1},C_{2},C_{3},C_{1},C_{4},C_{5})$	$q^{2}$	$\rho^{5}-\rho^{3}+\rho^{2}$
4	$(C_{1},C_{2},C_{3},C_{4},C_{1},C_{5})$	$q$	$\rho^{5}-\rho^{4}+\rho$
5	$(C_{1},C_{2},C_{3},C_{4},C_{5},C_{1})$	$1$	$1$

Table 1: Coefficient of the two similar tasks (

C_{1}

) for

T=6

, up to the factors

q+2

and

\frac{4\rho}{\rho-1}

. For

q>1

c_{1,j}

takes the form of

g_{1,j}

with

\rho

replaced by

q

Panel of Figure 1 — Figure 1: Impact of data amount, task similarity, and task order. (a) MLP: forgetting increases with the amount of data. (b) MLP, with the target task learned last: forgetting increases with the number of non-overlapping classes, and a larger data amount leads to larger forgetting. (c) MLP: forgetting increases with the distance between the two similar tasks and with the amount of data. All results are averaged over 5 random seeds.

$n$	PMNIST		Split CIFAR-100
$n$	ACC(%)	BWT(%)	ACC(%)	BWT(%)
1500	72.52	2.98	44.43	-18.01
2500	68.71	-7.28	42.51	-24.79
5000	57.89	-15.56	39.46	-25.29

Table 2: Average final test accuracy (ACC) and backward transfer (BWT; the negative of forgetting, higher is better) over all tasks on PMNIST and Split-CIFAR-100, as the mean over 5 random seeds.

$\delta$	PMNIST		Split CIFAR-100
$\delta$	ACC(%)	BWT(%)	ACC(%)	BWT(%)
0%	75.13	0.64	17.71	-22.35
40%	71.03	-4.51	16.90	-23.68
100%	70.20	-5.46	14.32	-26.53

Table 3: ACC and BWT under different dissimilarities

\delta

between the target task and Task 1 (

n=2500

$d$	PMNIST		Split CIFAR-100
$d$	ACC(%)	BWT(%)	ACC(%)	BWT(%)
1	75.42	5.86	23.59	2.02
3	74.24	5.78	23.24	1.81
5	73.62	5.28	21.74	-0.49

Table 4: ACC and BWT under different distances

d

between the two similar tasks (

n=2000

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$\displaystyle T,\ \mathcal{T}$	Number of tasks and the task index set $\mathcal{T}=\{1,\dots,T\}$ .
$\displaystyle S_{t},\ \mathcal{D}_{t}$	Training set of task $t$ , with $n$ samples drawn i.i.d. from the distribution $\mathcal{D}_{t}$ .
$\displaystyle f_{t},\ f_{t,i}$	Finite-sum objective of task $t$ and its $i$ -th component, $f_{t}=\frac{1}{n}\sum_{i=1}^{n}f_{t,i}$ .
$\displaystyle w_{t}^{*}$	A fixed minimizer of $f_{t}$ .
$\displaystyle w_{t},\ w_{0}$	Parameter obtained after training on task $t$ ; initialization $w_{0}=0$ .
$\displaystyle\pi_{t}$	Random permutation used by Shuffle SGD on task $t$ .
$\displaystyle K_{i}(w),\ F_{t}$	Model error $\\|w-w_{i}^{*}\\|^{2}$ on task $i$ , and forgetting after learning task $t$ .

Appendix

Table 6

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 104 papers

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability
Oct 22, 2026Guodong Zheng, Yan Sun, Peng Wang +5Catastrophic Forgetting Generalization Bounds
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China. · College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University, Singapore. · Tencent Hunyuan, Shenzhen, China.
From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning
Apr 15, 2026Zonghuan Xu, Xingjun MaContinual Learning Catastrophic Forgetting
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Information-Theoretic Generalization Bounds Replay-Based Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China

Oct 22, 2026J/K move · Enter open · S save

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang+4

School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China. · College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University, Singapore. · Tencent Hunyuan, Shenzhen, China.

Catastrophic Forgetting Generalization Bounds Algorithmic Stability

Open paper AskPDF arXiv

Understanding catastrophic forgetting, in which learning new tasks degrades performance on previously learned tasks, remains a fundamental challenge in continual learning (CL). Existing theoretical analyses often rely on specific model or task structures and offer limited insight into how iterative optimization affects forgetting. To address this gap, we develop an algorithmic stability framework for expected population forgetting under SGD without prescribing a particular model architecture. Specifically, we decompose forgetting into intra-task and cross-task generalization gaps and empirical forgetting. To bound the generalization gaps, we track how a sample perturbation affects updates within a task and propagates through subsequent tasks. We consider constant, intra-task decaying, and continuously decaying learning-rate schedules. Comparing the two decaying schedules allows us to characterize how restarting learning-rate decay at task boundaries affects the propagation of historical sample perturbations. For empirical forgetting, we derive bounds for all three schedules by retaining cumulative gradient alignment along the optimization trajectory, without requiring convexity. Combining these results yields upper bounds on expected forgetting that capture the effects of training set size, the number of updates per task, the total number of tasks, learning-rate schedules, and cumulative gradient alignment. Finally, we conduct experiments on Split CIFAR-100 and Split MiniImageNet to examine the empirical implications of our theoretical findings.

Abstract

Online Continual Learning

Explore similar work

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning