Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Guodong Zheng; Yan Sun; Peng Wang; Enneng Yang; Jiafei Lyu; Xiaochun Cao; Dacheng Tao; Li Shen

cs.LGOct 22, 2026

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Authors: Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen

Organizations: School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China. · College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University, Singapore. · Tencent Hunyuan, Shenzhen, China.

Abstract

Understanding catastrophic forgetting, in which learning new tasks degrades performance on previously learned tasks, remains a fundamental challenge in continual learning (CL). Existing theoretical analyses often rely on specific model or task structures and offer limited insight into how iterative optimization affects forgetting. To address this gap, we develop an algorithmic stability framework for expected population forgetting under SGD without prescribing a particular model architecture. Specifically, we decompose forgetting into intra-task and cross-task generalization gaps and empirical forgetting. To bound the generalization gaps, we track how a sample perturbation affects updates within a task and propagates through subsequent tasks. We consider constant, intra-task decaying, and continuously decaying learning-rate schedules. Comparing the two decaying schedules allows us to characterize how restarting learning-rate decay at task boundaries affects the propagation of historical sample perturbations. For empirical forgetting, we derive bounds for all three schedules by retaining cumulative gradient alignment along the optimization trajectory, without requiring convexity. Combining these results yields upper bounds on expected forgetting that capture the effects of training set size, the number of updates per task, the total number of tasks, learning-rate schedules, and cumulative gradient alignment. Finally, we conduct experiments on Split CIFAR-100 and Split MiniImageNet to examine the empirical implications of our theoretical findings.

Figures & tables

Case	LR	IGG $\epsilon_{i,i}$	CGG $\epsilon_{T,i}$	EF $\tilde{\epsilon}_{T,i}$
Convex	C	$\frac{2cL^{2}N}{n}$	$\frac{2cL^{2}N}{n}$	$\scriptstyle-\mathcal{G}_{T,i}+\frac{\beta c^{2}L^{2}}{2}N(T-i)$
	ID	$\frac{2cL^{2}\ln(eN)}{n}$	$\frac{2cL^{2}\ln(eN)}{n}$	$\scriptstyle-\mathcal{G}_{T,i}+\frac{\beta c^{2}L^{2}}{2}(2-1/N)(T-i)$
	CD	$\frac{2cL^{2}}{n}\ln\!\left(1+\frac{1}{i-1}\right)$	$\frac{2cL^{2}}{n}\ln\!\left(1+\frac{1}{i-1}\right)$	$\scriptstyle-\mathcal{G}_{T,i}+\frac{\beta c^{2}L^{2}}{2}(1/(iN)-1/(TN))$
Non-convex	ID	$\begin{gathered}\frac{(2cL^{2})^{1-p}N^{p}}{pn}-\frac{2L^{2}}{\beta n}\end{gathered}$	$e^{p(T-i)}\mathcal{K}_{N^{T-i+1}}-\frac{2L^{2}(eN)^{c\beta(T-i)}}{\beta n}$	$\scriptstyle-\mathcal{G}_{T,i}+\frac{\beta c^{2}L^{2}}{2}(2-1/N)(T-i)$
	CD a	$\mathcal{C}_{1}^{1}:\ \mathcal{K}_{iN}-\frac{(i-1)N}{n}-\frac{2L^{2}}{\beta n}$	$\mathcal{C}_{2}^{1}:\ \mathcal{K}_{TN}-\frac{(i-1)N}{n}-\frac{2L^{2}}{\beta n}(T/i)^{c\beta}$	$\scriptstyle-\mathcal{G}_{T,i}+\frac{\beta c^{2}L^{2}}{2}(1/(iN)-1/(TN))$
	CD a	$\mathcal{C}_{1}^{2}:\ \frac{2L^{2}}{\beta n}\bigl[(\frac{i}{i-1})^{c\beta}-1\bigr]$	$\mathcal{C}_{2}^{2}:\ \frac{2L^{2}}{\beta n}\bigl[(\frac{T}{i-1})^{c\beta}-(\frac{T}{i})^{c\beta}\bigr]$

TABLE I: Results for the IGG, CGG, and EF under convex and non-convex loss functions with three learning-rate (LR) schedules: constant (C), intra-task decaying (ID), and continuously decaying (CD). Here,

T

is the number of tasks and

i\in[T-1]

is the index of a previous task.

N

and

n

denote the number of iterations and training samples per task, respectively.

L

and

\beta

are the Lipschitz and smoothness constants. We define

p=\frac{c\beta}{c\beta+1}

, where

c

is the constant factor in the learning rate. The term

\mathcal{G}_{T,i}

is cumulative gradient alignment defined in Equation ( 6 ). For the non-convex ID setting, the IGG and CGG upper bounds reported in the table apply when

2cL^{2}<N

and

2cL^{2}(eN)^{c\beta(T-i)}<N

, respectively. When a condition is not satisfied, the corresponding upper bound is

N/n

. The full assumptions and detailed piecewise results are provided in Section IV .

Fig. 1 : EF and CGA under different learning-rate schedules on Split CIFAR-100. C, ID, and CD denote constant, intra-task decaying, and continuously decaying learning rates, respectively. Curves show the means over three seeds, and shaded regions indicate one standard deviation.

Fig. 2 : IGG, CGG, EF,

F_{T}

, and ACC under different learning rate schedules on Split CIFAR-100. The top row shows results for different initial learning rates

c

, while the bottom row shows results for different numbers of training epochs per task.

Fig. 3 : IGG and CGG with different numbers of training samples per task on Split CIFAR-100.

Fig. 4 : ACC and BWT for original CL, GEM, and A-GEM under different learning-rate schedules on Split CIFAR-100. Higher ACC indicates better predictive performance, while higher BWT indicates less forgetting.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 5 : IGG, CGG, EF,

F_{T}

, and ACC under different learning-rate schedules on the Split MiniImageNet benchmark. The top row shows results for different initial learning rates

c

, while the bottom row shows results for different numbers of training epochs per task.

Fig. 6 : EF and CGA under different learning-rate schedules on Split MiniImageNet. C, ID, and CD denote constant, intra-task decaying, and continuously decaying learning rates, respectively. Curves show the means over three seeds, and shaded regions indicate one standard deviation.

Fig. 7 : IGG and CGG with different numbers of training samples per task on Split MiniImageNet.

Fig. 8 : ACC and BWT for original CL, GEM, and A-GEM under different learning-rate schedules on Split MiniImageNet. Higher ACC indicates better predictive performance, while higher BWT indicates less forgetting.

Catastrophic Forgetting

Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.

See all 351 papers

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning
Oct 10, 2026Renjie Cao, Yifei Cheng, Li ShenOnline Continual Learning Catastrophic Forgetting
Boston College · Sun Yat-sen University
From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning
Apr 15, 2026Zonghuan Xu, Xingjun MaContinual Learning Catastrophic Forgetting
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Information-Theoretic Generalization Bounds Replay-Based Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China

Oct 10, 2026J/K move · Enter open · S save

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

Renjie Cao, Yifei Cheng, Li Shen

Boston College · Sun Yat-sen University

Online Continual Learning Catastrophic Forgetting Stochastic Gradient Descent

Open paper AskPDF arXiv

Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.

Abstract

Catastrophic Forgetting

Explore similar work

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning