The Sequential Price of Continual Learning

Zonghuan Xu; Xingjun Ma

cs.LGAug 30, 2026

The Sequential Price of Continual Learning

Authors: Zonghuan Xu, Xingjun Ma

Organizations: Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China

Abstract

Sequential task updates are fundamental to continual learning, but their recency bias can impose a lasting performance cost. We study this cost in an overparameterized linear-regression model with i.i.d. task sampling. We prove that distribution-level forgetting and population loss converge to the same stationary limit. We quantify the additional loss incurred by sequential exact fitting, or the sequential price. In more homogeneous task geometries, it equals the intrinsic loss asymptotically attained by joint training, making the total loss twice as large. We further analyze fixed-strength elastic weight consolidation (EWC) under general task curvatures and characterize its stationary sequential price at every regularization strength. Under strong regularization, the price decays inversely with EWC strength while the mean-square coupling horizon grows proportionally. Experiments on Jester and Rotated MNIST support the predicted sequential price and its reduction by EWC, with quantitative agreement on real-world tasks satisfying the theory's assumptions and qualitative agreement under nonlinear finite-step training.

Figures & tables

Figure 1: Sequential learning from Jester user preferences. Panel (a) compares the exact forgetting and population-loss curves. Panels (b)–(c) show the stationary population loss and the number of tasks required to reduce the initial-state effect to

1\%

under EWC. In these two panels, solid curves are exact finite-distribution calculations, dashed curves show the large-

\lambda

expressions in Theorem 5.2 , and the dotted line marks

R^{\star}

. Markers show averages over

2{,}000

independent task streams in panel (a) and

1{,}000

in panel (b). Error bars denote standard errors and are mostly smaller than the markers.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Additional diagnostics for sequential exact fitting on Jester. Panel (a) displays the intrinsic-loss, mean-bias, and stationary-variance components of

G_{\infty}

, together with an independent task-stream estimate of the total. Panel (b) compares the stationary mean

\mu_{j}

with the population minimizer

w_{j}^{\star}

for every joke; filled markers identify the five largest coordinate differences.

Figure 3: A nonlinear Rotated-MNIST stress test. Panel (a) compares forgetting and population loss under sequential training (

\lambda=0

). Curves are five-evaluation moving averages and shaded bands denote standard errors across five seeds. Panel (b) reports stationary population loss as a function of functional-EWC strength. The dotted line is the joint-training loss, and the shaded gap is the empirical sequential price. Panel (c) reports the number of tasks required to enter a

1\%

neighborhood of the stationary population loss. Error bars in panels (b)–(c) denote standard errors across seeds.

Explore similar work

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning
Apr 15, 2026Zonghuan Xu, Xingjun MaContinual Learning Learning-Augmented Algorithms
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
To Retain or to Adapt? Generalizing Continual Learning
Jul 6, 2026Giulia Lanzillotta, Mandana Samiei, Doina Precup +2Replay-Based Continual Learning Continual Learning
ETH AI Center, Department of Computer Science, ETH Zürich · Mila - Quebec AI Institute, School of Computer Science, McGill University · Mila - Quebec AI Institute +1
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Replay-Based Continual Learning Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China

Apr 15, 2026, cs.LGJ/K move · Enter open · S save

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

Zonghuan Xu, Xingjun Ma

Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China

Continual Learning Learning-Augmented Algorithms Forgetting Distributions Exact History

Open paper AskPDF arXiv

A central challenge in continual learning is forgetting: the loss of performance on previously learned tasks after learning new ones. Prior theory has analyzed forgetting under random orderings of fixed task collections in overparameterized linear regression. We shift the focus from task order to task distribution, asking how its structure determines forgetting. In the linear setting with a shared solution, i.i.d. task sampling, and sequential exact fitting, we derive an exact operator identity expressing historical forgetting directly in terms of the task distribution. Building on this identity, we establish an exponential decay guarantee for expected historical forgetting under every fixed task distribution in finite dimensions, characterize its asymptotic behavior, and relate decay to the distribution's coverage of observable directions. For an individual learned task, we show that subsequent tasks can collectively support recovery without exact revisits. We derive a lower bound on recovery time and construct a task distribution attaining its inverse-coverage scaling.

Abstract

Explore similar work

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

To Retain or to Adapt? Generalizing Continual Learning

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning