Forgetting, plasticity, and co-observation: a third facet of continual learning

Timm Hess; Abhishek Jha; Gido M. van de Ven; Tinne Tuytelaars

cs.LGAug 19, 2026

Forgetting, plasticity, and co-observation: a third facet of continual learning

Authors: Timm Hess, Abhishek Jha, Gido M. van de Ven, Tinne Tuytelaars

Organizations: VIB.AI † KU Leuven, Belgium · Bernoulli Institute ‡ University of Groningen, Netherlands

Abstract

Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

Figures & tables

Figure 1: Intuitive illustration of the generalization properties of separate training in comparison to joint training. Finding a common structure across the presented information, here depicted as puzzle pieces, is easier when observing the information together, as in the joint training case (right). In contrast, we can think of separate training as placing information piece-by-piece while focusing on one piece of information at a time (left). Here, local fits emerge that struggle to map relationships across multiple pieces of information.

Panel of Figure 2 — Figure 2: Overview of training and evaluation for joint and separate training and expected empirical decomposition of the co-observation effect. (a) In the (incremental) joint training settings (top), the backbone $\theta_{B_{t}}$ is trained with simultaneous access to all data (accumulated up to stage $t$ ). The incremental joint model is continually updated from the previous stage’s initialization at each step, whereas the offline joint model is trained from scratch and depicted only at the final stage. In the separate training setting (bottom), the model observes only the current data chunk. To demonstrate the effect of co-observation, we eliminate forgetting without breaking the sequential visibility constraint using the ensemble baseline. The ensemble stores a frozen copy of the backbone after each task. During evaluation, representations from all $t$ checkpoints are concatenated, optionally projected back to the original dimension $d$ to control for representational capacity (dim. reduce), and evaluated via linear probing (LP). (b) Hypothesis of the expected results across four chunks. The performance difference between the naive sequential model and the ensemble quantifies forgetting , while the gap between the incremental joint model and the (offline) joint target highlights the loss of plasticity . The focus of this work lies on the remaining difference between the ensemble and the incremental joint model, which we attribute to the co-observation effect. (c) Summary of how the four baselines inherently account for the presence ( ✓ ) or absence ( ✗ ) of forgetting mitigation, data co-observation, and maintained plasticity.

Figure 3: Proof-of-principle MNIST experiment, demonstrating the co-observation effect. Four approaches are compared: naive sequential training, ensemble, incremental joint, and (offline) joint training. Task 1 is binary classification of odd vs. even digits on clean images. Task 2 is binary classification of small ( $\leq 4$ ) vs. large ( $\geq 5$ ) digits on images augmented with Gaussian noise. The evaluation task is odd vs. even classification on images that exhibit similar Gaussian noise as Task 2. Both joint training variants achieve higher LP-accuracy, demonstrating that noise features transfer to the “odd vs. even”-evaluation task when noisy data is being observed simultaneously, even when associated with a different task. This is contrasted by the ensemble, which preserves all learned representations but still achieves lower LP-accuracy. The gap between ensemble and joint training we attribute to the co-observation effect.

Panel of Figure 4 — Figure 4: The co-observation effect across supervised and self-supervised learning paradigms. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds (6 for I-JEPA). (a) Supervised learning (Cross-Entropy) on CIFAR-100 and ImageNet-100 (IN-100). (b) Self-supervised learning using Barlow Twins and I-JEPA on IN-100. In all settings, the beneficial effect of co-observation is clearly visible. The ensemble baseline prevents the forgetting seen in the Naive baseline, and consistently falls short of the incremental joint training. This residual difference between ensemble (separate training) and incremental joint highlights the impact of co-observation, demonstrating that simultaneously observing data is important for representation learning, regardless of the objective function.

Panel of Figure 5 — Figure 5: Interaction of continual learning mechanisms with the co-observation gap. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds for (a) Supervised Learning and (b) Self-Supervised Learning (Barlow Twins and I-JEPA). Comparing experience replay and knowledge distillation (LwF for SL; CaSSLe for Barlow) against the naive-, joint-incremental-, and ensemble-baselines, empirically demonstrates the relationship of both methods to mitigation of forgetting and obtained generalization from the representation learning perspective.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Empirical verification of zero forgetting in the diagnostic ensemble baseline. The linear probe accuracy is evaluated on the training sets of individual chunks (

T_{0}

through

T_{3}

) across 4 sequential training stages. After a chunk is actively trained on, its train-set accuracy does not decrease in subsequent stages.

Figure 7: Ablation of forward/backward transfer using distillation-enhanced checkpoints. Comparing the standard diagnostic ensemble (built from naive sequential checkpoints) against an ensemble built from distillation-enhanced checkpoints (LwF/CaSSLe) across sequential chunks. Even with enhanced transfer properties, the ensemble at best barely surpasses the standard baseline and fails to close the gap to joint-incremental training.

Panel of Figure 8 — Figure 8: The co-observation effect across extended sequences. Linear probing accuracy, reported as the mean and standard error over 3 seeds, for supervised learning on ImageNet-100 partitioned into $8$ chunks (left) and $12$ chunks (right). The performance gap between the diagnostic ensemble (perfect retention) and incremental joint (simultaneous observation) persists consistently across longer sequences.

Method	Chunk 1	Chunk 2	Chunk 3	Chunk 4
Naive	$60.80{\scriptstyle\pm 0.31}$	$63.54{\scriptstyle\pm 0.17}$	$63.92{\scriptstyle\pm 0.20}$	$64.64{\scriptstyle\pm 0.16}$
Ensemble	$60.78{\scriptstyle\pm 0.30}$	$65.93{\scriptstyle\pm 0.22}$	$68.19{\scriptstyle\pm 0.37}$	$70.39{\scriptstyle\pm 0.04}$
Ensemble PCA	$60.83{\scriptstyle\pm 0.37}$	$65.81{\scriptstyle\pm 0.12}$	$68.12{\scriptstyle\pm 0.31}$	$70.25{\scriptstyle\pm 0.06}$
Incremental Joint	$61.15{\scriptstyle\pm 0.16}$	${68.22}{\scriptstyle\pm 0.03}$	${72.45}{\scriptstyle\pm 0.03}$	${74.29}{\scriptstyle\pm 0.31}$
Joint (offline)	–	–	–	$75.18{\scriptstyle\pm 0.55}$

Appendix

Table 1: SL-CIFAR100 - LP-accuracy[ % ] CIFAR100. Runs reported as mean

\pm

standard error over

3

runs.

Method	Chunk 1	Chunk 2	Chunk 3	Chunk 4
Naive	$67.47{\scriptstyle\pm 0.38}$	$69.04{\scriptstyle\pm 0.46}$	$69.64{\scriptstyle\pm 0.14}$	$69.25{\scriptstyle\pm 0.06}$
Ensemble	$67.56{\scriptstyle\pm 0.29}$	$69.81{\scriptstyle\pm 0.29}$	$70.37{\scriptstyle\pm 0.43}$	$71.25{\scriptstyle\pm 0.19}$
Ensemble PCA	$67.57{\scriptstyle\pm 0.37}$	$70.20{\scriptstyle\pm 0.30}$	$71.43{\scriptstyle\pm 0.30}$	$71.53{\scriptstyle\pm 0.31}$
Incremental Joint	$67.19{\scriptstyle\pm 0.45}$	${71.61}{\scriptstyle\pm 0.36}$	${73.28}{\scriptstyle\pm 0.34}$	${73.50}{\scriptstyle\pm 0.41}$
Joint (offline)	–	–	–	$76.42{\scriptstyle\pm 0.40}$
Replay ( $m_{t}\!=\!1250$ )	$67.63{\scriptstyle\pm 0.42}$	$69.25{\scriptstyle\pm 0.24}$	$69.83{\scriptstyle\pm 0.14}$	$69.68{\scriptstyle\pm 0.16}$

Appendix

Table 3: Barlow-IN-100 - LP-accuracy[ % ] IN-100 Runs reported as mean

\pm

standard error over

3

runs.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Jun 7, 2026

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

One of the critical limitations of artificial neural networks is their lack of ability to continually learn: training on new tasks often leads to interference and forgetting of the previous ones. While several algorithms have been proposed to protect old memories from interference, they are typically applied during or immediately after each new episode of training. In contrast, humans and animals can learn continuously, acquiring multiple new memories during active learning before consolidating all of them into long-term storage. Here we show that multiple new tasks can be trained sequentially before an unsupervised sleep-like replay phase is applied to partially restore performance across all previously learned tasks. Our study further suggests that task-specific information remains resilient to new training but decays gradually as network is trained on new tasks. These findings point to novel principles for developing a broad range of continual learning AI solutions.

Anthony Bazhenov, Jean Erik Delanois, Giri P. Krishnan

Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA

Replay-Based Continual Learning Memory Consolidation Catastrophic Forgetting

Ask PDF

Aug 12, 2026

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting. Yet its generalization behavior is shaped by two coupled effects that existing analyses fold into a single hypothesis-level quantity: finite memory replaces each past distribution with an empirical proxy, and repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. We develop a layer-wise information-theoretic framework that separates these effects at every depth. Our main result decomposes the expected generalization gap into a replay-induced representation drift and an optimization-dependence term, the latter further resolved into stability, plasticity, interaction, and residual-coupling components. Two refinements make the framework operational. A Wasserstein relaxation of the drift term, valid under support mismatch, yields a depth-dependent drift--sensitivity trade-off whose minimizer identifies which interior layer to stabilize. An SGLD instantiation of the optimization term reduces it to a trajectory-level log-determinant budget, exposing a curvature-aware gradient-alignment statistic that serves as an online diagnostic of task-wise forgetting. Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.

Tieliang Gong, Zhongbo Zhang, Wen Wen +1

Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China

Information-Theoretic Generalization Bounds Replay-Based Continual Learning Representation Drift Catastrophic Forgetting Generalization Bounds

Ask PDF

Oct 7, 2026

Decentralized collaborative continual learning: A multi-objective minimization-based technique

In this work, we formulate decentralized continual learning within a multi-objective optimization framework. For a given inference task t (corresponding to a common minimizer shared by the cost functions of all agents), agents collecting data in a distributed and streamed manner are only allowed to perform local computations and to exchange information with neighboring agents over the underlying communication graph. As tasks evolve sequentially over time, agents must adapt to newly arriving tasks while retaining knowledge acquired from previously learned ones. This requirement leads to the wellknown stability plasticity dilemma, where stability refers to the ability to retain previous knowledge, while plasticity refers to the ability to learn and adapt to new tasks. To address the stability challenge, agents store subsets of samples from past tasks in local memory buffers. Then, through an appropriate multiobjective formulation, the stored information is incorporated into the learning process so that parameter updates account jointly for the current task and previously learned tasks. The proposed decentralized continual learning approach is analyzed in the mean square error sense under general assumptions on the individual cost functions and gradient noise processes. The analysis reveals that cooperation among agents improves the performance of continual learning. In particular, by exchanging information with neighboring agents, decentralized collaborative learning can exploit the diversity of locally observed data and memory buffers to improve the network average mean-square deviation (MSD) across tasks. Finally, simulations illustrate the theoretical findings and the effectiveness of the method in reducing forgetting and improving the average MSD across tasks.

Yara Zgheib, Marc Antonini, Roula Nassif

Universit´e Cˆote d’Azur, I3S Laboratory, CNRS, France

Decentralized Learning Replay-Based Continual Learning

Ask PDF

Abstract

Continual Learning

Explore similar work

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

Decentralized collaborative continual learning: A multi-objective minimization-based technique