Forgetting, plasticity, and co-observation: a third facet of continual learning

Timm Hess; Abhishek Jha; Gido M. van de Ven; Tinne Tuytelaars

cs.LGAug 19, 2026

Forgetting, plasticity, and co-observation: a third facet of continual learning

Authors: Timm Hess, Abhishek Jha, Gido M. van de Ven, Tinne Tuytelaars

Organizations: VIB.AI † KU Leuven, Belgium · Bernoulli Institute ‡ University of Groningen, Netherlands

Abstract

Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

Figures & tables

Figure 1: Intuitive illustration of the generalization properties of separate training in comparison to joint training. Finding a common structure across the presented information, here depicted as puzzle pieces, is easier when observing the information together, as in the joint training case (right). In contrast, we can think of separate training as placing information piece-by-piece while focusing on one piece of information at a time (left). Here, local fits emerge that struggle to map relationships across multiple pieces of information.

Panel of Figure 2 — Figure 2: Overview of training and evaluation for joint and separate training and expected empirical decomposition of the co-observation effect. (a) In the (incremental) joint training settings (top), the backbone $\theta_{B_{t}}$ is trained with simultaneous access to all data (accumulated up to stage $t$ ). The incremental joint model is continually updated from the previous stage’s initialization at each step, whereas the offline joint model is trained from scratch and depicted only at the final stage. In the separate training setting (bottom), the model observes only the current data chunk. To demonstrate the effect of co-observation, we eliminate forgetting without breaking the sequential visibility constraint using the ensemble baseline. The ensemble stores a frozen copy of the backbone after each task. During evaluation, representations from all $t$ checkpoints are concatenated, optionally projected back to the original dimension $d$ to control for representational capacity (dim. reduce), and evaluated via linear probing (LP). (b) Hypothesis of the expected results across four chunks. The performance difference between the naive sequential model and the ensemble quantifies forgetting , while the gap between the incremental joint model and the (offline) joint target highlights the loss of plasticity . The focus of this work lies on the remaining difference between the ensemble and the incremental joint model, which we attribute to the co-observation effect. (c) Summary of how the four baselines inherently account for the presence ( ✓ ) or absence ( ✗ ) of forgetting mitigation, data co-observation, and maintained plasticity.

Figure 3: Proof-of-principle MNIST experiment, demonstrating the co-observation effect. Four approaches are compared: naive sequential training, ensemble, incremental joint, and (offline) joint training. Task 1 is binary classification of odd vs. even digits on clean images. Task 2 is binary classification of small ( $\leq 4$ ) vs. large ( $\geq 5$ ) digits on images augmented with Gaussian noise. The evaluation task is odd vs. even classification on images that exhibit similar Gaussian noise as Task 2. Both joint training variants achieve higher LP-accuracy, demonstrating that noise features transfer to the “odd vs. even”-evaluation task when noisy data is being observed simultaneously, even when associated with a different task. This is contrasted by the ensemble, which preserves all learned representations but still achieves lower LP-accuracy. The gap between ensemble and joint training we attribute to the co-observation effect.

Panel of Figure 4 — Figure 4: The co-observation effect across supervised and self-supervised learning paradigms. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds (6 for I-JEPA). (a) Supervised learning (Cross-Entropy) on CIFAR-100 and ImageNet-100 (IN-100). (b) Self-supervised learning using Barlow Twins and I-JEPA on IN-100. In all settings, the beneficial effect of co-observation is clearly visible. The ensemble baseline prevents the forgetting seen in the Naive baseline, and consistently falls short of the incremental joint training. This residual difference between ensemble (separate training) and incremental joint highlights the impact of co-observation, demonstrating that simultaneously observing data is important for representation learning, regardless of the objective function.

Panel of Figure 5 — Figure 5: Interaction of continual learning mechanisms with the co-observation gap. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds for (a) Supervised Learning and (b) Self-Supervised Learning (Barlow Twins and I-JEPA). Comparing experience replay and knowledge distillation (LwF for SL; CaSSLe for Barlow) against the naive-, joint-incremental-, and ensemble-baselines, empirically demonstrates the relationship of both methods to mitigation of forgetting and obtained generalization from the representation learning perspective.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Empirical verification of zero forgetting in the diagnostic ensemble baseline. The linear probe accuracy is evaluated on the training sets of individual chunks (

T_{0}

through

T_{3}

) across 4 sequential training stages. After a chunk is actively trained on, its train-set accuracy does not decrease in subsequent stages.

Figure 7: Ablation of forward/backward transfer using distillation-enhanced checkpoints. Comparing the standard diagnostic ensemble (built from naive sequential checkpoints) against an ensemble built from distillation-enhanced checkpoints (LwF/CaSSLe) across sequential chunks. Even with enhanced transfer properties, the ensemble at best barely surpasses the standard baseline and fails to close the gap to joint-incremental training.

Panel of Figure 8 — Figure 8: The co-observation effect across extended sequences. Linear probing accuracy, reported as the mean and standard error over 3 seeds, for supervised learning on ImageNet-100 partitioned into $8$ chunks (left) and $12$ chunks (right). The performance gap between the diagnostic ensemble (perfect retention) and incremental joint (simultaneous observation) persists consistently across longer sequences.

Method	Chunk 1	Chunk 2	Chunk 3	Chunk 4
Naive	$60.80{\scriptstyle\pm 0.31}$	$63.54{\scriptstyle\pm 0.17}$	$63.92{\scriptstyle\pm 0.20}$	$64.64{\scriptstyle\pm 0.16}$
Ensemble	$60.78{\scriptstyle\pm 0.30}$	$65.93{\scriptstyle\pm 0.22}$	$68.19{\scriptstyle\pm 0.37}$	$70.39{\scriptstyle\pm 0.04}$
Ensemble PCA	$60.83{\scriptstyle\pm 0.37}$	$65.81{\scriptstyle\pm 0.12}$	$68.12{\scriptstyle\pm 0.31}$	$70.25{\scriptstyle\pm 0.06}$
Incremental Joint	$61.15{\scriptstyle\pm 0.16}$	${68.22}{\scriptstyle\pm 0.03}$	${72.45}{\scriptstyle\pm 0.03}$	${74.29}{\scriptstyle\pm 0.31}$
Joint (offline)	–	–	–	$75.18{\scriptstyle\pm 0.55}$

Appendix

Table 1: SL-CIFAR100 - LP-accuracy[ % ] CIFAR100. Runs reported as mean

\pm

standard error over

3

runs.

Method	Chunk 1	Chunk 2	Chunk 3	Chunk 4
Naive	$67.47{\scriptstyle\pm 0.38}$	$69.04{\scriptstyle\pm 0.46}$	$69.64{\scriptstyle\pm 0.14}$	$69.25{\scriptstyle\pm 0.06}$
Ensemble	$67.56{\scriptstyle\pm 0.29}$	$69.81{\scriptstyle\pm 0.29}$	$70.37{\scriptstyle\pm 0.43}$	$71.25{\scriptstyle\pm 0.19}$
Ensemble PCA	$67.57{\scriptstyle\pm 0.37}$	$70.20{\scriptstyle\pm 0.30}$	$71.43{\scriptstyle\pm 0.30}$	$71.53{\scriptstyle\pm 0.31}$
Incremental Joint	$67.19{\scriptstyle\pm 0.45}$	${71.61}{\scriptstyle\pm 0.36}$	${73.28}{\scriptstyle\pm 0.34}$	${73.50}{\scriptstyle\pm 0.41}$
Joint (offline)	–	–	–	$76.42{\scriptstyle\pm 0.40}$
Replay ( $m_{t}\!=\!1250$ )	$67.63{\scriptstyle\pm 0.42}$	$69.25{\scriptstyle\pm 0.24}$	$69.83{\scriptstyle\pm 0.14}$	$69.68{\scriptstyle\pm 0.16}$

Appendix

Table 3: Barlow-IN-100 - LP-accuracy[ % ] IN-100 Runs reported as mean

\pm

standard error over

3

runs.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 305 papers

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks
Jun 7, 2026Anthony Bazhenov, Jean Erik Delanois, Giri P. KrishnanReplay-Based Continual Learning Memory Consolidation
Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
Aug 12, 2026Tieliang Gong, Zhongbo Zhang, Wen Wen +1Information-Theoretic Generalization Bounds Replay-Based Continual Learning
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China
Decentralized collaborative continual learning: A multi-objective minimization-based technique
Oct 7, 2026Yara Zgheib, Marc Antonini, Roula NassifDecentralized Learning Replay-Based Continual Learning
Universit´e Cˆote d’Azur, I3S Laboratory, CNRS, France

Jun 7, 2026J/K move · Enter open · S save

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

Anthony Bazhenov, Jean Erik Delanois, Giri P. Krishnan

Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA

Replay-Based Continual Learning Memory Consolidation Catastrophic Forgetting

Open paper AskPDF arXiv

One of the critical limitations of artificial neural networks is their lack of ability to continually learn: training on new tasks often leads to interference and forgetting of the previous ones. While several algorithms have been proposed to protect old memories from interference, they are typically applied during or immediately after each new episode of training. In contrast, humans and animals can learn continuously, acquiring multiple new memories during active learning before consolidating all of them into long-term storage. Here we show that multiple new tasks can be trained sequentially before an unsupervised sleep-like replay phase is applied to partially restore performance across all previously learned tasks. Our study further suggests that task-specific information remains resilient to new training but decays gradually as network is trained on new tasks. These findings point to novel principles for developing a broad range of continual learning AI solutions.

Abstract

Continual Learning

Explore similar work

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

Decentralized collaborative continual learning: A multi-objective minimization-based technique