cs.LGSep 28, 2026

Neural Succession: A Mesoscopic Theory of Invasion, Coexistence, and Stabilization in Continual Learning

Authors: Shoaib Ahmed Dipu, Md Salman Shamil, Sayeed Shafayet Chowdhury

Organizations: Indiana University Indianapolis · North South University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual learning is usually studied through mechanisms that preserve old knowledge. We develop Successional Learning Theory (SLT), a mesoscopic account in which the current representation is a resident community, the incoming task is an invader, forgetting is resident displacement, joint retention is coexistence, replay is resident reinforcement, and training moves from establishment toward stabilization. Its empirical coordinate is directional pre-invasion compatibility, measured on the resident model before the incoming task is learned. Across eight experiments, compatibility orders later forgetting on the 20 directed Split-CIFAR-10 transitions (three-repeat r=-0.789, incoming-task cluster 95% CI [-0.90,-0.72], every repeat alone r<=-0.67), forecasts held-out forgetting with 24% lower error than a no-information baseline, and reproduces under controlled MNIST permutations and CIFAR-10 rotations (r=-0.804, -0.718). On an 84-transition suite, compatibility separates coexistence from exclusion at every retention threshold (AUC 0.93-0.97). Replay repairs every transition with at most 325 stored examples and is most efficient where displacement is largest. Compatibility reaches |r|=0.720, while activation, representation, Jacobian, and fixed-coefficient Lotka-Volterra specializations do not. Plasticity and feature turnover fall reliably from early to late training (15/15 and 14/15 runs). We formalize a minimum habitat-modification bound, a displacement floor, a sufficient coexistence condition, an identifiability law with a range-restriction corollary, successional stabilization, and local reinforcement. The identifiability law also predicts where the coordinate loses leverage, and the prediction matches three CIFAR-100 partitions and five optimizer regimes. SLT is a pre-adaptation diagnostic that complements replay, regularization, and projection methods.

Figures & tables

Figure 1: The SLT pipeline. A resident representation is probed before the incoming task is learned. The resulting coordinate diagnoses the transition, which then unfolds as establishment followed by stabilization and ends in displacement or coexistence. Grey labels map each stage to the result that formalizes it and the experiment that tests it.
FamilyMain objectPreCoexistenceScale testStages
Transferability / SPOT / NTKtask relation, kernel✓\checkmark   
EWC / SI / MASparameter importance    
GEM / OGD / GPMgradient subspaces implicit  
Replay / DER++stored examples, logits    
Task orderingsequence relation✓\checkmark   
Plasticity studiestraining dynamics   ✓\checkmark
Table 1: Positioning by the question each family answers. “Pre”: computable before task- BB adaptation.
ResultObservable predictionTestsEvidenceStatus
Thm. 1 , Cor. 1forgetting rises with Ω\Omega ; Ω\Omega tracks demandE1, E2, E8r=−0.789r=-0.789 ; −0.804-0.804 / −0.718-0.718 ; Ω\Omega –probe loss 0.7670.767supported
Thm. 2Ω\Omega separates coexistence from exclusionE3AUC 1.001.00 (natural); 0.930.93 – 0.970.97 (84-suite)supported
Thm. 3 , Cor. 2∣r∣|r| falls with coordinate and outcome spreadscopeCIFAR-100: 0.38/0.18/−0.050.38/0.18/{-0.05} vs. predicted 0.32/0.30/0.130.32/0.30/0.13supported
Prop. 1early >> late plasticity and turnoverE74.784.78 [4.26,5.53][4.26,5.53] ; 0.370.37 [0.11,0.48][0.11,0.48]supported
Prop. 2replay repairs displacement; efficiency grows with displacementE420/20 repaired ( ≤325\leq 325 ex.); r(η,F0)=0.70r(\eta,F_{0})=0.70supported
Scale claimtransition-level ≫\gg feature-levelE6∣r∣|r| : 0.720.72 vs. ≤0.45\leq 0.45supported
Table 2: From theory to evidence. Each result is stated as an observable prediction and tested by a named experiment.
IDQuestionSettingUnitsPrimary statistic
E1Does ρ\rho order displacement?Split-CIFAR-10, ResNet-1820×320\times 3 repeatsr(ρ,F)r(\rho,F) on pair means
E2Is the order prospective?same; ρ\rho from resident only20, LOPOheld-out rr , MAE
E3Is coexistence a phase?natural; pure-domain; MNIST suite20; 60; 84rates; ROC AUC
E4Does replay reinforce?buffers 00 – 1000100020efficiency, threshold vs. ρ\rho
E5Does ρ\rho help an intervention?CIFAR-10/100, 6 methods3 seedsACC, BWT
E6At which scale?5 predictors; probe/LV tests20∣r∣|r| , pre-registered gates
Table 3: The eight SLT experiments. Units are directed transitions unless noted.
Figure 2: Compatibility is prospective. (a) Pair means over three independent repeats (bars: s.d.; band: OLS 95%). (b) Leave-one-pair-out forecasts (dashed: identity; band: ±\pm MAE). (c) The resident-only score agrees with a score that also uses the reverse direction.
Figure 3: Left: controlled manipulations of task relation (E8); black line and band: linear fit with 95% interval; red dashed: quadratic fit on MNIST. Right: identifiability (Theorem 3 ). (a) Predictive strength falls with the between-transition spread of ρ\rho along the range-restriction curve of Corollary 2 , anchored at E1. (b) Across optimizer regimes, the one regime with collapsed outcome variance is the one where ordering vanishes.
Figure 4: Coexistence is a phase of the transition. (a) Natural transitions: the three that coexist (green, F<0.20F<0.20 ) are the three most compatible; dotted line: compatibility split. (b) Pure-domain construction: only same-domain transitions reach near-zero forgetting. (c) 84-transition suite: coexistence AUC of ρ\rho and of a representation-compatibility score SS across mutual-retention thresholds (band: 95% cluster bootstrap over 28 class-pair ×\times transform clusters).
Figure 5: Scale selection. (a) On the same 20 transitions, the transition-level coordinate predicts displacement far better than feature- or step-level quantities. (b) Increasing Jacobian probes 16-fold at two layers does not produce a predictive microscopic coefficient.
Figure 6: Establishment precedes stabilization. (a, b) Median and interquartile range over 15 runs of per-epoch plasticity and turnover of the top-quartile resident features, while AA is established and then while BB invades. (c) Pre-specified contrasts with 95% intervals; grey: interval includes zero.
ClaimResultUncertainty / comparison
E1 natural orderingr=−0.789r=-0.789 , R2=0.622R^{2}=0.622CI [−0.90,−0.72][-0.90,-0.72] ; each repeat ≤−0.67\leq-0.67 ; single run −0.710-0.710
E2 held-out forecastMAE 7.627.62 pp, r=0.648r=0.648no-information MAE 10.0310.03 pp; LOTO r=0.669r=0.669
E8 controlled relationr=−0.804r=-0.804 / −0.718-0.718group-CV R2R^{2} 0.6260.626 / 0.4840.484
E3 coexistence phaseAUC 1.001.00 / 0.9270.927natural / 84-suite (CI [0.83,0.98][0.83,0.98] ); 8/24 vs. 0/36
E4 reinforcement20/20 repaired≤325\leq 325 examples (median 100); r(η,F0)=0.70r(\eta,F_{0})=0.70
E5 interventionACC 0.8150.815 / 0.5310.531above fine-tune, EWC, A-GEM on both benchmarks
Table 4: Main results with uncertainty. CI: 95% interval (cluster bootstrap unless stated).
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Threshold τ\tauCoexistence rateρpre\rho_{\mathrm{pre}} (SLT)Representation compatibility SSDifference
0.800.6670.936 [0.85,0.99][0.85,0.99]0.866 [0.74,0.96][0.74,0.96]+0.070+0.070 [0.001,0.169][0.001,0.169]
0.900.5710.927 [0.83,0.98][0.83,0.98]0.887 [0.76,0.97][0.76,0.97]+0.040+0.040 [−0.014,0.113][-0.014,0.113]
0.950.4760.966 [0.89,1.00][0.89,1.00]0.936 [0.81,1.00][0.81,1.00]+0.030+0.030 [−0.005,0.106][-0.005,0.106]
0.990.3100.969 [0.91,1.00][0.91,1.00]0.940 [0.85,0.99][0.85,0.99]+0.029+0.029 [−0.026,0.112][-0.026,0.112]
Appendix
Table 5: E3c: coexistence AUC on the 84-transition suite (95% cluster-bootstrap intervals).
Figure 7: Establishment bridge. Within incoming task, Ω\Omega tracks the incoming probe loss on Split-CIFAR-10 ( r=0.767r=0.767 , cluster 95% CI [0.28,0.95][0.28,0.95] ; clipped deficit HH : 0.6920.692 , [−0.14,0.94][-0.14,0.94] with five clusters) but not on semantic CIFAR-100 ( r=0.131r=0.131 ), where the coordinate has little spread.
Figure 8: Replay (E4). (a) The global minimum buffer is not ordered by ρ\rho . (b) Initial per-sample efficiency decreases with ρ\rho . (c) Efficiency rises with forgetting at zero replay, as Proposition 2 (ii) predicts; part of this relation is expected from headroom.
Figure 9: Interventions (E5). (a, b) Task-incremental accuracy (mean ±\pm s.d., three seeds). (c) Paired differences for the gated sign tests.
 Fine-tuneEWCA-GEMρ\rho -EWCρ\rho -EWC+replayDER++
Split-CIFAR-100.536 / −- 0.4800.526 / −- 0.4920.773 / −- 0.1840.569 / −- 0.4360.815 / −- 0.1110.851 / −- 0.068
Split-CIFAR-1000.229 / −- 0.5220.349 / −- 0.3300.460 / −- 0.2590.243 / −- 0.5010.531 / −- 0.1170.596 / −- 0.071
Appendix
Table 6: E5 benchmark (task-incremental, three seeds): ACC / BWT.
Figure 10: Robustness. (a) All 24 estimator settings. (b) Label budget: agreement with the full score and correlation with forgetting; red dotted: label-free k-means variant. (c) Optimizer regimes with task-cluster intervals (grey: interval includes zero). (d) Leave-one-task-out slopes on a pretrained ResNet-50.
TestCriterionResult
Jacobian competition, task levelr≥0.5r\geq 0.5 , perm. p≤0.05p\leq 0.05r=0.177r=0.177 , p=0.54p=0.54
Jacobian competition, feature levelmedian partial r>0.10r>0.10≤0.105\leq 0.105 at 1024 probes; ≤6.7%\leq 6.7\% significant
Fixed-coefficient LVbeats baselines (Wilson >0.5>0.5 )0/60; median held-out R2=−29.2R^{2}=-29.2
Locally stationary LVstable windows exist0/330 stable; unstable R2=−1.08R^{2}=-1.08
Neural R∗R^{\ast}resources identifiable1.9%; precision 0.0 vs. base 0.58
Niche/fitness (ND–FD) decompositionexplains forgettingR2=0.21R^{2}=0.21 (held-out 0.110.11 ), 45 pairs
Appendix
Table 7: Pre-registered microscopic tests (E6). All run in a regime with forgetting s.d. 0.0640.064 .
More in this topic

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Explore similar work

CardsList
Aug 23, 2026

Functional compatibility as a determinant of persistent neural learning

Neural networks can acquire new capabilities while damaging existing ones, but what determines whether new learning persists remains unclear. We identify functional compatibility, the extent to which incoming learning can coexist with behaviour that must be preserved, as an experimentally manipulable causal determinant of persistence. From identical neural states, we vary compatibility while matching unrestricted learning opportunity and imposing a common retention requirement. Persistent learning increases with compatibility across independent directions, convolutional and transformer architectures, vision and text, and a ten-seed replication. Learning rules and retention constraints determine how much compatible opportunity is retained, whereas nonlinear geometry limits the matched intervention at larger update norms. Functional compatibility therefore reframes stability-plasticity from preventing forgetting to determining which new learning can coexist with existing function and persist.

Hossein Javidnia

School of Computing Dublin City University Dublin, Ireland

Stability-Plasticity DilemmaContinual LearningNeural Network Training DynamicsCatastrophic Forgetting
AskPDF
Jul 10, 2026

Interference and Retention in Continual Learning

Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation. This work argues that forgetting should instead be modeled directly as interference between tasks. In the frozen-feature regime, forgetting from learning a new task is exactly the interference energy induced on the old task. In deep networks, the same quantity is recovered through path-averaged curvature with minimal additional forward passes. When task supports are disjoint, forgetting can be eliminated structurally and when task supports overlap in conflicting directions, a non-zero distortion floor is unavoidable. The same geometry optimally merges models through task-aware orthogonalization. From this analysis we derive Interference-Gated Functional Allocation (IGFA), a replay-free, Fisher-free method that shares directions when tasks align and protects them when they conflict. Across benchmarks, IGFA achieves lossless retention when tasks are structurally separable and moves unavoidable cost from irreversible forgetting into deferred but recoverable plasticity when they are not. It matches the strongest replay-free structural baselines on dissimilar-task streams and improves on unconditional projection when similarity makes transfer worth preserving.

Julius Störk

VARTA Microbattery GmbH, Ellwangen, Germany

Continual LearningOnline Continual LearningTask InterferenceCatastrophic Forgetting
AskPDF
Aug 19, 2026

Forgetting, plasticity, and co-observation: a third facet of continual learning

Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

Timm Hess, Abhishek Jha, Gido M. van de Ven +1

VIB.AI † KU Leuven, Belgium · Bernoulli Institute ‡ University of Groningen, Netherlands

Continual LearningCatastrophic Forgetting
AskPDF