Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

Sima Hashemi; Daniel Durstewitz; Georgia Koppe

cs.LGSep 29, 2026

Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

Authors: Sima Hashemi, Daniel Durstewitz, Georgia Koppe

Organizations: Faculty of Mathematics and Computer Science, Interdisciplinary Center for Scientific Computing, Heidelberg University, Heidelberg, Germany · Department of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Medical Faculty Mannheim, Heidelberg University, Heidelberg, Germany · Hector Institute for AI in Psychiatry & Department of Psychiatry and Psychotherapy, CIMH, Medical Faculty Mannheim, Heidelberg University, Heidelberg, Germany · Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany

Abstract

Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continual DSR (cDSR) requires learning new systems while preserving previously learned dynamics, yet even small parameter updates in recurrent models can qualitatively alter their behavior over long autonomous rollouts. We benchmark established continual learning (CL) methods spanning parameter regularization, replay, and parameter isolation on the fully trainable and interpretable Almost-Linear RNN (AL-RNN). Parameter isolation preserves earlier dynamics most effectively, but excessive task-specific allocations can rapidly exhaust a fixed-size network. We therefore introduce Continually-Recyclable Unit-Gating (CRUG), which conserves capacity through compact allocation and forward transfer. Differentiable gates trained with an $L_0$ -based penalty select task-specific units, while unused units are recycled for subsequent tasks. Directed connections allow later tasks to reuse earlier representations without affecting the dynamics of previously committed units. CRUG achieves the strongest reconstruction--capacity trade-off among the tested methods with zero forgetting and reliably learns a heterogeneous sequence of nonlinear and chaotic systems. Furthermore, we show that forward transfer is more pronounced and useful when tasks share similar underlying dynamics. Lastly, we demonstrate that CRUG's advantages extend beyond autonomous cDSR to sequential cognitive tasks.

Figures & tables

Figure 1: Continually-Recyclable Unit-Gating (CRUG). A) Dynamical systems are learned sequentially while retaining previously learned dynamics. B) During training on task

k

, the connections of previously committed units remain frozen. Forward transfer connections from committed to free units are trainable, whereas reverse connections are held at zero to protect earlier dynamics from interference. C) All free units initially participate in learning task

k

. During training, the

L_{0}

penalty closes unnecessary gates. At the end of the task’s training phase, units with gate values above a threshold are committed and frozen, while the remaining units are reinitialized and recycled for task

k+1

. Darker shades indicate higher gate values. D) For inference after all tasks have been learned, the model is initialized from a given starting point and evolved for

T

time steps. After the initial transient, all learned dynamical systems run concurrently within the model. Each task’s reconstruction can be read out from its corresponding set of readout units. For clarity, only transfer connections are shown; connections among units within each task are omitted.

	$D_{\mathrm{stsp}}$				Overall performance
Method	Van der Pol	Lorenz-63	Rössler	Chua	$D_{\mathrm{stsp}}\downarrow$	$D_{H}\downarrow$
Interleaved	$0.05^{+0.18}_{-0.02}$	$0.25^{+0.06}_{-0.07}$	$3.37^{+1.69}_{-1.68}$	$0.90^{+0.48}_{-0.18}$	$3.37^{+1.69}_{-1.65}$	$0.05^{+0.00}_{-0.00}$
Naive	$0.02^{+0.01}_{-0.01}\!\rightarrow\!\mathrm{div.}$	$0.20^{+0.02}_{-0.03}\!\rightarrow\!\mathrm{div.}$	$1.00^{+0.52}_{-0.13}\!\rightarrow\!\mathrm{div.}$	$0.69^{+0.10}_{-0.10}$	$\mathrm{div.}$	$\mathrm{div.}$
EWC	$0.02^{+0.01}_{-0.00}\!\rightarrow\!\mathrm{div.}$	$0.20^{+0.15}_{-0.01}\!\rightarrow\!\mathrm{div.}$	$3.20^{+1.74}_{-1.38}\!\rightarrow\!15.35^{\dagger}_{-0.07}$	$1.15^{+0.40}_{-0.25}$	$\mathrm{div.}$	$\mathrm{div.}$
SI	$0.02^{+0.01}_{-0.01}\!\rightarrow\!0.50^{+0.62}_{-0.24}$	$15.00^{+0.02}_{-0.03}\!\rightarrow\!14.99^{+0.03}_{-0.02}$	$15.20^{\dagger}_{-0.70}\!\rightarrow\!15.19^{+0.35}_{-0.66}$	$14.51^{+0.06}_{-0.06}$	$15.19^{+0.35}_{-0.19}$	$\mathrm{div.}$
ER	$0.02^{+0.01}_{-0.01}\!\rightarrow\!0.04^{+0.17}_{-0.02}$	$0.24^{+0.04}_{-0.04}\!\rightarrow\!0.21^{+0.07}_{-0.03}$	$2.67^{+0.94}_{-0.90}\!\rightarrow\!10.11^{+0.66}_{-2.20}$	$0.82^{+0.37}_{-0.08}$	$10.11^{+0.66}_{-2.20}$	$0.04^{+0.01}_{-0.00}$

Table 1: Benchmarking CRUG against other CL methods on cDSR. Each entry in the four first columns reports the

D_{\mathrm{stsp}}

measure immediately after the task’s own training phase

\rightarrow

after all tasks have been learned sequentially (except for interleaved training). Values are reported in the format

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across 10 runs (lower is better). Divergent or incomplete runs were assigned the worst score when computing the median. Overall performance is the maximum final task

D_{\text{stsp}}

and

D_{H}

within tasks in each run, summarized across all runs. “div.” denotes an infinite median, and † denotes an infinite upper quartile. Chua is learned last and therefore has no subsequent forgetting period. For CLNP method

3/10

runs failed to complete the sequence.

Figure 2: Benchmarking CRUG against other CL methods. A–B) Dynamical systems reconstruction. Four dynamical systems, ranging from limit-cycle to chaotic dynamics, are learned sequentially. A) Rollout trajectories generated by the final model for all four tasks. B)

D_{\text{stsp}}

D_{H}

, and the number of committed linear (dashed) and nonlinear (solid) units throughout training. Gray vertical lines mark task boundaries; lines and shaded regions show the median and interquartile range. C–D) Cognitive tasks. Nine cognitive tasks are learned sequentially using each CL method. C) Mean final accuracy as a function of the number of tasks learned. D) Mean capacity usage as a function of the number of tasks learned for the parameter-isolation methods CLNP and CRUG. CRUG achieves the highest final accuracy while using substantially less network capacity.

Figure 3: Forward-transfer mechanisms analysis. A, B) Final overall

D_{\text{stsp}}

(worst task after sequential training on four tasks) versus total units retained; lower is better. Faded points show individual hyperparameter configurations (median over 10 seeds), and step curves their Pareto fronts. Stars mark runs analyzed in C. A) Heterogeneous sequence: transfer provides no clear advantage. B) Related sequence (Lorenz-63,

\rho=28\rightarrow 313\rightarrow 200\rightarrow 160

): all transfer mechanisms outperform no transfer. C) Root-mean-square weights,

\mathrm{RMS}(W)

, of the task-specific and transfer blocks for the starred runs (tasks 2–4; 10 seeds). Bracket indicates two-sided Mann–Whitney

U

tests; ***

p<0.001

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Dynamical systems used in the cDSR benchmarks. All eight dynamical systems considered in this study are shown. A) First benchmark: Van der Pol $\rightarrow$ Lorenz-63 $\rightarrow$ Rössler $\rightarrow$ Chua. B) Second benchmark: Blasius $\rightarrow$ Laser $\rightarrow$ Genesio–Tesi $\rightarrow$ Finance.

Transfer Mechanism	$\lambda_{\text{tr}}$
Naive	–
$L_{0}$	$[10^{-6},10^{-3}]$
$L_{1}$	$[10^{-6},10^{-2}]$
$L_{2}$	$[3\times 10^{-5},3\times 10^{-1}]$

Appendix

Table 2: Hyperparameter search ranges for the transfer mechanisms.

Figure 5: cDSR on synthetic DS sequence 2. Four additional chaotic dynamical systems were trained sequentially. A) The final model was used to generate rollout trajectories for all four tasks. B) Reconstructed and ground-truth $x$ -dimensions are shown over 1000 time steps.

	$D_{\mathrm{stsp}}$				Overall performance
Method	Blasius	Laser	Genesio–Tesi	Finance	$D_{\mathrm{stsp}}\downarrow$	$D_{H}\downarrow$
Interleaved	$0.29^{+0.02}_{-0.10}\!\rightarrow\!0.29^{+0.02}_{-0.10}$	$0.41^{+0.17}_{-0.11}\!\rightarrow\!0.41^{+0.17}_{-0.11}$	$0.57^{+0.16}_{-0.20}\!\rightarrow\!0.57^{+0.16}_{-0.20}$	$0.31^{+0.09}_{-0.00}$	$0.73^{+0.14}_{-0.26}$	$0.06^{+0.00}_{-0.01}$
Naive	$0.16^{+0.01}_{-0.01}\!\rightarrow\!\text{div.}$	$0.20^{+0.02}_{-0.02}\!\rightarrow\!15.28^{\dagger}_{-0.06}$	$0.29^{+0.07}_{-0.03}\!\rightarrow\!14.76^{\dagger}_{-0.25}$	$0.24^{+0.00}_{-0.01}$	div.	div.
EWC	$0.15^{+0.02}_{-0.01}\!\rightarrow\!\text{div.}$	$0.18^{+0.02}_{-0.05}\!\rightarrow\!15.23^{\dagger}_{-0.01}$	$0.41^{+0.10}_{-0.12}\!\rightarrow\!14.86^{\dagger}_{-0.36}$	$0.31^{+0.03}_{-0.03}$	div.	div.
SI	$0.17^{+0.02}_{-0.03}\!\rightarrow\!0.16^{+0.01}_{-0.00}$	$15.25^{+0.02}_{-0.04}\!\rightarrow\!15.25^{+0.02}_{-0.04}$	$14.50^{+0.35}_{-0.00}\!\rightarrow\!14.50^{+0.35}_{-0.00}$	$15.01^{+0.13}_{-0.03}$	$15.25^{+0.02}_{-0.04}$	div.
ER	$0.14^{+0.02}_{-0.02}\!\rightarrow\!0.51^{+0.07}_{-0.34}$	$0.18^{+0.06}_{-0.00}\!\rightarrow\!0.47^{+0.16}_{-0.06}$	$0.38^{+0.04}_{-0.01}\!\rightarrow\!0.95^{+0.56}_{-0.63}$	$0.31^{+0.01}_{-0.08}$	$1.51^{+7.68}_{-0.87}$	$0.09^{+0.11}_{-0.01}$

Appendix

Table 3: Benchmarking CRUG against other continual learning methods on four additional DSR tasks (Blasius

\rightarrow

Laser

\rightarrow

Genesio–Tesi

\rightarrow

Finance). Each entry in the four first columns reports the

D_{\mathrm{stsp}}

measure immediately after the task’s own training phase

\rightarrow

after all tasks have been learned sequentially (except for interleaved training). Values are reported in the format

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across 5 runs (lower is better). Divergent or incomplete runs are not omitted but counted as worst results in the median. Overall performance is the maximum final task

D_{\text{stsp}}

and

D_{H}

within each run, summarized across all runs. “div.” denotes an infinite median, and † denotes an infinite upper quartile. Finance is learned last and therefore has no subsequent forgetting period.

	$D_{\mathrm{stsp}}$				Overall performance
$(\lambda_{\mathrm{ReLU}},\lambda_{\mathrm{lin}},\lambda_{\mathrm{tr}})$	Van der Pol	Lorenz-63	Rössler	Chua	$D_{\mathrm{stsp}}\downarrow$	$D_{H}\downarrow$	Units kept
$(6.58\times 10^{-4},2.43\times 10^{-4},4.94\times 10^{-2})$	$0.06^{+0.00}_{-0.01}\to 0.06^{+0.00}_{-0.01}$	$0.23^{+0.13}_{-0.03}\to 0.28^{+0.12}_{-0.08}$	$0.91^{+0.72}_{-0.17}\to 0.83^{+0.95}_{-0.16}$	$0.75^{+0.08}_{-0.09}$	$0.92^{+0.85}_{-0.12}$	$0.05^{+0.01}_{-0.00}$	$116^{+13}_{-8}$
$(7.66\times 10^{-4},2.52\times 10^{-4},8.47\times 10^{-2})$	$0.08^{+0.03}_{-0.02}\to 0.08^{+0.03}_{-0.02}$	$0.29^{+0.09}_{-0.08}\to 0.34^{+0.13}_{-0.13}$	$0.86^{+0.29}_{-0.13}\to 0.89^{+0.14}_{-0.15}$	$0.79^{+0.09}_{-0.22}$	$0.92^{+0.24}_{-0.08}$	$0.06^{+0.01}_{-0.01}$	$112^{+3}_{-4}$
$(5.69\times 10^{-4},1.88\times 10^{-4},7.50\times 10^{-2})$	$0.07^{+0.04}_{-0.01}\to 0.07^{+0.04}_{-0.01}$	$0.26^{+0.16}_{-0.07}\to 0.34^{+0.12}_{-0.12}$	$0.77^{+0.58}_{-0.07}\to 0.77^{+0.16}_{-0.14}$	$0.64^{+0.19}_{-0.05}$	$0.93^{+0.21}_{-0.12}$	$0.06^{+0.01}_{-0.01}$	$124^{+3}_{-5}$
$(5.87\times 10^{-4},2.02\times 10^{-4},2.40\times 10^{-1})$	$0.07^{+0.01}_{-0.01}\to 0.07^{+0.01}_{-0.01}$	$0.27^{+0.14}_{-0.05}\to 0.24^{+0.16}_{-0.02}$	$1.04^{+0.05}_{-0.14}\to 0.86^{+0.31}_{-0.21}$	$0.73^{+0.02}_{-0.05}$	$0.95^{+0.26}_{-0.06}$	$0.05^{+0.02}_{-0.01}$	$120^{+4}_{-2}$
$(5.56\times 10^{-4},2.34\times 10^{-4},2.36\times 10^{-1})$	$0.06^{+0.02}_{-0.01}\to 0.06^{+0.02}_{-0.01}$	$0.21^{+0.21}_{-0.01}\to 0.21^{+0.18}_{-0.02}$	$0.93^{+0.98}_{-0.11}\to 0.90^{+0.74}_{-0.16}$	$0.81^{+0.03}_{-0.14}$	$0.96^{+0.68}_{-0.12}$	$0.05^{+0.00}_{-0.00}$	$116^{+6}_{-1}$

Appendix

Table 4: CRUG with

L_{2}

transfer regularization: ten best configurations on the canonical and the additional DSR benchmark. Top: Van der Pol

\to

Lorenz-63

\to

Rössler

\to

Chua, 40 Optuna trials, seeds 0–9. Bottom: Blasius

\to

Laser

\to

Genesio–Tesi

\to

Finance, 20 Optuna trials, 10 seeds. Configurations are ranked by the median across seeds of the maximum final

D_{\mathrm{stsp}}

over the four tasks. Task entries show own-training

\to

final

D_{\mathrm{stsp}}

; the last task has only its own-training evaluation. Entries show

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across all seeds. “div.” denotes an infinite median. Overall

D_{H}

is the maximum final

D_{H}

over the four tasks within each run. Units kept is the total number of committed units across the four tasks (of

M=160

). Bold rows mark the configurations reported in the method comparison Tables 1 and 3 .

	$D_{\mathrm{stsp}}$				Overall performance
Method	Blasius	Laser	Genesio–Tesi	Finance	$D_{\mathrm{stsp}}\downarrow$	$D_{H}\downarrow$	Kept units
LRCV	$0.12^{+0.03}_{-0.02}$	$0.39^{+0.14}_{-0.10}\!\rightarrow\!0.42^{+0.11}_{-0.12}$	$1.10^{+0.45}_{-0.16}\!\rightarrow\!1.10^{+0.75}_{-0.17}$	$0.75^{+0.06}_{-0.10}\!\rightarrow\!0.79^{+0.03}_{-0.08}$	$1.10^{+0.75}_{-0.13}$	$0.06^{+0.02}_{-0.02}$	$70.0^{+3.5}_{-6.5}$
RCVL	$0.06^{+0.02}_{-0.01}\!\rightarrow\!0.06^{+0.02}_{-0.01}$	$0.41^{+0.06}_{-0.12}$	$2.21^{+0.93}_{-0.63}\!\rightarrow\!2.21^{+0.89}_{-0.49}$	$0.97^{+0.22}_{-0.24}\!\rightarrow\!0.97^{+0.15}_{-0.24}$	$2.21^{+0.89}_{-0.49}$	$0.07^{+0.01}_{-0.02}$	$61.0^{+3.0}_{-3.0}$
CVLR	$0.10^{+0.04}_{-0.03}\!\rightarrow\!0.10^{+0.04}_{-0.02}$	$0.31^{+0.12}_{-0.06}\!\rightarrow\!0.31^{+0.12}_{-0.06}$	$1.15^{+0.16}_{-0.14}$	$0.92^{+0.15}_{-0.17}\!\rightarrow\!0.86^{+0.12}_{-0.11}$	$1.20^{+0.44}_{-0.11}$	$0.07^{+0.01}_{-0.00}$	$65.5^{+3.5}_{-2.5}$
VLRC	$0.15^{+0.16}_{-0.06}\!\rightarrow\!0.15^{+0.16}_{-0.06}$	$0.40^{+0.13}_{-0.12}\!\rightarrow\!0.41^{+0.06}_{-0.12}$	$1.22^{+1.02}_{-0.21}\!\rightarrow\!1.26^{+1.16}_{-0.20}$	$0.98^{+0.12}_{-0.17}$	$1.26^{+1.16}_{-0.18}$	$0.08^{+0.02}_{-0.01}$	$63.0^{+3.0}_{-0.8}$

Appendix

Table 5: Task Order Robustness. Each entry reports performance immediately after the task’s own training phase

\rightarrow

performance after completion of the full sequence. Values are reported in the format

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across 10 runs (lower is better). Overall performance is computed within each run as the maximum final

D_{\mathrm{stsp}}

across tasks and then summarized across runs. The final task has no subsequent forgetting period, therefore reported as a single number.

$\bm{\lambda_{\mathrm{EWC}}}$	Van der Pol	Lorenz-63	Rössler	Chua	Overall $D_{\mathrm{stsp}}$
$2.325$	$0.02^{+0.00}_{-0.00}\to\mathrm{div.}$	$0.24^{+0.05}_{-0.05}\to\mathrm{div.}$	$0.54^{+1.00}_{-0.08}\to\mathrm{div.}$	$0.71^{+0.06}_{-0.08}$	$\mathrm{div.}$
$13.586$	$0.02^{+0.00}_{-0.00}\to\mathrm{div.}$	$0.21^{+0.10}_{-0.01}\to\mathrm{div.}$	$0.75^{+0.23}_{-0.07}\to 12.25^{*}$	$0.68^{+0.01}_{-0.00}$	$\mathrm{div.}$
$96.519$	$0.02^{+0.01}_{-0.01}\to 17.69^{*}$	$0.26^{+0.03}_{-0.10}\to\mathrm{div.}$	$3.25^{+2.53}_{-1.22}\to 15.08^{*}$	$0.74^{+0.15}_{-0.09}$	$\mathrm{div.}$
$1014.9$	$0.02^{+0.00}_{-0.01}\to 16.07^{*}$	$0.28^{+0.00}_{-0.03}\to 12.33^{*}$	$4.24^{+0.04}_{-0.11}\to 14.80^{+0.18}_{-0.18}$	$0.72^{+0.07}_{-0.11}$	$\mathrm{div.}$
$9792.4$	$0.02^{+0.00}_{-0.01}\to 16.60^{+0.55}_{-0.55}$	$0.25^{+0.14}_{-0.06}\to 13.73^{+0.12}_{-0.12}$	$2.60^{+0.72}_{-0.01}\to 15.27^{+0.01}_{-0.94}$	$1.00^{+0.01}_{-0.22}$	$\mathrm{div.}$
$89254$	$0.02^{+0.00}_{-0.00}\to 16.37^{+0.66}_{-0.66}$	$0.32^{+0.04}_{-0.10}\to 14.44^{+0.32}_{-0.32}$	$4.79^{+1.21}_{-1.92}\to 15.29^{*}$	$6.03^{+5.01}_{-3.39}$	$\mathrm{div.}$

Appendix

Table 6: EWC for different values of

\bm{\lambda_{\mathrm{EWC}}}

. For each task,

D_{\text{stsp}}

is reported right after that task’s own training phase and after the fourth (final) task’s training phase, in the format own

\to

final, given as

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

over 5 runs (diverged runs excluded; “div.” marks that all runs diverged). For small

\lambda_{\mathrm{EWC}}

, the model can learn all four tasks but substantially forgets earlier tasks by the end of training. As

\lambda_{\mathrm{EWC}}

increases, the model progressively loses plasticity, impairing the learning of later tasks. Thus, none of the tested values simultaneously provides sufficient stability and plasticity. Since all displayed settings yield an infinite median overall divergence, no setting is uniquely optimal; for the method comparison, we selected

\lambda_{\mathrm{EWC}}4214.16

(Optuna best) for the first benchmark and

\lambda_{\mathrm{EWC}}=10

for the second benchmark. ∗ Indicates that only one of five runs remained non-divergent; hence,

Q_{1}

and

Q_{3}

are not reported.

$\bm{\lambda_{\mathrm{SI}}}$	Van der Pol	Lorenz-63	Rössler	Chua	Overall $D_{\mathrm{stsp}}$
$10^{-2}$	$0.02^{+0.00}_{-0.01}\to\mathrm{div.}$	$1.06^{+0.28}_{-0.01}\to\mathrm{div.}$	$7.60^{+1.17}_{-1.09}\to 15.25^{+0.01}_{-1.10}$	$4.28^{+0.07}_{-0.23}$	$\mathrm{div.}$
$10^{-1}$	$0.02^{+0.00}_{-0.01}\to 17.70^{*}$	$8.83^{+0.17}_{-1.19}\to 15.06^{+0.01}_{-0.46}$	$12.73^{+0.87}_{-0.40}\to\mathrm{div.}$	$13.58^{+0.06}_{-0.08}$	$\mathrm{div.}$
$10^{0}$	$0.02^{+0.00}_{-0.01}\to 16.45^{+0.63}_{-0.23}$	$14.78^{+0.28}_{-0.02}\to 14.98^{+0.09}_{-0.12}$	$14.25^{+0.04}_{-0.09}\to 14.52^{+0.36}_{-0.01}$	$14.44^{+0.06}_{-0.03}$	$\mathrm{div.}$
$10^{1}$	$0.02^{+0.00}_{-0.00}\to 10.07^{+0.64}_{-2.16}$	$15.03^{+0.03}_{-0.14}\to 15.03^{+0.03}_{-0.11}$	$14.21^{+0.95}_{-0.05}\to 15.14^{+0.05}_{-0.42}$	$14.59^{+0.03}_{-0.07}$	$15.25^{\dagger}_{-0.11}$
$\mathbf{10^{2}}$	$\mathbf{0.02^{+0.00}_{-0.00}\to 0.31^{+0.07}_{-0.07}}$	$\mathbf{15.03^{+0.00}_{-0.03}\to 15.03^{+0.00}_{-0.04}}$	$\mathbf{15.16^{+0.04}_{-0.34}\to 14.83^{+0.35}_{-0.34}}$	$\mathbf{14.52^{+0.06}_{-0.19}}$	$\mathbf{15.16^{+0.11}_{-0.13}}$
$10^{3}$	$0.02^{+0.00}_{-0.00}\to 0.31^{+0.11}_{-0.28}$	$15.03^{+0.02}_{-0.00}\to 15.03^{+0.02}_{-0.00}$	$14.84^{+0.35}_{-0.37}\to 15.17^{+0.03}_{-0.33}$	$14.51^{+0.00}_{-0.17}$	$15.24^{\dagger}_{-0.07}$

Appendix

Table 7: SI for different values of

\bm{\lambda_{\mathrm{SI}}}

. For each task, the

D_{\text{stsp}}

metric is reported immediately after that task’s own training phase and after training on the fourth (final) task, in the format own

\to

final. Values are reported in the format

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across 5 runs (lower is better). “div.” denotes an infinite median, and † denotes an infinite upper quartile. For small

\lambda_{\text{SI}}

, the method is unable to prevent forgetting of previously learned tasks and also interferes with learning the new task. As

\lambda_{\text{SI}}

increases, retention of the first task improves, but the model progressively loses its plasticity and eventually fails to learn even the second task: its “own” value remains high (around 15) rather than decreasing toward zero. Thus, no value of

\lambda_{\text{SI}}

achieves both stability and plasticity simultaneously. ∗ Indicates that only one of five runs remained non-divergent; hence,

Q_{1}

and

Q_{3}

are not reported.

Method	$\bm{r}$	Van der Pol	Lorenz-63	Rössler	Chua	Overall $D_{\mathrm{stsp}}$
ER	$\bm{1}$	$\bm{0.03^{+0.01}_{-0.00}\to 0.03^{+0.25}_{-0.02}}$	$\bm{0.27^{+0.01}_{-0.00}\to 0.27^{+0.01}_{-0.05}}$	$\bm{3.84^{+2.40}_{-1.11}\to 10.82^{+0.67}_{-0.96}}$	$\bm{1.15^{+0.05}_{-0.35}}$	$\bm{10.82^{+0.67}_{-0.96}}$
	$5$	$0.02^{+0.00}_{-0.00}\to 0.08^{+1.21}_{-0.05}$	$0.31^{+0.08}_{-0.09}\to 0.38^{+0.04}_{-0.13}$	$2.36^{+0.21}_{-0.85}\to 11.99^{+0.68}_{-2.74}$	$0.82^{+0.01}_{-0.13}$	$11.99^{+0.68}_{-2.74}$
	$37$	$0.02^{+0.00}_{-0.00}\to 3.46^{+0.15}_{-0.60}$	$0.31^{+0.04}_{-0.08}\to 1.34^{+0.39}_{-0.04}$	$2.67^{+2.02}_{-0.93}\to 12.81^{+0.75}_{-1.21}$	$0.83^{+0.00}_{-0.07}$	$13.39^{+0.70}_{-1.15}$
	$74$	$0.02^{+0.00}_{-0.00}\to 6.40^{+0.83}_{-0.83}$	$0.32^{+0.06}_{-0.10}\to 3.62^{+11.39}_{-0.26}$	$2.90^{+1.21}_{-0.92}\to 14.10^{+0.09}_{-0.21}$	$0.60^{+0.17}_{-0.01}$	$\mathrm{div.}$
	$146$	$0.02^{+0.00}_{-0.00}\to 17.69^{+0.01}_{-0.16}$	$0.29^{+0.16}_{-0.12}\to 15.07^{+0.00}_{-0.00}$	$2.02^{+0.48}_{-0.40}\to 14.71^{+0.27}_{-0.27}$	$0.58^{+0.14}_{-0.01}$	$\mathrm{div.}$
	$193$	$0.02^{+0.01}_{-0.01}\to 17.69^{+0.00}_{-0.07}$	$0.31^{+0.01}_{-0.09}\to 13.62^{+1.39}_{-1.48}$	$2.55^{+0.27}_{-1.00}\to 13.48^{+1.74}_{-0.04}$	$0.59^{+0.19}_{-0.02}$	$17.70^{\dagger}_{-0.01}$

Appendix

Table 8: Experience replay (ER) and generative replay (GR) for different replay intervals

\mathbf{r}

. For each task,

D_{\text{stsp}}

is reported right after that task’s own training phase and after training on the fourth (final) task, in the format own

\to

final, given as

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

over 5 seeds. “div.” denotes an infinite median, and † denotes an infinite upper quartile. Forgetting increases steadily with

r

$\bm{P_{\mathrm{XdG}}}$	Van der Pol	Lorenz-63	Rössler	Chua	Overall $D_{\mathrm{stsp}}$
$0.08$	$0.01^{+0.00}_{-0.00}\to\mathrm{div.}$	$0.37^{+0.07}_{-0.15}\to 14.54^{+0.26}_{-0.33}$	$1.48^{+2.41}_{-0.14}\to\mathrm{div.}$	$0.78^{+0.17}_{-0.20}$	$\mathrm{div.}$
$0.29$	$0.04^{+0.01}_{-0.02}\to 17.23^{+0.23}_{-0.23}$	$0.25^{+0.03}_{-0.02}\to 15.06^{+0.02}_{-0.04}$	$4.71^{+1.70}_{-1.44}\to\mathrm{div.}$	$0.78^{+0.05}_{-0.05}$	$\mathrm{div.}$
$0.49$	$0.02^{+0.00}_{-0.00}\to\mathrm{div.}$	$0.35^{+0.06}_{-0.04}\to 14.89^{+0.13}_{-0.36}$	$2.23^{+0.48}_{-0.53}\to\mathrm{div.}$	$0.64^{+0.05}_{-0.11}$	$\mathrm{div.}$
$0.70$	$0.05^{+0.01}_{-0.00}\to\mathrm{div.}$	$0.35^{+0.18}_{-0.11}\to 15.06^{+0.00}_{-0.39}$	$2.60^{+0.28}_{-1.23}\to\mathrm{div.}$	$0.60^{+0.03}_{-0.04}$	$\mathrm{div.}$
$0.80$	$0.06^{+0.01}_{-0.03}\to\mathrm{div.}$	$0.49^{+0.18}_{-0.06}\to 15.04^{+0.03}_{-0.35}$	$2.98^{+0.39}_{-0.49}\to\mathrm{div.}$	$0.53^{+0.13}_{-0.04}$	$\mathrm{div.}$
$0.85$	$0.07^{+0.01}_{-0.02}\to 15.18^{*}$	$0.85^{+0.13}_{-0.00}\to 14.97^{+0.04}_{-0.32}$	$3.31^{+0.57}_{-1.72}\to\mathrm{div.}$	$0.55^{+0.22}_{-0.02}$	$\mathrm{div.}$

Appendix

Table 9: Context-dependent gating for different values of

\mathbf{P_{\text{XdG}}}

. The probability that a given unit is masked out (inactive) for a task is determined by

P_{\text{XdG}}

. For each task,

D_{\text{stsp}}

is reported right after that task’s own training phase and after the fourth (final) task’s training phase, in the format own

\to

final, given as

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

over 5 seeds (diverged seeds excluded; “div.” marks that all seeds diverged). For small

P_{\text{XdG}}

, task masks overlap substantially, and training later tasks causes earlier tasks to diverge. As

P_{\text{XdG}}

increases, overlap decreases, but even a few shared units still disrupt earlier tasks, while the capacity available to each task becomes too limited for the tasks to be learned well (

P_{\text{XdG}}=0.90

and

0.95

). ∗ Indicates that only one of five runs remained non-divergent; hence,

Q_{1}

and

Q_{3}

are not reported.

$\bm{(\alpha_{\text{ReLU}},\alpha_{\text{lin}},m)}$	$\bm{n}$	Van der Pol	Lorenz-63	Rössler	Chua	overall $D_{\text{stsp}}$	Total units kept
$(1.76\times 10^{-3},1.58\times 10^{-3},0.0215)$	$6/10$	$3.62^{\dagger}_{-1.41}\to 3.62^{\dagger}_{-1.41}$	$2.19^{\dagger}_{-0.27}\to 2.19^{\dagger}_{-0.27}$	$3.31^{\dagger}_{-0.75}\to 3.31^{\dagger}_{-0.75}$	$2.60^{\dagger}_{-0.46}\to 2.60^{\dagger}_{-0.46}$	$3.72^{\dagger}_{-0.43}$	$133.0^{\dagger}_{-58.8}$
$\mathbf{(1.67\times 10^{-3},1.56\times 10^{-3},0.0200)}$	$\mathbf{7/10}$	$\mathbf{2.59^{\dagger}_{-1.54}\to 2.59^{\dagger}_{-1.54}}$	$\mathbf{2.15^{\dagger}_{-0.51}\to 2.15^{\dagger}_{-0.51}}$	$\mathbf{3.53^{\dagger}_{-1.22}\to 3.53^{\dagger}_{-1.29}}$	$\mathbf{2.44^{\dagger}_{-0.07}\to 2.44^{\dagger}_{-0.07}}$	$\mathbf{3.81^{\dagger}_{-1.09}}$	$\mathbf{123.5^{\dagger}_{-47.0}}$
$(1.55\times 10^{-3},1.51\times 10^{-3},0.0372)$	$7/10$	$2.66^{\dagger}_{-0.93}\to 2.66^{\dagger}_{-0.93}$	$2.03^{\dagger}_{-0.23}\to 2.00^{\dagger}_{-0.20}$	$3.86^{\dagger}_{-1.32}\to 3.88^{\dagger}_{-1.24}$	$2.28^{\dagger}_{-0.39}\to 2.17^{\dagger}_{-0.32}$	$4.87^{\dagger}_{-1.45}$	$98.0^{\dagger}_{-12.8}$
$(1.73\times 10^{-3},1.71\times 10^{-3},0.0222)$	$7/10$	$2.79^{+3.20}_{-1.48}\to 2.79^{+3.20}_{-1.48}$	$1.58^{+10.20}_{-0.54}\to 1.58^{+10.20}_{-0.54}$	$3.33^{\dagger}_{-0.98}\to 3.33^{\dagger}_{-0.98}$	$2.61^{\dagger}_{-0.54}\to 2.61^{\dagger}_{-0.54}$	$5.90^{\dagger}_{-2.42}$	$106.5^{+37.2}_{-36.8}$
$(3.77\times 10^{-3},2.13\times 10^{-3},0.0966)$	$8/10$	$5.28^{+3.30}_{-0.98}\to 5.28^{+3.30}_{-0.98}$	$3.34^{+1.01}_{-0.72}\to 3.26^{+0.95}_{-0.65}$	$3.98^{+1.12}_{-0.58}\to 3.98^{+1.14}_{-0.50}$	$7.82^{+5.10}_{-2.62}\to 7.82^{+5.10}_{-2.76}$	$10.82^{+3.81}_{-4.72}$	$101.0^{+7.0}_{-22.8}$
$(3.13\times 10^{-3},1.77\times 10^{-3},0.0749)$	$6/10$	$4.10^{\dagger}_{-1.75}\to 4.10^{\dagger}_{-1.75}$	$2.42^{\dagger}_{-0.23}\to 2.34^{\dagger}_{-0.11}$	$5.75^{\dagger}_{-2.72}\to 4.56^{\dagger}_{-1.53}$	$11.47^{\dagger}_{-7.94}\to 11.47^{\dagger}_{-7.89}$	$11.47^{\dagger}_{-7.28}$	$120.0^{\dagger}_{-40.8}$

Appendix

Table 10: CLNP for different

L_{1}

coefficients and graceful-forgetting margins. For each task,

D_{\text{stsp}}

is reported right after that task’s own training phase and after the fourth (final) task’s training phase, in the format own

\to

final, given as

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

over the seeds (out of 10) that completed training (

n

). Divergent or incomplete runs are not omitted but counted as worst results in the median. Overall performance is the maximum final task

D_{\text{stsp}}

within each run, summarized across all runs. “div.” denotes an infinite median, and † denotes an infinite upper quartile. The last column gives the median total number of units kept across all four tasks (out of

M=160

Figure 6: Reconstructions of the Lorenz-63 family DSR tasks ( $\rho\in\{28,313,200,160\}$ ) by a trained model (colored), shown together with the ground-truth trajectories (gray). The model successfully reconstructs all four dynamical systems.

Figure 7: Reconstruction of gait dynamics. An AL-RNN using CRUG method was trained on 12 subjects from the gait dataset, three per group, in the order $\text{Control}_{1}\rightarrow\text{Parkinson}_{1}\rightarrow\text{Huntington}_{1}\rightarrow\text{ALS}_{1}\rightarrow\text{Control}_{2}\rightarrow\cdots$ . After all 12 subjects had been learned sequentially, the final model was used to reconstruct each subject’s trajectories. For each subject, the top panel shows the test data (gray) with the mean trajectory (black), and the colored trajectory in bottom panel shows the model’s reconstruction, demonstrating that the model successfully learned all subjects.

	$\widetilde{D}_{\mathrm{stsp}}$ per group				Overall
$(\lambda_{\mathrm{ReLU}},\,\lambda_{\mathrm{lin}}/\lambda_{\mathrm{ReLU}},\,\lambda_{\mathrm{tr}})$	Control	PD	HD	ALS	$\widetilde{D}_{\mathrm{stsp}}$	$D_{H}$	Units	Rel. forgetting (%)
$(1.12\times 10^{-3},\,0.20,\,1.70\times 10^{-4})$	$1.72^{+0.34}_{-0.13}$	$2.35^{+0.01}_{-0.17}$	$4.25^{+0.46}_{-0.09}$	$1.67^{+0.06}_{-0.01}$	$2.04^{+0.16}_{-0.03}$	$0.041^{+0.002}_{-0.000}$	$730^{+2}_{-42}$	$+0.0^{+0.2}_{-0.0}$
$(1.09\times 10^{-3},\,0.21,\,3.63\times 10^{-5})$	$1.97^{+0.25}_{-0.12}$	$2.07^{+0.03}_{-0.08}$	$5.16^{+0.26}_{-0.73}$	$1.54^{+0.09}_{-0.00}$	$2.07^{+0.04}_{-0.02}$	$0.046^{+0.001}_{-0.004}$	$690^{+31}_{-14}$	$+0.3^{+0.1}_{-1.5}$
$(1.09\times 10^{-3},\,0.21,\,2.83\times 10^{-4})$	$1.88^{+0.20}_{-0.10}$	$2.28^{+0.03}_{-0.14}$	$4.49^{+0.22}_{-0.17}$	$1.62^{+0.12}_{-0.10}$	$2.10^{+0.10}_{-0.08}$	$0.044^{+0.004}_{-0.002}$	$741^{+4}_{-6}$	$+0.0^{+0.1}_{-0.0}$
$(2.06\times 10^{-3},\,0.30,\,2.33\times 10^{-4})$	$2.24^{+0.09}_{-0.22}$	$2.08^{+0.03}_{-0.08}$	$4.51^{+0.24}_{-0.25}$	$1.81^{+0.06}_{-0.07}$	$2.19^{+0.11}_{-0.05}$	$0.047^{+0.001}_{-0.001}$	$555^{+7}_{-16}$	$-0.4^{+0.2}_{-0.1}$
$(2.52\times 10^{-3},\,0.36,\,5.84\times 10^{-4})$	$1.65^{+0.22}_{-0.00}$	$2.25^{+0.14}_{-0.10}$	$4.94^{+0.11}_{-0.50}$	$1.89^{+0.04}_{-0.05}$	$2.20^{+0.06}_{-0.06}$	$0.043^{+0.002}_{-0.001}$	$487^{+12}_{-0}$	$+0.0^{+0.2}_{-0.0}$

Appendix

Table 11: Hyperparameter search for CRUG on the gait dataset. We ran a hyperparameter search with 40 trials (3 seeds each); the nine best trials are shown, ranked by overall

\widetilde{D}_{\mathrm{stsp}}

aggregated over all 12 subjects. Entries show

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across seeds. Per-group columns report the median

\widetilde{D}_{\mathrm{stsp}}

over the three subjects in each group (PD: Parkinson’s disease; HD: Huntington’s disease; ALS: amyotrophic lateral sclerosis). Overall

D_{H}

and the total number of kept units indicate temporal agreement and capacity usage, respectively. Relative forgetting is the median over subjects of the relative increase in

\widetilde{D}_{\mathrm{stsp}}

from immediately after training on a subject to the end of the sequence; positive values indicate forgetting. The selected configuration is highlighted in bold.

$(\lambda_{\mathrm{ReLU}},\,\lambda_{\mathrm{lin}}/\lambda_{\mathrm{ReLU}},\,\lambda_{\mathrm{tr}})$	$D_{\mathrm{stsp}}$	$D_{H}$	Units
(a) Control
$(1.09\times 10^{-3},\,0.21,\,4.63\times 10^{-4})$	$1.91^{+0.03}_{-0.06}$	$0.040^{+0.001}_{-0.001}$	$176^{+0.5}_{-14.5}$
$(1.02\times 10^{-3},\,0.22,\,8.81\times 10^{-4})$	$1.92^{+0.28}_{-0.00}$	$0.044^{+0.005}_{-0.005}$	$163^{+7.0}_{-4.0}$
$(1.08\times 10^{-3},\,0.20,\,1.93\times 10^{-4})$	$1.94^{+0.16}_{-0.02}$	$0.045^{+0.002}_{-0.002}$	$164^{+6.5}_{-2.5}$
$(1.28\times 10^{-3},\,0.26,\,5.13\times 10^{-4})$	$1.95^{+0.16}_{-0.07}$	$0.041^{+0.001}_{-0.001}$	$151^{+2.5}_{-0.0}$
$(1.18\times 10^{-3},\,0.22,\,3.60\times 10^{-3})$	$2.00^{+0.12}_{-0.09}$	$0.040^{+0.002}_{-0.001}$	$158^{+1.5}_{-4.5}$

Appendix

Table 12: Hyperparameter search for CRUG on the individual subject groups of the gait dataset. Each block shows the best configurations for one subject group, trained sequentially on its three subjects and ranked by overall

\widetilde{D}_{\mathrm{stsp}}

. Entries show

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

across 3 seeds.

	$\widetilde{D}_{\mathrm{stsp}}$			Kept units
Group	Joint	Group	$p$	Joint	Group	$p$
Control	$\mathbf{1.65}^{+1.20}_{-0.04}$	$1.91^{+0.70}_{-0.09}$	0.80	$\mathbf{113}^{+0.0}_{-2.0}$	$176^{+0.5}_{-14.5}$	0.10
PD	$\mathbf{2.25}^{+0.38}_{-0.76}$	$2.95^{+0.27}_{-0.08}$	0.040	$\mathbf{118}^{+3.0}_{-7.0}$	$173^{+3.0}_{-10.5}$	0.10
HD	$\mathbf{4.94}^{+0.86}_{-0.82}$	$5.83^{+4.48}_{-0.78}$	0.077	$\mathbf{152}^{+2.0}_{-9.0}$	$221^{+8.0}_{-3.5}$	0.10
ALS	$\mathbf{1.84}^{+0.12}_{-0.18}$	$2.56^{+0.53}_{-0.10}$	0.0005	$\mathbf{120}^{+2.0}_{-1.0}$	$154^{+0.5}_{-5.0}$	0.10
All 12 subjects	$\mathbf{2.12}^{+1.20}_{-0.47}$	$2.92^{+0.79}_{-0.60}$	0.021	$\mathbf{487}^{+12.0}_{-0.0}$	$697^{+19.5}_{-0.5}$	0.10

Appendix

Table 13: Joint versus per-group training of CRUG on the gait dataset. Joint: a single model trained sequentially on all 12 subjects. Group: a separate model trained for each subject group. Entries show

\text{median}^{\,Q_{3}-\text{median}}_{\,Q_{1}-\text{median}}

;

\widetilde{D}_{\mathrm{stsp}}

is pooled over subjects and seeds (9 values per group, 36 for all subjects), and kept units over seeds (3 values).

p

-values are from two-sided Mann–Whitney

U

tests comparing joint and per-group training; with three seeds per condition,

0.10

is the smallest attainable

p

-value for kept units.

Method	Accuracy (%) $\uparrow$	Used Capacity (%) $\downarrow$
Naive	$40.19\pm 5.00$	–
EWC	$93.78\pm 7.17$	–
SI	$80.74\pm 13.89$	–
ER	$98.11\pm 0.98$	–
GR	$97.46\pm 1.69$	–
XdG	$79.14\pm 6.40$	–

Appendix

Table 14: Benchmarking CRUG against other continual learning methods on sequential cognitive tasks. The model is trained sequentially on nine cognitive tasks. Naive fine-tuning, without any continual learning mechanism, exhibits pronounced catastrophic forgetting. All continual learning methods substantially outperform naive fine-tuning, although their effectiveness varies: EWC, SI and XdG show the largest performance degradation, the replay-based methods (ER, GR) perform considerably better, and the capacity-allocating methods perform best. CRUG and CLNP achieve comparable accuracy, but CRUG attains the highest mean final accuracy while using the least network capacity (58.4%), whereas CLNP occupies almost the entire network (94.2%). For CLNP, 4 of the 10 seeds exhausted the unit pool before completing the task sequence and therefore produced no final model; these runs are excluded from the reported mean and standard deviation. Capacity usage is not reported (“–”) for methods that do not isolate task-specific units and instead operate on all recurrent units.

Explore similar work

May 12, 2026cs.LG

Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length

T

, achieving logarithmic time complexity,

\mathcal{O}(\log T)

. Since sequence lengths have been practically limited due to the linear runtime complexity

\mathcal{O}(T)

of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences (

T>10^4

) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS.

Florian Hess, Florian Götz, Daniel Durstewitz

1Dept. of Theoretical Neuroscience, Central Institute of Mental Health,2026 Mannheim, Germany · Faculty of Mathematics and Computer Science, Heidelberg University, Germany · Faculty of Physics and Astronomy, Heidelberg University, Germany +1

Recurrent Neural Networks Recurrent Model Nonlinear Dynamics Sequence Modeling Backpropagation

Ask PDF

Jun 29, 2026cs.LG

Neural Subspace Reallocation: Continual Learning as Retrieval-Based Subspace Memory Management

We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks. We prove that in cyclic environments any memoryless allocation policy incurs cumulative regret Omega(T(M-1)Delta_switch) relative to a history-aware policy backed by the Bank (Theorem 1). Empirically, on Split-CIFAR-100 the Bank reduces cyclic recovery time by 10x, exactly as predicted, and on the heterogeneous 5-Datasets benchmark NSR achieves the highest accuracy and the least forgetting, about 9x closer to zero backward transfer than the memoryless heuristics. Crucially, we run a controlled study that isolates which component matters: holding the Bank fixed and varying only the allocation rule, we find that a simple similarity-based retrieval rule matches or beats a learned reinforcement-learning controller (recovering recurring tasks in 0 vs 1.8 steps and reaching equal accuracy). Our central, honest finding is therefore that the memory mechanism -- compression and similarity retrieval -- rather than a learned allocation policy, drives continual-learning performance under fixed capacity. A memory-budget analysis confirms the compressed Bank stays small -- 0.29 MB of parameter memory per task -- so a top-K retention cap bounds the total footprint while preserving fast recovery for retained tasks.

Byeong Hoon Yoon

Independent Researcher

Replay-Based Continual Learning Continual Learning Subspace Peak Memory Forgetting

Ask PDF

Sep 8, 2026cs.LG

Learning Length-Extrapolatable Recurrent Models

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.

Hanwen Jiang

Adobe Research

Recurrent Model Backpropagation Step-Level Credit

Ask PDF