Recent work in continual learning has highlighted the stability gap -- a temporary performance drop on previously learned tasks when new ones are introduced. This phenomenon reflects a mismatch between rapid adaptation and strong retention at task boundaries, underscoring the need for optimization mechanisms that balance plasticity and stability over abrupt distribution changes. While optimizers such as momentum-SGD and Adam introduce implicit multi-timescale behavior, they still exhibit pronounced stability gaps. Importantly, these gaps persist even under ideal joint training, making it crucial to study them in this setting to isolate their causes from other sources of forgetting. Motivated by how noradrenergic (neuromodulatory) bursts transiently increase neuronal gain under uncertainty, we introduce a dynamic gain scaling mechanism as a two-timescale optimization technique that balances adaptation and retention by transiently increasing the effective update magnitude while dynamically reparameterizing the weights governing the forward pass, thereby empirically mitigating transition-induced curvature amplification. Across domain- and class-incremental MNIST, CIFAR, and mini-ImageNet benchmarks under task-agnostic joint training, dynamic gain scaling effectively attenuates stability gaps while maintaining competitive accuracy, improving robustness at task transitions.
Figures & tables
Figure 1: Schematic of the stability gap. a) Conceptual illustration highlighting the distinction between the stability gap and catastrophic forgetting. While catastrophic forgetting arises from an inaccurate approximation of the joint loss, the stability gap emerges from a deviation of the optimization trajectory from the path of non- or minimally-increasing loss, leading to a transient drop in old-task performance. b) Old-task accuracy during sequential learning on Split CIFAR-10 under stochastic gradient descent with 0.9 momentum. The sharp performance drop at the task boundary quantifies the stability gap that arises from the deviated trajectory.
Figure 2: Simple proof of principle showing how gain boosts approximate a two-timescale optimizer. a) Schematic of gain-induced effective weight decomposition under noradrenergic neuromodulation. Phasic noradrenergic signals transiently increase neuronal gain, thereby “virtually” splitting the effective weight into a fast-slow scheme with wfast=[g(t)−g0]w(t) and wslow=g0w(t) . b) Left panel. Simplified model y(t)=Weff(t)x with constant input x predicting a target T(t) under mean squared error loss L=21[T(t)−y(t)]2 . Comparison of gain-modulated (orange), fast-slow (blue) and slow (purple) weight methods. The unshaded region corresponds to optimization toward target T1 , and the shaded region to optimization toward target T2 . The dashed vertical line marks the time of peak neuronal gain. Right panel. Effective-coordinate flattening induced by gain neuromodulation at the highest neuronal gain. In the effective-weight space, gain boosts reparameterize the loss, effectively reducing its curvature and scaling its gradient step. Dots illustrate each method’s state at the peak gain time, and arrow lengths indicate the gradient-step magnitudes.
Figure 3: Gain modulation attenuates transition-induced instability in Split MNIST. a) Samples of data around the first task transition, T1→2 , where the cumulative training distribution expands from digits {0,1} to {0,1,2,3} . b) Test accuracy on Task 1 after the first task transition. c) Neuronal gain dynamics g(t) for NGM-SGD, showing the phasic gain response following the task transition. d) Task 1 test loss increase after the task switch. e) Effective-coordinate curvature ratio Rλ(t)=∣λeff(t)∣/∣λeff(0)∣ , where ∣λeff(0)∣ denotes the effective curvature measured immediately before the transition. The horizontal dashed line indicates Rλ(t)=1 , corresponding to the pre-transition curvature. The vertical dashed lines in (b–d) mark the task switch at iteration zero. Curves show the mean, and shaded regions show the standard deviation across five independent runs. The curvature ratio is represented as the geometric mean ±1 standard deviation in log-space. f–g) Transition-level stability measures across the complete five-task sequence. f) Stability gap, computed as the maximum relative reduction in previous-task accuracy following each task switch (see Table 2 ). g) Previous-task loss spike, computed as the maximum post-transition loss minus the loss immediately before the switch. Bars and error bars denote the mean and standard deviation across five independent runs. Color coding denotes the different optimizers: our method (NGM-SGD, Algorithm 1 ) in orange, momentum-SGD (MSGD) in blue, Adam in green, and vanilla SGD in gray.
Figure 4: Effect of ablating gain modulation on Split MNIST. Comparison of the full NGM-SGD method (Algorithm 1 , orange) with three targeted ablations: Tonic gain (gold), which removes entropy-driven phasic modulation ( η=0 ); Instantaneous gain (brown), which removes temporal integration of the gain dynamics ( γ=0 ); and Entropy-LR (dark red), which applies the entropy-dependent signal directly to the learning rate ( α(t)=α0q(t)2 ). a) Test accuracy on Task 1 as the model is incrementally trained across all benchmark tasks. b) Cumulative test accuracy over all tasks observed up to the current training stage. c) End-of-task accuracy on the newly introduced task following each task switch. d) Trade-off between the average stability gap across task transitions and the final cumulative accuracy at the end of training. Curves represent the mean ± standard deviation (shaded area) over five runs with different random seeds. Bars and error bars denote the corresponding mean ± standard deviation. In panel (d), markers denote the mean values, while horizontal and vertical error bars indicate the propagated standard deviation of the average stability gap and the standard deviation of the final cumulative accuracy, respectively. Vertical dashed lines in panels (a–b) indicate task transitions. Horizontal dotted lines in panel a illustrate the min-ACC per task.
avg-ACC ( ↑ )
avg-min-ACC ( ↑ )
WC-ACC ( ↑ )
avg-SG ( ↓ )
Split MNIST
NGM-SGD (ours)
99.015 ± 0.047
97.570 ± 0.465
97.696 ± 0.373
0.017 ± 0.005
MSGD
98.971 ± 0.086
72.518 ± 0.403
77.628 ± 0.329
0.267 ± 0.004
Adam
99.088 ± 0.108
71.047 ± 3.753
76.537 ± 3.003
0.283 ± 0.038
SGD
98.975 ± 0.099
95.887 ± 2.887
96.293 ± 2.311
0.034 ± 0.029
Split CIFAR-10
NGM-SGD (ours)
90.630 ± 1.576
79.485 ± 3.875
80.880 ± 3.349
0.134 ± 0.043
MSGD
89.434 ± 2.134
52.828 ± 3.898
58.348 ± 3.521
0.425 ± 0.044
Table 1: Main quantitative metrics. For all benchmarks we report (across tasks) the average final accuracy (avg-ACC), average minimum accuracy (avg-min-ACC), average stability gap drop (avg-SG), and the worst-case accuracy (WC-ACC). Highlighted values indicate the best results.
Figure 5: Stability gaps across class- and domain-incremental learning. a) Class-incremental results on Split MNIST, Split CIFAR-10, and Split mini-ImageNet. b) Domain-incremental results on Rotated MNIST and Domain CIFAR-100. Within each block, the top row shows Task 1 test accuracy as the model is trained incrementally across all benchmark tasks, the middle row shows the corresponding Task 1 test loss, and the bottom row shows the evolution of neuronal gain. Curves and shaded regions represent the mean ± standard deviation over five runs with different random seeds, and dots indicate the minimum Task 1 accuracy reached within each task. NGM-SGD is shown in orange, momentum-SGD in blue, Adam in green, and vanilla SGD in gray. Accuracy panels are vertically restricted to emphasize the stability gaps; consequently, some performance drops may appear truncated.
Figure 6: Effect of resetting optimizer inertia at task transitions. Stability gaps for MSGD and Adam, with and without resetting their accumulated optimizer states at each task switch, across Split MNIST, Split CIFAR-10, Split mini-ImageNet, Rotated MNIST, and Domain CIFAR-100. Bars and error bars represent the mean ± standard deviation over five runs with different random seeds. Orange dotted segments indicate the corresponding NGM-SGD stability gap at each transition.
Figure 7: Gain across tasks. Plots of the mean neuronal gain in the first three tasks for class-incremental (top) and domain-incremental (bottom) experiments. We used a linear fit and obtained the slopes 2.16×10−1 for Split MNIST, 4.20×10−1 for Split CIFAR-10, 1.36×10−1 for Split mini-ImageNet, −6.63×10−2 for Rotated MNIST, and −1.59×10−1 for Domain CIFAR-100.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
SG( T1→2 )
SG( T2→3 )
SG( T3→4 )
SG( T4→5 )
Split MNIST
NGM-SGD (ours)
0.028 ± 0.015
0.012 ± 0.006
0.014 ± 0.008
0.013 ± 0.004
MSGD
1.000 ± 0.000
0.032 ± 0.013
0.018 ± 0.006
0.019 ± 0.009
Adam
0.883 ± 0.133
0.172 ± 0.062
0.052 ± 0.032
0.025 ± 0.013
SGD
0.100 ± 0.115
0.009 ± 0.005
0.007 ± 0.006
0.019 ± 0.012
Split CIFAR-10
NGM-SGD (ours)
0.133 ± 0.097
0.072 ± 0.060
0.072 ± 0.094
0.260 ± 0.090
MSGD
0.476 ± 0.007
0.321 ± 0.089
0.116 ± 0.117
0.786 ± 0.095
Appendix
Table 2: Stability gap metrics. For all joint training benchmarks, we report the per-transition stability gaps (mean ± std). Highlighted values indicate the best results.
LR
T1
T2
T3
T4
T5
1e-4
NGM-SGD (ours)
83.12 ± 1.35
74.14 ± 1.75
75.68 ± 2.38
75.83 ± 2.17
54.95 ± 5.02
MSGD
80.40 ± 1.09
74.89 ± 0.74
77.15 ± 1.00
77.33 ± 0.67
61.42 ± 1.78
Adam
92.07 ± 0.67
84.16 ± 1.89
87.53 ± 0.87
84.55 ± 2.65
75.02 ± 5.24
SGD
55.99 ± 4.80
64.75 ± 2.75
65.12 ± 1.73
60.76 ± 1.95
55.75 ± 2.35
1e-3
NGM-SGD (ours)
91.09 ± 1.46
85.89 ± 2.82
85.84 ± 1.50
85.77 ± 1.44
70.84 ± 4.72
MSGD
91.36 ± 1.35
85.31 ± 1.77
85.76 ± 1.22
86.24 ± 1.12
73.72 ± 2.82
Appendix
Table 3: Hyperparameter search: Final accuracies (ACC) for all tasks in the Split CIFAR-10 benchmark comparing different learning rates.
γ
η
avg-ACC ( ↑ )
avg-SG ( ↓ )
0.80
0.1
90.134 ± 1.292
0.182 ± 0.038
0.80
0.2
90.920 ± 1.396
0.182 ± 0.062
0.80
0.3
91.678 ± 0.875
0.215 ± 0.053
0.80
0.4
91.220 ± 1.068
0.176 ± 0.047
0.80
0.5
90.198 ± 1.471
0.170 ± 0.049
0.90
0.1
91.098 ± 1.263
0.159 ± 0.054
Appendix
Table 4: Hyperparameter search: Average final accuracies (avg-ACC) and average stability gaps (avg-SG) in the Split CIFAR-10 benchmark comparing different combinations of the (γ,η) parameters in the NGM-SGD method.
Figure 8: Hyperparameter sweep on Split CIFAR-10. a) Scores on best accuracy with the learning rate sweep for all methods. b) Scores on best accuracy and least stability gap with the hyperparameter sweep of (γ,η) over the NGM-SGD method.
Figure 9: Gain dynamics: a) Relationship between the average output entropy of each dataset and the corresponding gain scale. b) Gain trajectories (Task 1) comparing NGM-SGD with EMA-based baselines for the domain- and class-incremental benchmarks. EMA (+ baseline) settings match only the special case η=1−γ , whereas NGM-SGD allows independent control of decay and entropy-driven modulation.
avg-ACC ( ↑ )
avg-min-ACC ( ↑ )
WC-ACC ( ↑ )
avg-SG ( ↓ )
Split MNIST
NGM-SGD (ours)
99.015 ± 0.047
97.570 ± 0.465
97.696 ± 0.373
0.017 ± 0.005
SGD
98.975 ± 0.099
95.887 ± 2.887
96.293 ± 2.311
0.034 ± 0.029
Adam
99.088 ± 0.108
71.047 ± 3.753
76.537 ± 3.003
0.283 ± 0.038
Adam Reset
99.075 ± 0.147
86.563 ± 6.869
88.921 ± 5.496
0.127 ± 0.069
MSGD
98.971 ± 0.086
72.518 ± 0.403
77.628 ± 0.329
0.267 ± 0.004
MSGD Reset
98.939 ± 0.100
72.823 ± 0.374
77.876 ± 0.305
0.264 ± 0.004
Appendix
Table 5: Oracle reset results across benchmarks. Main quantitative metrics for all optimizers across benchmarks under joint training with momentum resets. Bold values indicate the best result per column. For Adam and MSGD, color highlights denote the metrics in which the momentum-reset variant outperforms its non-reset counterpart ( Green for Adam Reset and Blue for MSGD Reset).
Figure 10: Task 1 accuracy across replay experiments with a 1K buffer. Each panel shows per-iteration accuracy at task switches for all optimizers under a limited replay memory. On simple benchmarks (Split MNIST and Rotated MNIST), the stability gap is visible, whereas on more complex datasets (Split CIFAR-10, Split mini-ImageNet, and Domain CIFAR-100), the small buffer induces substantial steady-state forgetting that dominates over the transient drop. This illustrates how poor approximations of the joint loss can obscure the stability gap we aim to study.
Figure 11: Effect of backbone freezing on the stability gap in Split CIFAR-10. a) Task 1 test accuracy across five sequential tasks for NGM-SGD (Algorithm 1 ; orange), momentum-SGD (MSGD; blue), and MSGD with the convolutional backbone frozen after Task 1 (MSGD-frozen; cyan). Dashed horizontal segments and circular markers indicate the minimum Task 1 accuracy within each task. b) Training accuracy across tasks. Curves and shaded regions show the mean ± standard deviation across seeds.
The magnitude of parameter updates are considered a key factor in continual learning. However, most existing studies focus on designing diverse update strategies, while a theoretical understanding of the underlying mechanisms remains limited. Therefore, we characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space. By deriving the optimal parameter update magnitude that minimizes forgetting, we unify two representative update paradigms, frozen training and initialized training, within an optimization framework for constrained parameter updates. Our theoretical results further reveals that sequence tasks with small parameter distances exhibit better generalization and less forgetting under frozen training rather than initialized training. These theoretical insights inspire a novel hybrid parameter update strategy that adaptively adjusts update magnitude based on gradient directions. Experiments on deep neural networks demonstrate that this hybrid approach outperforms standard training strategies, providing new theoretical perspectives and practical inspiration for designing efficient and scalable continual learning algorithms.
JinLi He, Liang Bai, Xian Yang
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK
Continual learning (CL) requires models to learn tasks sequentially, yet deep neural networks often suffer from plasticity loss and poor knowledge transfer, which can impede their long-term adaptability. Drawing high-level inspiration from global neuromodulatory mechanisms in the brain, we introduce Neuromodulation and Synchronization (NeuMoSync), a novel architecture that integrates dynamic, neuron-specific modulation into deep neural networks to enhance their adaptability and plasticity. NeuMoSync extends standard neural network architectures with learnable feature vectors for each neuron that track network-wide historical context and with a module operating at a higher level of abstraction. This module synthesizes neuron-specific signals, conditioned on both current inputs and the network's evolving state, to adaptively regulate activation dynamics and synaptic plasticity. Evaluated on diverse CL benchmarks, including memorization (Random Label CIFAR-10 and Random Label MNIST), concept drift (Shuffle CIFAR-10 and Shuffle Mini-ImageNet), class-incremental learning (Class Split ImageNet and Class Split CIFAR-100), and domain-incremental learning (Permuted MNIST), NeuMoSync demonstrates strong performance in retaining plasticity and achieves improvements in both forward and backward adaptation compared with existing methods. Ablation studies validate the necessity of each component, while analysis of the learned modulatory signals reveals interpretable coordination patterns across tasks. Our work underscores the potential of integrating global coordination mechanisms into deep learning systems to advance robust, adaptive continual learning. The code is publicly available at https://github.com/RoozbehRazavi/NeuMoSync.
Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.