Replay in the Silent Degrees of Freedom: Continual Learning Without an Offline Phase
Authors: Yanhai Zhang, Jie Zhang, Chi Xu
Organizations: Agency for Science, Technology and Research (A⋆STAR), Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
Replay-based continual learning rehearses past data either in an offline phase, during which the agent stops acting, or interleaved with the live stream, where it perturbs the computation serving the current input. An agent that learns in deployment can afford neither. Motivated by local sleep, the use-dependent off periods of individual cortical circuits in awake animals, we show that replay can instead be written into the degrees of freedom the current input leaves unused. In a network with k-winner-take-all hidden layers, confining replay updates to synapses whose presynaptic unit is silent or whose postsynaptic unit is inactive leaves the hidden computation on the current batch invariant: exactly so for silent and suppressed units, and for all but 0.3% of samples in practice. A refractory rule under which units that have just fired sit out the next competition doubles the width of this channel and carries most of the accuracy. On class-incremental split-MNIST the resulting learner, with no offline phase, matches or exceeds the best offline rehearsal schedule and outperforms experience replay, ER-ACE and unmasked interleaved replay, each re-tuned under the same micro-batch schedule. Against DER++ the comparison splits by protocol: with five epochs per task DER++ leads by 1.5 points once it runs under that schedule, and in a single pass over the stream, the regime closest to the agent deployment setting, the system leads it by 1.6 while offline rehearsal falls 15 points behind. On split CIFAR-10 it again leads offline rehearsal and experience replay, but trails ER-ACE and DER++ by two to three points. The construction is not tied to the local learner: on a backprop network with k-winner-take-all hidden layers under the same schedule, refractory rotation adds half a point in five epochs and three in a single pass, and isolation again costs nothing on top of it.
Figures & tables
Figure 1: Mechanism. (a) During each waking batch, replay from the episodic buffer is written only into synapses whose update is invisible to the current input, exactly so for silent presynaptic and suppressed units (Propositions 1 – 2 ); refractory rotation forces recently used units into the consolidable set (Corollary 1 ). (b) Two internal signals: unit-level homeostatic pressure triggers replay bursts, and a relative-novelty gate decides when rotation runs. The default system runs rotation always on and replays one micro-batch at every waking batch; the pressure trigger and the novelty gate are the schedulers studied under sparse replay budgets and lower activity fractions.
Figure 2: Consolidation schedules. Ticks are waking batches. The offline night pauses the stream; local sleep consolidates during it, in replay bursts confined to asleep units.
Figure 3: Training data flow. Each waking batch is inferred under the current suppression mask; its awake sets define the synapse and unit masks for the replay micro-batch applied in the same step and the rotation mask for the next batch. The masked step confines the velocity and the decay to the mask; the readout row is the one deliberately plastic exception. Unit-level pressure decides when replay bursts run.
Figure 4: Buffer size. Split-MNIST sequential accuracy against the buffer size K for the system and the replay references (held-out split, six seeds).
Figure 5: Left (isolation): final split-MNIST accuracy vs. replay batch size (default substrate, heavy-ball SGD); the number of updates is fixed within each series (one per waking batch; 20 per epoch for the night, whose samples therefore fall with its batch). The full system is flat down to 8 samples and loses 1.9 at 4 ( 12.7 at 2 , Appendix H ); rotation with unmasked replay (dashed) trails it by 0.1 to 0.3 , within seed noise, and fails severely in half the seeds at batch 2 ; without rotation, unmasked and isolated replay and the offline night all degrade. Right (timing): accuracy vs. number of waking replay batches. Trickles and clocked bursts fail alike; pressure-triggered bursts (star) keep most of the accuracy at a third of the events; surprise triggering fires no events in this regime.
Online
Anytime
Worst
Stability gap
Churn
isolated replay + rotation
97.7±0.1
94.9±0.3
66.2±4.4
1.5±0.3
0.33±0.04
rotation, unmasked replay
97.8±0.1
95.4±0.1
69.7±2.9
2.3±1.1
0.62±0.06
isolated replay, no rotation
97.6±0.1
94.1±0.3
63.9±1.9
16.9±1.8
0.30±0.03
unmasked, no rotation
95.9±1.3
92.9±1.4
32.9±26.0
15.9±7.3
4.39±2.15
offline rehearsal
87.6±0.2
83.3±0.7
33.1±4.9
38.4±0.2
—
no replay
91.3±0.7
43.0±0.1
18.7±0.2
51.8±0.0
—
Table 1: Measured while the stream runs (split-MNIST, K=1000 , held-out split; six seeds for the first two rows, three for the rest). Churn is undefined for schedules that do not replay during the stream.
Figure 6: Left: accuracy on the classes seen so far, measured every 50 waking batches; vertical lines are task switches and bands are one standard deviation over seeds. Offline rehearsal recovers the old tasks only when its night arrives, so for most of a task the stream is served by a network that has not yet been repaired. Right: the fraction of the current batch whose prediction the replay update changes, on a logarithmic axis, for the four cells of the isolation and rotation square; every seed is drawn.
Backprop, 512 – 256
lr
Sequential
i.i.d. (static)
Seq., single pass
+k -WTA, interleaved replay (unmasked)
3⋅10−5
91.7±0.4
97.2±0.1
89.3±0.6
+k -WTA, isolated replay
10−4
84.7±2.6
97.6±0.1
81.9±2.1
+k -WTA, interleaved replay + rotation
10−4
92.2±0.3
96.7±0.3
92.7±0.3
+k -WTA, isolated replay + rotation
10−4
92.1±0.3
96.6±0.1
92.8±0.2
ER
3⋅10−5
90.8±0.6
—
87.8±0.6
ER-ACE
3⋅10−4
89.7±0.4
—
90.5±0.6
Table 2: Backprop learners under our schedule ( 512 – 256 , Adam with weight decay 10−3 , waking batches of 16 with one 16 -sample replay micro-batch each, K=1000 ; held-out split, six seeds; the learning rate of every row re-selected on the development split). Upper block: k -WTA hidden layers at the same 10% fraction as the local learner. Middle block: the published baselines of Table 3 , unchanged except for the schedule. Lower block: the local system, for reference.
Variant
Sequential
i.i.d. (static)
Seq., single pass
full system : isolated replay + always-on rotation (heavy-ball SGD)
91.6±0.3
94.1±0.2
91.8±0.3
rotation gated by relative novelty, committed to bouts
91.8±0.3
94.4±0.3
rotation gated per batch
87.5±1.6
94.7±0.1
+ two-timescale anchor ( λ=μ=3⋅10−4 ), rotation always on
Table 3: Component ablation of the consolidation mechanism on the default substrate ( 512 – 256 / 10% ; backprop references dense, at the widths selected in Section 4.4 ). Sequential = final accuracy on split-MNIST; static = i.i.d. MNIST (the axis that must not degrade). Dashes: combination not applicable; blank: not run. Single pass = one epoch per task, otherwise identical. Held-out split; six seeds, except the gate, anchor, optimiser and unmasked-Adam rows (three).
Figure 7: The operating frontier at the 10% fraction under heavy-ball SGD: split-MNIST sequential accuracy against i.i.d. static accuracy (headline points six seeds). Blue: non-dominated by seed means (the bout-committed gate and the silent mask); purple: dominated; held-out split. The top cluster (always-on rotation, bout gate, anchor, masked Adam) lies within seed noise of one another; the frontier trades sequential for static accuracy through the silent mask, the unmasked rotation-free control is dominated by it once its collapsed seed is counted, and the adaptive- λ point (Section 7 ) is dominated on this split.
Figure 8: Left: the substrate decomposition on feature CIFAR-10, transplanting the local network’s ingredients into BP+ER one at a time (each cell at its better of two learning rates; three seeds). Right: spiking inference, i.i.d. MNIST accuracy against the idealised 45-nm FP32 arithmetic-energy proxy for Ts∈{4,8,16,32} , with the dense (square) and event-driven (diamond) rate references; held-out split; the default is monotone in Ts , the rotation-free control dips 0.4 at Ts=32 .
fixed λ (tuned)
controller (drive)
seq.
static
seq.
static
split-MNIST
91.6±0.3
94.1±0.2
90.6±0.6
93.5±4.3
split CIFAR-10
28.4±0.8
28.0±1.2
26.2±1.1
30.1±0.2
feature CIFAR-10
42.3±0.6
42.4±1.1
39.1±0.4
43.6±0.6
split CIFAR-100
1.2±0.1
1.0±0.1
3.2±0.8
4.1±0.6
+ features
1.1±0.3
1.1±0.2
5.2±1.6
6.4±0.6
Table 4: One dataset-independent ratio target replaces per-dataset decay tuning. Sequential / static accuracy (%) under the tuned fixed decay ( 10−3 on the image benchmarks; 10−4 with 3× targets as the CIFAR-100 rescue) against the drive-referenced controller of ( 10 ) at the same ρ=0.25 everywhere; held-out split, three seeds (six on split-MNIST and in the fixed-decay split CIFAR-10 cell); the full system (refractory rotation, isolated replay, K=1000 ) in every cell.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Channel
Condition
Invisible to X
Source
presynaptic silent
j∈/Aℓ−1(X)
for any magnitude
Prop. 1
postsynaptic naturally asleep, active pre
i∈/Aℓ(X) , siℓ=0
under the margin τk,bℓ
Prop. 2
postsynaptic suppressed
siℓ=1
for any magnitude
Cor. 1
Appendix
Table 5: The three consolidation channels opened by the synapse mask.
Substrate
Accuracy
Syn. ops
dense backpropagation MLP (autograd reference)
97.7±0.1
1.00
local burst-error learner, unconstrained
97.8±0.1
0.85
+ bounded error, Dale’s law, sign-concordant feedback (no transport)
97.1±0.0
0.85
+k -WTA 10%
97.0±0.2
0.80
+ relaxation T=5
97.0±0.1
0.80
+30% distance-dependent connectivity
96.5±0.2
0.24
Appendix
Table 6: Substrate ablation on i.i.d. MNIST ( 256 – 128 hidden core, three seeds; development phase, official test set). “Syn. ops” is the fraction of the dense MLP’s synaptic operations in one forward pass. Upper block: constraints added cumulatively; lower block: one ingredient replaced at its stage.
Figure 9: Rotation: price, trade-off, and gate. Left: on i.i.d. MNIST the novelty gate restores no-rotation accuracy. Middle: on split-MNIST rotation is required and the gate keeps it. Right: on split CIFAR-10 (grayscale) rotation is a gain on both axes.
Figure 10: K=200 mechanism triangulation. Isolated rotating replay beats the unmasked trickle, diversity helps only during wake, and the irreducible advantage of the night at tiny buffers is its interference-free window.
Figure 11: Isolation across replay batch sizes (held-out split). Left: split-MNIST accuracy against replay batch size for isolated replay (the full system) and for rotation with unmasked replay, every seed shown; at batch 2 no unmasked mean is drawn because three of six seeds fail severely (values at the axis floor). Right: the seed-paired margin of isolation with a 95% bootstrap interval at the batch sizes where the unmasked run does not collapse.
development set
held-out split 1
held-out split 2
always-on rotation (default)
92.7±0.3
91.6±0.3
91.8±0.4
static
94.7±0.3
94.1±0.2
94.1±0.6
bout-committed gate
92.3±0.2
91.8±0.3
92.0±0.6
rotation alone (unmasked replay)
92.1±0.5
91.5±0.3
91.8±0.6
static
94.7±0.1
93.7±0.3
93.7±0.3
silent mask (no rotation)
88.8±0.8
88.0±0.7
88.6±1.3
Appendix
Table 7: Development-set (official test split) numbers beside the held-out numbers and a second held-out split. Sequential accuracy, with the static axis on its own row where the paper reports it; six seeds on the reported split, three on the development set and on the second split. Lower blocks: single pass, and raw split CIFAR-10.
after task
T1
T2
T3
T4
T5
1
99.8
2
98.7
97.2
3
97.7
92.7
97.0
4
97.5
89.6
92.9
97.5
5
96.7
88.1
89.7
91.5
91.8
Appendix
Table 8: Per-task accuracy matrix of the headline configuration on split-MNIST (held-out split, mean over six seeds): row = after training task i , column = accuracy on task j . Forgetting F , the mean of the diagonal minus the last row over tasks 1 – 4 , is 6.4±0.6 ; the drop is concentrated on tasks 2 – 3 and the first task is retained.
One of the critical limitations of artificial neural networks is their lack of ability to continually learn: training on new tasks often leads to interference and forgetting of the previous ones. While several algorithms have been proposed to protect old memories from interference, they are typically applied during or immediately after each new episode of training. In contrast, humans and animals can learn continuously, acquiring multiple new memories during active learning before consolidating all of them into long-term storage. Here we show that multiple new tasks can be trained sequentially before an unsupervised sleep-like replay phase is applied to partially restore performance across all previously learned tasks. Our study further suggests that task-specific information remains resilient to new training but decays gradually as network is trained on new tasks. These findings point to novel principles for developing a broad range of continual learning AI solutions.
Anthony Bazhenov, Jean Erik Delanois, Giri P. Krishnan
Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA
Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting. Yet its generalization behavior is shaped by two coupled effects that existing analyses fold into a single hypothesis-level quantity: finite memory replaces each past distribution with an empirical proxy, and repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. We develop a layer-wise information-theoretic framework that separates these effects at every depth. Our main result decomposes the expected generalization gap into a replay-induced representation drift and an optimization-dependence term, the latter further resolved into stability, plasticity, interaction, and residual-coupling components. Two refinements make the framework operational. A Wasserstein relaxation of the drift term, valid under support mismatch, yields a depth-dependent drift--sensitivity trade-off whose minimizer identifies which interior layer to stabilize. An SGLD instantiation of the optimization term reduces it to a trajectory-level log-determinant budget, exposing a curvature-aware gradient-alignment statistic that serves as an online diagnostic of task-wise forgetting. Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.
Tieliang Gong, Zhongbo Zhang, Wen Wen +1
Tieliang Gong, Zhongbo Zhang and Wen Wen are with the School of Computer Science and Technology, Xi’an Jiaotong University, China · Yong-Jin Liu is with Department of Computer Science and Technology, Tsinghua University, China
Class-balanced replay controls class frequency but does not determine the interval between successive replay appearances of a class. We study this interval, the rehearsal gap, separately from the class marginal and class co-occurrence, and introduce randomised-pass replay (RPR), which visits each resident class once per shuffled pass. For a fixed set of C resident classes and replay batch size b less than or equal to C, RPR preserves the balanced time-averaged class marginal and bounds every gap by 2*ceil(C/b)-1; a churn-conditional bound applies while the resident set changes. The scheduler uses no future class information and adds no replay examples or forward passes. In a linear-head ER-ACE diagnostic, joint absence from the incoming and replay batches produces a one-sided classifier-bias gradient. Longer absence episodes are associated with larger negative bias displacement, and removing the incoming-loss mask attenuates the scheduling effect. In the primary ER-ACE experiments, RPR improves final average accuracy by 0.72-1.67 percentage points relative to independent class-balanced retrieval under reservoir storage, with positive effects also observed under balanced storage. Pretrained ViTs show positive effects on the tested LT10 streams with small replay batches, while matched larger-batch controls show no material effect. Fixed-cycle and reused-pass controls change more than one temporal statistic, so the experiments do not isolate rehearsal-gap length from all other forms of temporal dependence. The accuracy effects depend on the learner and operating regime.
Congren Dai, Nat Roongjirarat, Fei Ye
Imperial College London · King’s College London · University of Electronic Science and Technology of China