Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Fujie Gao; Zuyue Zhang; Gang Sun

cs.LGSep 30, 2026

Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Authors: Fujie Gao, Zuyue Zhang, Gang Sun

Abstract

Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.

Figures & tables

Panel of Figure 1 — Figure 1: Independent-feature signatures of RPF. Left: learned strength after $10^{5}$ updates exhibits the predicted soft support-dependent transition. Right: the number of sampled memories above a relative threshold grows sublinearly and approaches saturation, in qualitative agreement with the finite-time-to-steady-state picture. The expected-strength law $k^{*}\sim S^{1/\alpha}$ describes the accumulation-dominated regime. These batch-wise, finite-run observations support the predicted pattern of selective retention and growth toward a forgetting-limited ceiling.

Figure 2: Conditional retention and its reinforcement–forgetting phase diagram. Left: correct-input and same-rank control probabilities, averaged over focal patterns across all $140$ runs; the dashed line is the uniform label marginal. Right: tail focal retention ( $c_{s}\geq 0.5$ ), averaged over five seeds. Resampling moves associations into the retained region near the transition. The horizontal coordinate is the resampling weight, not the normalized exposure multiplier; the $8\times$ weight increases each focal pattern’s sampling probability by $3.448\times$ .

Figure 3: Timing changes the retained value of identical sample multisets. Top: focal probability trajectories with zero attenuation and with

d_{\rm step}=0.012

; curves show ten-seed means and shading shows standard deviations. Bottom left: the paired timing contrast in Eq. ( 23 ), with 95% bootstrap intervals across seeds. Bottom right: at

d_{\rm step}=0.012

, later focal replay increases focal correct-label probability while decreasing background correct-label probability; points are schedule means and bars are seed standard deviations. The intervention changes the composition of the retained set.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$d_{\rm step}$	$1\times$	$2\times$	$4\times$	$8\times$
0.000	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.003	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.006	$0.981\pm 0.028$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.009	$0.550\pm 0.084$	$0.925\pm 0.036$	$0.994\pm 0.014$	$1.000\pm 0.000$
0.012	$0.013\pm 0.017$	$0.175\pm 0.052$	$0.525\pm 0.124$	$0.819\pm 0.056$
0.015	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.013\pm 0.028$

Appendix

Table 1: Complete tail focal retention grid, mean

\pm

seed standard deviation. The retention criterion is

c_{s}\geq 0.5

; columns are focal resampling weights.

Criterion	$1\times$	$2\times$	$4\times$	$8\times$
0.4	0.200	0.706	0.969	1.000
0.5	0.013	0.175	0.525	0.819
0.6	0.000	0.013	0.069	0.125

Appendix

Table 2: Threshold sensitivity at

d_{\rm step}=0.012

. Tail focal retention means across five seeds. The intervention direction persists across all three criteria.

$d_{\rm step}$	Late–early	$C(d_{\rm step})$	95% interval
0.000	0.0186	0.0000	—
0.009	0.2185	0.1999	$[0.1929,\,0.2069]$
0.012	0.3081	0.2895	$[0.2814,\,0.2978]$
0.015	0.3195	0.3009	$[0.2934,\,0.3081]$

Appendix

Table 3: Matched schedule effects on mean focal conditional probability. The contrast subtracts each seed’s zero-attenuation late–early gap. Intervals are paired-seed bootstrap intervals.

AdamW decay coefficient	Focal gain	95% interval
$0$	$0.165$	$[0.136,0.199]$
$3$	$0.374$	$[0.267,0.479]$
$4$	$0.158$	$[0.094,0.222]$
$5$	$0.055$	$[0.052,0.058]$

Appendix

Table 4: Factorized synthetic task: paired gain in focal correct-label probability on held-out combinations for resampling weight

m=4

versus

m=1

(ten seeds; paired 95% bootstrap intervals).

Explore similar work

May 27, 2026cs.LG

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size

b=128

, test accuracy varies by

26.5

percentage points across initialization scales despite

\ge99.5\%

training accuracy. This is not undertraining: extending the same low-learning-rate regime to

5{,}000

epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit

L_2

norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2

MIT · Northwestern University

Neural Network Training Batch Forgetting Inductive Bias Spectral Preconditioning

Ask PDF

Jun 7, 2026cs.LG

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

One of the critical limitations of artificial neural networks is their lack of ability to continually learn: training on new tasks often leads to interference and forgetting of the previous ones. While several algorithms have been proposed to protect old memories from interference, they are typically applied during or immediately after each new episode of training. In contrast, humans and animals can learn continuously, acquiring multiple new memories during active learning before consolidating all of them into long-term storage. Here we show that multiple new tasks can be trained sequentially before an unsupervised sleep-like replay phase is applied to partially restore performance across all previously learned tasks. Our study further suggests that task-specific information remains resilient to new training but decays gradually as network is trained on new tasks. These findings point to novel principles for developing a broad range of continual learning AI solutions.

Anthony Bazhenov, Jean Erik Delanois, Giri P. Krishnan

Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA

Catastrophic Forgetting Continual Learning Forgetting New Tasks

Ask PDF

Mar 26, 2026stat.ML

The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction

1 - \varepsilon

of training labels is generated by a structured teacher rule, while a fraction

\varepsilon

consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regularization and the choice of kernel or nonlinearity control the allocation of capacity between rule learning and memorization. Experiments on a CIFAR10-based RAF task show that the qualitative trade-off persists for fixed kernels, whereas representations learned by deeper neural networks substantially mitigate it, preserving factual recall while improving generalization. The RAF model provides a theoretical foundation for understanding how modern neural networks can infer structure while storing rare or non-compressible information.

Gabriele Farné, Fabrizio Boncoraglio, Lenka Zdeborová

Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne

Memorization Singular Learning Theory Neural Network Improved Generalization Factual Recall Capacity

Ask PDF