Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Fujie Gao; Zuyue Zhang; Gang Sun

cs.LGSep 30, 2026

Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Authors: Fujie Gao, Zuyue Zhang, Gang Sun

Abstract

Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.

Figures & tables

Panel of Figure 1 — Figure 1: Independent-feature signatures of RPF. Left: learned strength after $10^{5}$ updates exhibits the predicted soft support-dependent transition. Right: the number of sampled memories above a relative threshold grows sublinearly and approaches saturation, in qualitative agreement with the finite-time-to-steady-state picture. The expected-strength law $k^{*}\sim S^{1/\alpha}$ describes the accumulation-dominated regime. These batch-wise, finite-run observations support the predicted pattern of selective retention and growth toward a forgetting-limited ceiling.

Figure 2: Conditional retention and its reinforcement–forgetting phase diagram. Left: correct-input and same-rank control probabilities, averaged over focal patterns across all $140$ runs; the dashed line is the uniform label marginal. Right: tail focal retention ( $c_{s}\geq 0.5$ ), averaged over five seeds. Resampling moves associations into the retained region near the transition. The horizontal coordinate is the resampling weight, not the normalized exposure multiplier; the $8\times$ weight increases each focal pattern’s sampling probability by $3.448\times$ .

Figure 3: Timing changes the retained value of identical sample multisets. Top: focal probability trajectories with zero attenuation and with

d_{\rm step}=0.012

; curves show ten-seed means and shading shows standard deviations. Bottom left: the paired timing contrast in Eq. ( 23 ), with 95% bootstrap intervals across seeds. Bottom right: at

d_{\rm step}=0.012

, later focal replay increases focal correct-label probability while decreasing background correct-label probability; points are schedule means and bars are seed standard deviations. The intervention changes the composition of the retained set.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$d_{\rm step}$	$1\times$	$2\times$	$4\times$	$8\times$
0.000	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.003	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.006	$0.981\pm 0.028$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$
0.009	$0.550\pm 0.084$	$0.925\pm 0.036$	$0.994\pm 0.014$	$1.000\pm 0.000$
0.012	$0.013\pm 0.017$	$0.175\pm 0.052$	$0.525\pm 0.124$	$0.819\pm 0.056$
0.015	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.013\pm 0.028$

Appendix

Table 1: Complete tail focal retention grid, mean

\pm

seed standard deviation. The retention criterion is

c_{s}\geq 0.5

; columns are focal resampling weights.

Criterion	$1\times$	$2\times$	$4\times$	$8\times$
0.4	0.200	0.706	0.969	1.000
0.5	0.013	0.175	0.525	0.819
0.6	0.000	0.013	0.069	0.125

Appendix

Table 2: Threshold sensitivity at

d_{\rm step}=0.012

. Tail focal retention means across five seeds. The intervention direction persists across all three criteria.

$d_{\rm step}$	Late–early	$C(d_{\rm step})$	95% interval
0.000	0.0186	0.0000	—
0.009	0.2185	0.1999	$[0.1929,\,0.2069]$
0.012	0.3081	0.2895	$[0.2814,\,0.2978]$
0.015	0.3195	0.3009	$[0.2934,\,0.3081]$

Appendix

Table 3: Matched schedule effects on mean focal conditional probability. The contrast subtracts each seed’s zero-attenuation late–early gap. Intervals are paired-seed bootstrap intervals.

AdamW decay coefficient	Focal gain	95% interval
$0$	$0.165$	$[0.136,0.199]$
$3$	$0.374$	$[0.267,0.479]$
$4$	$0.158$	$[0.094,0.222]$
$5$	$0.055$	$[0.052,0.058]$

Appendix

Table 4: Factorized synthetic task: paired gain in focal correct-label probability on held-out combinations for resampling weight

m=4

versus

m=1

(ten seeds; paired 95% bootstrap intervals).

Explore similar work

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network Training Batch
MIT · Northwestern University
Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks
Jun 7, 2026Anthony Bazhenov, Jean Erik Delanois, Giri P. KrishnanCatastrophic Forgetting Continual Learning
Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA
The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks
Mar 26, 2026Gabriele Farné, Fabrizio Boncoraglio, Lenka ZdeborováMemorization Singular Learning Theory
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne

May 27, 2026, cs.LGJ/K move · Enter open · S save

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey, Gareth H. McKinkey+1

MIT · Northwestern University

Neural Network Training Batch Forgetting Inductive Bias Spectral Preconditioning

Open paper AskPDF arXiv

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size

b=128

, test accuracy varies by

26.5

percentage points across initialization scales despite

\ge99.5\%

training accuracy. This is not undertraining: extending the same low-learning-rate regime to

5{,}000

epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit

L_2

norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

Abstract

Explore similar work

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks