Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation
Authors: Zafeirios Fountas, Adnan Oomerjee, Haitham Bou-Ammar, Jun Wang, Neil Burgess
Organizations: Huawei Noah’s Ark Lab, London, UK · AI Centre, Department of Computer Science, University College London, London, UK · UCL Institute of Cognitive Neuroscience, University College London, London, UK · UCL Queen Square Institute of Neurology, University College London, London, UK
Standard accounts of memory consolidation emphasise the stabilisation of stored representations, but struggle to explain representational drift, semanticisation, or the necessity of offline replay. Here we propose that high-capacity neocortical networks optimise stored representations for generalisation by reducing complexity via predictive forgetting, i.e. the selective retention of experienced information that predicts future outcomes or experience. We show that predictive forgetting formally improves information-theoretic generalisation bounds on stored representations. Under high-fidelity encoding constraints, such compression is generally unattainable in a single pass; high-capacity networks therefore benefit from temporally separated, iterative refinement of stored traces without re-accessing sensory input. We demonstrate this capacity dependence with simulations in autoencoder-based neocortical models, biologically plausible predictive coding circuits, and Transformer-based language models, and derive quantitative predictions for consolidation-dependent changes in neural representational geometry. These results identify a computational role for off-line consolidation beyond stabilisation, showing that outcome-conditioned compression optimises the retention-generalisation trade-off.
Figures & tables
Figure 1: Consolidation as predictive forgetting. a, Conceptual illustration using visual classification. Initial encoding (distributions Z0 ; lowercase variables denote specific samples) preserves both diagnostic features (cat-specific: red; dog-specific: blue; shared: yellow) and task-irrelevant details (background, lighting: green) to minimise sensory prediction error. Iterative consolidation ( Z1→Z5 ) progressively discards noise (shared and task-irrelevant features) while sharpening category-predictive features. b, Online Learning (Wake) acts as standard feedforward learning. The encoder module (parameters Φ ) maps inputs X to high-fidelity representations Z to predict targets Y . Training objectives that maximise task performance can drive both mutual information terms I(X;Z) and I(Z;Y) upwards; however, a high-capacity readout (parameters Ω ) minimises error by memorising the input-specific noise retained in Z , leading to overfitting. c, Consolidation through iterative latent refinement (modelled as an offline “sleep” phase, cf. the wake-sleep algorithm [ 32 ] ). Augmenting the encoder with an offline consolidator Ψ (applied N times) enables a different optimisation: I(X;Z) decreases as I(Z;Y) is maintained (or increases). By actively reducing input dependence, this process enforces an information bottleneck on the downstream readout Ω (formally constraining the Markov chain S→Z→Ω ), physically preventing the overfitting that plagues high-capacity single-pass systems and thus tightening the generalisation bound (Equation 1 ). Note that this schematic depicts the consolidation of the neocortical representation used for generalisation; the hippocampal system may additionally retain veridical episodic details not captured by the compressed code.
Figure 2: Iterative refinement tightens the generalisation bound. a, Architecture comprising a frozen convolutional encoder, an iterative latent refiner, and a task-dependent readout network. b-c, Effect of consolidation steps on classification accuracy (b) and the generalisation gap Δ (c) across five image classification benchmarks (MNIST, Fashion-MNIST, EMNIST, CIFAR-10, SVHN). Refinement consistently improves performance while shrinking Δ . Online-only regularisation baselines defining the fidelity-generalisation frontier for MNIST are shown in Extended Data Fig. 1 . d, Information-theoretic validation (information measured in nats, i.e. natural units). We computed proxies for the mutual information terms in Equation 1 . As predicted, consolidation reduces superfluous input dependence I(X;Zt) (gray) while increasing task-relevant information I(Y;Zt) (blue), confirming that the system implements predictive forgetting.
Figure 3: The Bidirectional Consolidation Mechanism. a, Wake Perception: Sensory input x is clamped (Lock icon), driving bottom-up inference. The network balances sensory fidelity against internal priors, resulting in a high-fidelity representation where input information I(x;z) is high (Orange). b, Sleep Consolidation: Sensory units are unclamped. A stored memory trace m (representing stable synaptic weights or buffer entries) is retrieved to initialise the transient neural activity z . The generative model projects this trace top-down to create a "dream" x^ (Cloud icon), which then drives a second pass of precision-weighted inference. The strong homeostatic prior actively minimises representational cost ( I(x;z)↓ , Red), effectively denoising the trace while preserving task-relevant information ( I(z;y) , Blue). c, Qualitative Results: Comparison of noisy wake inputs (Left) vs. consolidated sleep replays (Right) across four datasets. The generative loop filters out high-frequency sensory noise, extracting the semantic gist.
Figure 4: Consolidation resolves the capacity-generalisation trade-off. a, Generalisation Gap ( Δ ) vs. Representational Capacity ( d ) on CIFAR-10. In low-capacity regimes ( d<64 ), architectural bottlenecks force such strong compression that models underfit training noise, resulting a negligible or negative gap (driven by regularisation). However, as capacity scales ( d→1024 ), the Online agent (Orange) suffers catastrophic overfitting, memorising sensory noise. Offline Replay (Purple) drastically reduces this gap, allowing the system to scale to high capacities with a significantly minimised generalisation penalty. b, Test Accuracy. In low-capacity networks, replay provides minimal benefit as the architecture itself enforces compression. In high-capacity regimes ( d≥256 ), characteristic of mammalian neocortex, replay translates into a net performance gain. The Replay agent (Purple) outperforms the Online agent because the sleep phase applies stronger internal constraints (priors) that are unavailable during rapid online perception. This active filtering removes the input-specific noise that high-capacity networks otherwise memorise, allowing the system to leverage its full capacity for semantic discrimination. c, Mechanism of Compression ( d=512 ). Latent norm distributions reveal the physical basis of this effect: strong homeostatic pressure during sleep contracts memory traces toward the generative manifold (Purple), explicitly reducing representational cost I(X;Z) compared to the high-entropy wake state (Orange).
Figure 5: Hierarchical and iterative cache refinement in large language models. a, Schematic of the Bottlenecked Transformer as a model of systems consolidation. A frozen Llama-3-8B backbone (right, lock icon) representing the stable neocortex interacts with a plastic Key-Value store whose entries are iteratively consolidated (left). During offline “sleep” phases, a Refiner network ( ψ ) iteratively updates cached representations to minimise conditional mutual information. b, Generalisation gap (train vs. test perplexity) on GSM8K reasoning tasks. Offline refinement ( Purple ) prevents the overfitting observed when fine-tuning the baseline online model ( Orange ). c, Step-wise mean cosine distance, d(zt,zt+1) , between cache states before and after each refinement step, for Keys (Grey) and Values (Purple) . Values undergo large initial corrections that diminish but persist across steps, indicating ongoing refinement without convergence to a fixed point. Keys remain nearly unchanged throughout, mirroring the biological distinction between stable structural addressing and malleable content [ 48 ] . d, Principal Component Analysis (PCA) of value update trajectories for a single example, coloured by refinement step (Purple=initial → Yellow=final). Explained Variance Ratios (EVR) for the first two components are shown for each layer. Layer 3 (left) exhibits coherent, parallel flow indicative of Global Renormalisation, whereas Layer 14 (right) shows divergent, complex trajectories indicative of Selective Editing. e, Global Update Fraction across layers ( N=1318 ). Updates transition from a shared global translation regime ( >0.8 ) in early layers to a selective editing regime ( <0.5 ) in deep layers. Grey shading indicates population variability ( ±1 s.d.); blue error bars indicate bootstrap uncertainty ( 95%CI ). f, Evolution of update coherence over refinement steps. Early layers ( Red , Layer 3) maintain global coherence, while deep layers ( Blue , Layer 14) progressively become more selective, indicating a coarse-to-fine refinement strategy.
- Compression ( I(X;Z)↓ ) accelerates during sleep vs. wake
Sleep manipulation, fMRI reactivation
Grounding
[ 56 , 57 ]
Generalisation Link
- Neural compression predicts OOD performance - Selective for task-irrelevant features
Behavioural transfer tests, individual differences
Grounding
[ 58 , 59 ]
Readout Sharpening
- Increased SNR in memory traces - Synaptic down-selection of noise
Longitudinal 7T fMRI, E-phys
Grounding
[ 60 , 61 ]
Predictive Codes
- Shift from retrospective to prospective Successor Representations
Sequential learning tasks
Unification
[ 34 , 62 , 55 , 63 ]
Hierarchical Gradients
- Compression magnitude scales linearly with cortical hierarchy
Multi-region fMRI
Prediction
Partial: [ 64 , 65 ]
Table 1: Predictions and Empirical Support for Predictive Forgetting
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Extended Data Fig. 1: The fidelity-generalisation frontier. To rigorously test whether the benefits of consolidation could be replicated by simply increasing regularisation during online training, we compared our Offline Replay model (Purple Star) against two single-pass baselines on the MNIST dataset. All models utilised the same convolutional architecture ( d=64 ) described in Section 3.1 to ensure a fair comparison of capacity. First, we trained an online agent with a Variational Information Bottleneck (VIB) objective (Gray circles), sweeping the regularisation strength β∈{10−4,…,100} and performing a fine-grained sweep around the optimal region ( β∈{0.08,…,0.11} ) to identify maximum performance. Second, we trained an online agent with dropout probabilities p∈{0.2,…,0.8} (Blue squares). The baselines define a convex “fidelity-generalisation frontier” (dashed lines), representing the unavoidable trade-off between minimising the gap and maintaining accuracy during single-pass learning. The Offline Sleep model lies beyond this frontier in the top-left quadrant. Data points represent mean ± standard deviation across n=50 independent seeds. This confirms that iterative offline refinement offers a computational advantage that cannot be recovered by online regularisation alone.
Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity. These low-dimensional representations have improved the interpretability of population-level coding. Yet whether such low-dimensional representations are biologically relevant and confer functional advantages in learning systems, or merely reflect neuron-level activity, remains contested in neuroscience. We show that an explicit information bottleneck forcing a recurrent neural network to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalisation in a time-series prediction task. Using information-theoretic measures of causal emergence, we characterise the dynamics of this representation across the memorisation-to-generalisation transition, finding a non-monotonic trajectory which shows an initial decrease, a minimum, and a subsequent rise to a maximum, even as prediction loss falls monotonically. This trajectory scales with task complexity, and the magnitude of emergent structure reliably predicts generalisation performance. Analysis of CA1 hippocampal activity in mice learning an alternating maze task reveals analogous non-monotonic emergence dynamics that track behavioural performance. Together, these findings indicate that the ability of neural networks to learn compact, distributed and emergent representations confers a functional advantage for generalisation, supporting a causal role for learned representations in cognition.
Hardik Rajpal, Dan Goodman
1I-X Centre for AI in Science, Imperial College London, W12 0BZ, UK · Department of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ, UK
Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.
Memory consolidation determines both what a learner can do now and which changes remain implementable later. We develop a finite-model synthesis of operational state abstraction and optimal control under the stability-evidence-revision (SER) framework. ``Poincaré meets Bellman'' names two complementary roles: qualitative dynamics identifies reusable action-response structure, and dynamic programming prices acquisition, retention, reuse, merging, and forgetting. Recurrence enters separately through the timing and value of future demands. We distinguish active quotient merging from historical information erasure, characterize exact repair by zero-error functional coding and causal migration, and derive a Bellman recursion over the joint law of hidden state and complete deployed memory. A first-return model yields an explicit retention rule. Conditional results show how factor sharing avoids enumerating combinations and how independent informative observations improve identification, while leaving some zero-error evidence budgets unchanged. Finite enumerations verify the coding and retention calculations. The synthesis gives an exact benchmark for specified finite models, without claiming universal recurrence, bounded-memory open-ended learning, or tractable global planning.
Xin Li
Department of Computer Science University at Albany