Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation

Zafeirios Fountas; Adnan Oomerjee; Haitham Bou-Ammar; Jun Wang; Neil Burgess

q-bio.NCMar 5, 2026

Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation

Authors: Zafeirios Fountas, Adnan Oomerjee, Haitham Bou-Ammar, Jun Wang, Neil Burgess

Organizations: Huawei Noah’s Ark Lab, London, UK · AI Centre, Department of Computer Science, University College London, London, UK · UCL Institute of Cognitive Neuroscience, University College London, London, UK · UCL Queen Square Institute of Neurology, University College London, London, UK

Abstract

Standard accounts of memory consolidation emphasise the stabilisation of stored representations, but struggle to explain representational drift, semanticisation, or the necessity of offline replay. Here we propose that high-capacity neocortical networks optimise stored representations for generalisation by reducing complexity via predictive forgetting, i.e. the selective retention of experienced information that predicts future outcomes or experience. We show that predictive forgetting formally improves information-theoretic generalisation bounds on stored representations. Under high-fidelity encoding constraints, such compression is generally unattainable in a single pass; high-capacity networks therefore benefit from temporally separated, iterative refinement of stored traces without re-accessing sensory input. We demonstrate this capacity dependence with simulations in autoencoder-based neocortical models, biologically plausible predictive coding circuits, and Transformer-based language models, and derive quantitative predictions for consolidation-dependent changes in neural representational geometry. These results identify a computational role for off-line consolidation beyond stabilisation, showing that outcome-conditioned compression optimises the retention-generalisation trade-off.

Figures & tables

Figure 1: Consolidation as predictive forgetting. a, Conceptual illustration using visual classification. Initial encoding (distributions $Z_{0}$ ; lowercase variables denote specific samples) preserves both diagnostic features (cat-specific: red; dog-specific: blue; shared: yellow) and task-irrelevant details (background, lighting: green) to minimise sensory prediction error. Iterative consolidation ( $Z_{1}\to Z_{5}$ ) progressively discards noise (shared and task-irrelevant features) while sharpening category-predictive features. b, Online Learning (Wake) acts as standard feedforward learning. The encoder module (parameters $\Phi$ ) maps inputs $X$ to high-fidelity representations $Z$ to predict targets $Y$ . Training objectives that maximise task performance can drive both mutual information terms $I(X;Z)$ and $I(Z;Y)$ upwards; however, a high-capacity readout (parameters $\Omega$ ) minimises error by memorising the input-specific noise retained in $Z$ , leading to overfitting. c, Consolidation through iterative latent refinement (modelled as an offline “sleep” phase, cf. the wake-sleep algorithm [ 32 ] ). Augmenting the encoder with an offline consolidator $\Psi$ (applied $N$ times) enables a different optimisation: $I(X;Z)$ decreases as $I(Z;Y)$ is maintained (or increases). By actively reducing input dependence, this process enforces an information bottleneck on the downstream readout $\Omega$ (formally constraining the Markov chain $S\to Z\to\Omega$ ), physically preventing the overfitting that plagues high-capacity single-pass systems and thus tightening the generalisation bound (Equation 1 ). Note that this schematic depicts the consolidation of the neocortical representation used for generalisation; the hippocampal system may additionally retain veridical episodic details not captured by the compressed code.

Figure 2: Iterative refinement tightens the generalisation bound. a, Architecture comprising a frozen convolutional encoder, an iterative latent refiner, and a task-dependent readout network. b-c, Effect of consolidation steps on classification accuracy (b) and the generalisation gap $\Delta$ (c) across five image classification benchmarks (MNIST, Fashion-MNIST, EMNIST, CIFAR-10, SVHN). Refinement consistently improves performance while shrinking $\Delta$ . Online-only regularisation baselines defining the fidelity-generalisation frontier for MNIST are shown in Extended Data Fig. 1 . d, Information-theoretic validation (information measured in nats, i.e. natural units). We computed proxies for the mutual information terms in Equation 1 . As predicted, consolidation reduces superfluous input dependence $I(X;Z_{t})$ (gray) while increasing task-relevant information $I(Y;Z_{t})$ (blue), confirming that the system implements predictive forgetting.

Figure 3: The Bidirectional Consolidation Mechanism. a, Wake Perception: Sensory input $x$ is clamped (Lock icon), driving bottom-up inference. The network balances sensory fidelity against internal priors, resulting in a high-fidelity representation where input information $I(x;z)$ is high (Orange). b, Sleep Consolidation: Sensory units are unclamped. A stored memory trace $m$ (representing stable synaptic weights or buffer entries) is retrieved to initialise the transient neural activity $z$ . The generative model projects this trace top-down to create a "dream" $\hat{x}$ (Cloud icon), which then drives a second pass of precision-weighted inference. The strong homeostatic prior actively minimises representational cost ( $I(x;z)\downarrow$ , Red), effectively denoising the trace while preserving task-relevant information ( $I(z;y)$ , Blue). c, Qualitative Results: Comparison of noisy wake inputs (Left) vs. consolidated sleep replays (Right) across four datasets. The generative loop filters out high-frequency sensory noise, extracting the semantic gist.

Figure 4: Consolidation resolves the capacity-generalisation trade-off. a, Generalisation Gap (

\Delta

) vs. Representational Capacity (

d

) on CIFAR-10. In low-capacity regimes (

d<64

), architectural bottlenecks force such strong compression that models underfit training noise, resulting a negligible or negative gap (driven by regularisation). However, as capacity scales (

d\to 1024

), the Online agent (Orange) suffers catastrophic overfitting, memorising sensory noise. Offline Replay (Purple) drastically reduces this gap, allowing the system to scale to high capacities with a significantly minimised generalisation penalty. b, Test Accuracy. In low-capacity networks, replay provides minimal benefit as the architecture itself enforces compression. In high-capacity regimes (

d\geq 256

), characteristic of mammalian neocortex, replay translates into a net performance gain. The Replay agent (Purple) outperforms the Online agent because the sleep phase applies stronger internal constraints (priors) that are unavailable during rapid online perception. This active filtering removes the input-specific noise that high-capacity networks otherwise memorise, allowing the system to leverage its full capacity for semantic discrimination. c, Mechanism of Compression (

d=512

). Latent norm distributions reveal the physical basis of this effect: strong homeostatic pressure during sleep contracts memory traces toward the generative manifold (Purple), explicitly reducing representational cost

I(X;Z)

compared to the high-entropy wake state (Orange).

Figure 5: Hierarchical and iterative cache refinement in large language models. a, Schematic of the Bottlenecked Transformer as a model of systems consolidation. A frozen Llama-3-8B backbone (right, lock icon) representing the stable neocortex interacts with a plastic Key-Value store whose entries are iteratively consolidated (left). During offline “sleep” phases, a Refiner network ( $\psi$ ) iteratively updates cached representations to minimise conditional mutual information. b, Generalisation gap (train vs. test perplexity) on GSM8K reasoning tasks. Offline refinement ( Purple ) prevents the overfitting observed when fine-tuning the baseline online model ( Orange ). c, Step-wise mean cosine distance, $d(z_{t},z_{t+1})$ , between cache states before and after each refinement step, for Keys (Grey) and Values (Purple) . Values undergo large initial corrections that diminish but persist across steps, indicating ongoing refinement without convergence to a fixed point. Keys remain nearly unchanged throughout, mirroring the biological distinction between stable structural addressing and malleable content [ 48 ] . d, Principal Component Analysis (PCA) of value update trajectories for a single example, coloured by refinement step (Purple=initial $\to$ Yellow=final). Explained Variance Ratios (EVR) for the first two components are shown for each layer. Layer 3 (left) exhibits coherent, parallel flow indicative of Global Renormalisation, whereas Layer 14 (right) shows divergent, complex trajectories indicative of Selective Editing. e, Global Update Fraction across layers ( $N=1318$ ). Updates transition from a shared global translation regime ( $>0.8$ ) in early layers to a selective editing regime ( $<0.5$ ) in deep layers. Grey shading indicates population variability ( $\pm 1$ s.d.); blue error bars indicate bootstrap uncertainty ( $95\%CI$ ). f, Evolution of update coherence over refinement steps. Early layers ( Red , Layer 3) maintain global coherence, while deep layers ( Blue , Layer 14) progressively become more selective, indicating a coarse-to-fine refinement strategy.

Mechanism	Key Neural Signature	Methods to Test	Status	Evidence
Temporal Compression	- Manifold radius/dimension reduction - Increased within-category similarity	fMRI RSA, Manifold Capacity Analysis	Grounding	[ 54 , 55 ]
Sleep Acceleration	- Compression ( $I(X;Z)\downarrow$ ) accelerates during sleep vs. wake	Sleep manipulation, fMRI reactivation	Grounding	[ 56 , 57 ]
Generalisation Link	- Neural compression predicts OOD performance - Selective for task-irrelevant features	Behavioural transfer tests, individual differences	Grounding	[ 58 , 59 ]
Readout Sharpening	- Increased SNR in memory traces - Synaptic down-selection of noise	Longitudinal 7T fMRI, E-phys	Grounding	[ 60 , 61 ]
Predictive Codes	- Shift from retrospective to prospective Successor Representations	Sequential learning tasks	Unification	[ 34 , 62 , 55 , 63 ]
Hierarchical Gradients	- Compression magnitude scales linearly with cortical hierarchy	Multi-region fMRI	Prediction	Partial: [ 64 , 65 ]

Table 1: Predictions and Empirical Support for Predictive Forgetting

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Extended Data Fig. 1: The fidelity-generalisation frontier. To rigorously test whether the benefits of consolidation could be replicated by simply increasing regularisation during online training, we compared our Offline Replay model (Purple Star) against two single-pass baselines on the MNIST dataset. All models utilised the same convolutional architecture (

d=64

) described in Section 3.1 to ensure a fair comparison of capacity. First, we trained an online agent with a Variational Information Bottleneck (VIB) objective (Gray circles), sweeping the regularisation strength

\beta\in\{10^{-4},\dots,10^{0}\}

and performing a fine-grained sweep around the optimal region (

\beta\in\{0.08,\dots,0.11\}

) to identify maximum performance. Second, we trained an online agent with dropout probabilities

p\in\{0.2,\dots,0.8\}

(Blue squares). The baselines define a convex “fidelity-generalisation frontier” (dashed lines), representing the unavoidable trade-off between minimising the gap and maintaining accuracy during single-pass learning. The Offline Sleep model lies beyond this frontier in the top-left quadrant. Data points represent mean

\pm

standard deviation across

n=50

independent seeds. This confirms that iterative offline refinement offers a computational advantage that cannot be recovered by online regularisation alone.

Component	Hyper-parameter	Default
Data	batch size	128
Encoder/Decoder (P1)	epochs / lr / wd / patience	20 / $1\!\times\!10^{-3}$ / $1\!\times\!10^{-4}$ / 5
Readout (P2)	epochs / lr / wd / patience	15 / $5\!\times\!10^{-4}$ / $5\!\times\!10^{-4}$ / 5
Readout (P2)	hidden / dropout	128 / 0.1
Refiner (P3, fixed/ponder)	$T$ (train)	3
Refiner (P3)	ref hidden	256

Appendix

Table 2: Key hyper-parameters (defaults).

Memory Consolidation

Memory consolidation studies how systems select, compress, reorganize, and retain experience for reliable future learning and recall.

See all 34 papers

Emergent Generalization by Representation Learning in Artificial Neural Networks
Jul 11, 2026Hardik Rajpal, Dan GoodmanNeural Network Generalization Representation Learning
1I-X Centre for AI in Science, Imperial College London, W12 0BZ, UK · Department of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ, UK
Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics
Sep 30, 2026Fujie Gao, Zuyue Zhang, Gang SunNeural Network Memorization Neural Network Training Dynamics
Poincaré Meets Bellman: Revisable Memory, Operational Quotients, and Evidence-Supported Learning in Changing Environments
Nov 28, 2025Xin LiMarkov Decision Processes Dynamic Programming
Department of Computer Science University at Albany

Jul 11, 2026J/K move · Enter open · S save

Emergent Generalization by Representation Learning in Artificial Neural Networks

Hardik Rajpal, Dan Goodman

1I-X Centre for AI in Science, Imperial College London, W12 0BZ, UK · Department of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ, UK

Neural Network Generalization Representation Learning Information Bottleneck Recurrent Neural Networks Neural Encoding

Open paper AskPDF arXiv

Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity. These low-dimensional representations have improved the interpretability of population-level coding. Yet whether such low-dimensional representations are biologically relevant and confer functional advantages in learning systems, or merely reflect neuron-level activity, remains contested in neuroscience. We show that an explicit information bottleneck forcing a recurrent neural network to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalisation in a time-series prediction task. Using information-theoretic measures of causal emergence, we characterise the dynamics of this representation across the memorisation-to-generalisation transition, finding a non-monotonic trajectory which shows an initial decrease, a minimum, and a subsequent rise to a maximum, even as prediction loss falls monotonically. This trajectory scales with task complexity, and the magnitude of emergent structure reliably predicts generalisation performance. Analysis of CA1 hippocampal activity in mice learning an alternating maze task reveals analogous non-monotonic emergence dynamics that track behavioural performance. Together, these findings indicate that the ability of neural networks to learn compact, distributed and emergent representations confers a functional advantage for generalisation, supporting a causal role for learned representations in cognition.

Abstract

Memory Consolidation

Explore similar work

Emergent Generalization by Representation Learning in Artificial Neural Networks

Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Poincaré Meets Bellman: Revisable Memory, Operational Quotients, and Evidence-Supported Learning in Changing Environments