Fisher information and the geometry of memorization in neural networks
Authors: Daniel T. Bernstein, Chase Goddard, Luca Di Carlo, Vudtiwat Ngampruetikorn, David J. Schwab
Organizations: Princeton University1; Lewis-Sigler Institute · Princeton University2; Joseph Henry Laboratories · University of Sydney3; School of Physics · CUNY Graduate Center4; Initiative for the Theoretical Sciences
A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.
Figures & tables
Figure 1: Layer-dependent decoupling of memorization and generalization via Fisher projection: memorized Gaussian images. Classification accuracy as a function of the number of retained top Fisher eigenvectors k , computed over the full training set (CIFAR-10 and Gaussian images) for a single layer of a ResNet-20. Green: CIFAR-10 test; blue: CIFAR-10 train; red: memorized Gaussian samples. A ) In the first layer, CIFAR-10 train and test accuracy rise in tandem and reach a maximum at k≈45 , while accuracy on memorized Gaussian samples remains at ≈30% , demonstrating selective de-memorization at negligible cost to generalization. B ) In the last layer, all three curves rise sharply even for very small k , with no separation between them at any point, indicating that memorized and structured representations have merged and can no longer be separated.
Figure 2: Layer-dependent decoupling of memorization and generalization via Fisher projection: noisy labels. Classification accuracy as a function of retained top Fisher eigenvectors k for a ResNet-20 trained on CIFAR-10 with 10% label noise. Blue: train accuracy; Green: test accuracy; Red: accuracy on poisoned samples under their (wrong) training labels; Orange: accuracy on poisoned samples under their true labels. A&B) Fisher computed on clean (non-corrupted) data only. C&D) Fisher computed on the full training set. First layer , A&C) : top Fisher eigenspaces treat memorized and clean samples differently under both Fisher variants. Penultimate convolutional layer , B) : with the clean Fisher, de-memorization is successful — the red curve has not reached its maximum by the time train and test accuracy do — and the model partially recovers the true labels of label-flipped samples, with the orange curve rising well above chance. Penultimate convolutional layer , D) : with the full Fisher, decoupling becomes weaker, highlighting the role of dataset choice in computing the Fisher.
Figure 3: Memorization and generalization across layers. Results of using a block-diagonal approximation to estimate the Fisher information across all model parameters in the noisy-label setting. A) The Fisher is computed on training data with clean labels only. The top eigenspace selectively captures information about the true classification task. We also see non-monotonic behavior in the performance on the true labels of memorized samples (orange dashed line). B) The Fisher is computed on a sample from the full training dataset, including samples with corrupted labels. In this case, top eigenspaces mix information across tasks, emphasizing the data-dependence of the Fisher.
Figure 4: From memorization to generalization, in-context. A) Transient ICL training dynamics. The three-block transformer first improves performance on held-out tasks, before overfitting to training tasks. B) Fisher projections for the attention layer, second transformer block. All projections decrease performance. Shaded region shows standard deviation across 16 models. C) Fisher projections for the MLP layer, second transformer block. Top Fisher subspaces improve task generalization (green) at the cost of performance on memorized tasks (blue).
Figure 5: Single-sample contributions and projections. Participation ratios for A) input-noise case and B) label-noise case. A) suggests that structured CIFAR images primarily contribute to the top of the eigenspectrum, while noisy samples primarily contribute to the bottom of the eigenspectrum. This separation does not occur in B) . C) We plot the self-hit rate of correct sample classification after projecting out the top-k subspace of the corresponding single-sample Fisher information (‘sample projection’) in the input-noise case, and D) shows the same statistic for the label-noise case. Noisy samples are less likely to be recovered than their counterparts.
Gaussian model
Noisy-label model (10% noise)
k=0
1
2
4
9
k=0
1
2
4
9
Fisher source CIFAR-10 (%)
Fisher source uncorrupted (%)
CIFAR-10 train (5k samples)
100.0
99.9
99.9
99.9
99.8
CIFAR-10 train (5k samples)
100.0
100.0
100.0
99.9
99.9
CIFAR-10 test
84.3
83.9
83.7
83.4
83.1
CIFAR-10 test
73.4
73.1
73.0
72.7
72.3
Fisher source Gaussian (%)
Fisher source corrupted (%)
Gaussian set
100.0
98.4
96.1
90.5
75.6
Corrupted CIFAR (assigned labels)
100.0
100.0
100.0
99.9
99.8
Table 1: Bulk accuracy after sample projection. The k=0 column is the matched unchanged-weight BatchNorm control (see Appendix D for results on other Fisher sources).
Figure 6: Finite-rank linear model illustrates the risk gain from Fisher projection. A) Generative model. A latent sign vector u is drawn uniformly from {−1,+1}r ; anisotropic scaling by amplitude vectors μ and a produces signal features s and clean label y , respectively. The observed features x result from concatenating the signal s and the uninformative barcode features z , modeled as an isotropic Gaussian vector in d dimensions. The training label y~ is the clean label y with random per-dimension sign flips, with a random sign vector η . B) Risk gain from removing the barcode block post hoc vs label flip probability Pflip , demonstrating quantitative agreement between theory (solid curve, Eq ( 7 )) and simulations (circles; error bars are smaller than the circle size). C) Prediction MSE vs retained Fisher rank for three flip rates (see legend). Two of four signal dimensions have population variance below the spectral separation threshold and therefore mix with the barcode bulk. In both theoretical prediction (solid curve, Eq ( 22 )) and simulations (circles), an interior optimum emerges as the flip rate increases. Shown results are for (n,r,d)=(500,4,250) with 200 replicates.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Layerwise Fisher projection experiments for the input-noise case.
Figure 8: Memorization concentrates in a single MLP. Here, we show all transformer blocks for the model considered in Fig 4 . All layers except the Block 2 MLP perform worse on both train/test tasks when restricted to a top Fisher subspace, but in the Block 2 MLP test performance improves at the cost of performance on the training tasks.
Figure 9: Fisher projections for every layer, in the noisy sample setting. Fisher computed on clean samples.
Figure 10: Fisher projections for every layer, in the noisy sample setting. Fisher computed on a subset of the full training set.
Figure 11: Control experiments for A)-C) SVD compression and D)-F) random basis projections.
Gaussian model
Noisy-label model (10% noise)
k=0
k=1
k=2
k=4
k=9
k=0
k=1
k=2
k=4
k=9
Fisher source CIFAR-10
Fisher source uncorrupted
CIFAR-10 train (5k samples)
100.0
99.9
99.9
99.9
99.8
CIFAR-10 train (5k samples)
100.0
100.0
100.0
99.9
99.9
CIFAR-10 test
84.3
83.9
83.7
83.4
83.1
CIFAR-10 test
73.4
73.1
73.0
72.7
72.3
Gaussian set
100.0
100.0
100.0
100.0
99.9
Corrupted CIFAR (assigned labels)
100.0
100.0
100.0
99.9
99.8
Corrupted CIFAR (true labels)
0.0
0.0
0.0
0.0
0.1
Appendix
Table 2: Bulk accuracy after sample projection. All accuracies are percentages. BatchNorm is reset after each projection and recalibrated by cumulative averaging on equal batches of 250 images. The k=0 column is the matched unchanged-weight BatchNorm control.
Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.
A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction 1−ε of training labels is generated by a structured teacher rule, while a fraction ε consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regularization and the choice of kernel or nonlinearity control the allocation of capacity between rule learning and memorization. Experiments on a CIFAR10-based RAF task show that the qualitative trade-off persists for fixed kernels, whereas representations learned by deeper neural networks substantially mitigate it, preserving factual recall while improving generalization. The RAF model provides a theoretical foundation for understanding how modern neural networks can infer structure while storing rare or non-compressible information.
Deep networks trained on noisy labels simultaneously generalize on clean data and memorize flipped labels. These are usually conflated as pressures on one capacity. We present Topo^2, a measurement framework that makes them causally separable, measurable, and law-governed. Persistent-homology H1 structure of the representation space separates into a within-class manifold channel (a function of the training stopping point) and a cross-class channel (a monotone readout of memorized flipped samples). An intervention, the FM0 prescription (zero loss on flipped samples from epoch 0), reaches each setting's generalization ceiling while memorizing essentially nothing. Within the framework we establish a law set with graded evidence: (L2) FM0 separation prescription (9/9); (L1) the within-channel as a training-position function (mid-rise 6/6; convergence-back CIFAR 3/3, SVHN 2/3); (L3) a ring-construction identity (definitional, not a law); and TLS (memory-generalization topological layering): memory is causally additive, anchored (silencing clean collapses the representation), invertible (stripping memory restores near-ceiling generalization), and quantitatively billable (the memorization cost law, effective slope coefficient C ~ 0.38 at the reference capacity: CIFAR-10 0.3801 / SVHN 0.3806 / CIFAR-100 0.384 / VGG 0.3715, capacity-dependent in general and traced to clean-sample feature displacement). We also publish the framework's boundaries: a falsification ledger of nine dead ends, and an instrument-vindication section that excludes six families of global statistics as explanations of the within-channel. The framework turns "memorization" from an ill-defined capacity into a measurable, separable, invertible topological layer.
Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investigate the underlying mechanisms of this coexistence using modular arithmetic tasks under heavy label noise. Through extensive experiments on two-layer neural networks, we find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data. Over-parameterized models internally form a generalization structure, but its expression in the output is suppressed by the need to fit noisy labels. Remarkably, even with 80% label noise, near-perfect test accuracy can be achieved by extracting this internal structure using frequency-based methods. We further propose a task-agnostic method to partition networks into generalization and memorization components. Although this subnetwork improves generalization, it is limited compared with frequency-based extraction, indicating that the generalization structure is distributed across neurons and motivating the development of new tools to retrieve generalizable knowledge from over-parameterized networks.
Linyu Liu, Pinyan Lu
1Taylor Lab, Huawei Technologies Co., Ltd. · 2Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics