On the Dynamics & Transferability of Latent Generalization during Memorization

Simran Ketha; Venkatakrishnan Ramaswamy

cs.LGMar 20, 2026

On the Dynamics & Transferability of Latent Generalization during Memorization

Authors: Simran Ketha, Venkatakrishnan Ramaswamy

Organizations: Anuradha & Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology & Science Pilani, Hyderabad 500078, India.

Abstract

Deep networks have been known to have extraordinary generalization abilities, via mechanisms that aren't yet well understood. It is also known that upon shuffling labels in the training data to varying degrees, deep networks, trained with standard methods, can still achieve perfect or high accuracy on this corrupted training data. This phenomenon is called memorization, and typically comes at the cost of poorer generalization to true labels. Our recent work has demonstrated, that the internal representations of such models retain significantly better latent generalization abilities than is directly apparent from the model. In particular, it has been shown that such latent generalization can be recovered via simple probes (called MASC probes) on the layer-wise representations of the model. However, the origin and dynamics over training of this latent generalization during memorization is not well understood. Here, we track the training dynamics, empirically, and find that latent generalization abilities largely peak early in training, with model generalization. Next, we investigate to what extent the specific nature of the MASC probe is critical for our ability to extract latent generalization from the model's layerwise outputs. To this end, we first examine the mathematical structure of the MASC probe and show that it is a quadratic classifier, i.e. is non-linear. This brings up the question of the extent to which this latent generalization might be linearly decodable from layerwise outputs. To investigate this, we designed a new linear probe for this setting. Next, we consider the question of whether it is possible to transfer latent generalization to model generalization by directly editing model weights. To this end, we devise a way to transfer the latent generalization present in last-layer representations to the model using the new linear probe.

Figures & tables

Figure 1: Minimum Angle Subspace Classifier (MASC) test accuracy over epochs of training for multiple models/datasets, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data with the indicated label corruption degree. The plots display MASC accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown. FC denotes fully connected layers with

ReLU

activation, and Flat refers to the flatten layer without

ReLU

Figure 2: Vector Linear Probe Intermediate-layer Classifier (VeLPIC) test accuracy during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown. FC denotes fully connected layers with

ReLU

activation, and Flat refers to the flatten layer without

ReLU

Figure 3: Difference in test accuracy (VeLPIC Accuracy - MASC Accuracy) during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 4: Logistic regression probe’s test accuracy over epochs of training for multiple models/datasets. The plots display logistic regression probe’s accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown.

Figure 5: Difference in test accuracy (MASC Accuracy - Logistic regression probe Accuracy) during training of the network, where for MASC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 6: Difference in test accuracy (VELPIC Accuracy - Logistic regression probe Accuracy) during training of the network, where for VELPIC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 7: Model test accuracy when the weight intervention is applied to the epoch in question during training. The test accuracy of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison.

Figure 8: Model test accuracy with true labels during training when a weight intervention that involves replacing pre-softmax weights with the VeLPIC vector, is performed at the 40th epoch and standard training is performed thereafter for 60 epochs. The corresponding test accuracies for the model with standard training (dotted) without intervention is overlaid for comparison. The results correspond to a single run in each case.

Figure 9: Vector Linear Probe Intermediate layer Classifier (VeLPIC) test accuracy during training on models when intervention is performed at 40th epoch and standard training is performed thereafter for 60 epochs. The VeLPIC test accuracy on models trained with standard training (dotted) without intervention is overlaid for comparison. Model test accuracy for models trained with and without intervention is overlaid for comparison. The results correspond to a single run in each case.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Minimum Angle Subspace Classifier (MASC) Test accuracy for 0% and 100% corruption degrees during training of the network, where test data is projected onto class-specific subspaces constructed from training data with the indicated label corruption degrees. The plots display MASC accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (dotted line) is also shown. Each row corresponds to a specific corruption degree, while columns represent different models, as labeled. FC denotes fully connected layers with

ReLU

activation, and Flat refers to the flatten layer without

ReLU

Figure 11: Vector Linear Probe Intermediate-layer Classifier (VeLPIC) test accuracy for 0% and 100% corruption degrees during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown. FC denotes fully connected layers with

ReLU

activation, and Flat refers to the flatten layer without

ReLU

Figure 12: Difference in test accuracy (VeLPIC Accuracy - MASC Accuracy) during training of the network, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 13: Logistic regression probe’s test accuracy over epochs of training for multiple models/datasets. The plots display logistic regression probe’s accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown.

Figure 14: Difference in test accuracy (MASC Accuracy - Logistic regression probe Accuracy) during training of the network, where for MASC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 15: Difference in test accuracy (VELPIC Accuracy - Logistic regression probe Accuracy) during training of the network, where for VELPIC test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display difference in accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown, which would be 0.

Figure 16: Comparing model test accuracy with VeLPIC transferred accuracy when the weight intervention is applied to the model at the epoch in question during training for corruption degrees 0% and 100%. The test accuracy of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison.

Figure 17: Model train accuracy on corrupted dataset when the VeLPIC weight intervention is applied to pre-softmax weights at the epoch in question during training. The training accuracy on corrupted dataset of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison. Observe that, except for 100% corruption degree, the transferred training accuracy tends to saturate at a level largely consistent with the fraction of true training labels in the corrupted dataset.

Figure 18: Model train accuracy on corrupted dataset when the VeLPIC weight intervention is applied to pre-softmax weights at the epoch in question during training. The training accuracy on corrupted dataset of the model with standard training without weight intervention (blue dotted line) is overlaid for comparison.

Figure 19: Vector Linear Probe Intermediate layer Classifier (VeLPIC) test accuracy during training on models when intervention is performed at 40th epoch and standard training is performed thereafter for 60 epochs for 0.0 corruption degree. The VeLPIC test accuracy on models trained with standard training (dotted) without intervention is overlaid for comparison. The results correspond to a single run in each case.

Figure 20: Model, MASC and VeLPIC test accuracy for randomly initialized and trained models across different corruption degrees and model–dataset pairs. For MASC and VeLPIC, results are reported for the best performing layer, defined as the layer achieving the highest MASC/VeLPIC test accuracy for the corresponding model–dataset pair. The test accuracies are averaged over three runs.

Figure 21: Minimum Angle Subspace Classifier (MASC) test accuracy over epochs of training for ResNet-18 and CNN models having trained with and with dropout, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data with the indicated label corruption degree. The plots display MASC accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.

Figure 22: Minimum Angle Subspace Classifier (MASC) test accuracy for 0% corruption degrees over epochs of training for ResNet-18 and CNN models having with and with dropout, where test data is projected onto class-specific subspaces constructed at each epoch from corrupted training data. The plots display MASC accuracy across different layers of the network. For reference, the evolution of test accuracy of the corresponding model (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.

Figure 23: Vector Linear Probe Intermediate-layer Classifier (VeLPIC) test accuracy during training of ResNet-18 and CNN models with and with dropout, where test data is projected onto class vectors constructed at each epoch from training data with the indicated label corruption degrees. The plots display VeLPIC accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.

Figure 24: Vector Linear Probe Intermediate-layer Classifier (VeLPIC) test accuracy for 0% corruption degrees during training of ResNet-18 and CNN models with and with dropout, where test data is projected onto class vectors constructed at each epoch from training data. The plots display VeLPIC accuracy across different layers of the network for various model–dataset combinations. For reference, the test accuracy of the models (blue dotted line) over epochs of training is also shown. WD corresponding to models trained with dropout.

Explore similar work

Jul 13, 2026

Random Label Prediction Heads for Studying Memorization in Deep Neural Networks

We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks. Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head). RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers. By interpreting the RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity. We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets. Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization. Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup. These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results. The source code is available at https://github.com/MarlonBecker/RandomLabelHeads

Marlon Becker, Jonas Konrad, Luis Garcia Rodriguez +1

University of Münster, Germany

Neural Network Generalization Neural Network Memorization Statistical Learning Theory

Ask PDF

Oct 12, 2026

Memorization in image classification: test set behavior and scaling laws

Deep neural networks are capable of both learning generalizing solutions on natural tasks and memorizing arbitrary mappings. We argue that understanding the former requires understanding the latter, which is much less studied. In what ways do networks that generalize or memorize differ in their behavior or their training dynamics? Do memorizing networks simply function as a lookup table where input-output mappings are stored independently? We investigate these questions in the context of CNNs trained on image classification tasks with randomized labels or pixels. We first study the behavior of such memorizing networks on unseen images. We find that independently initialized networks trained on the same random labels learn different predictive functions: they make different predictions on test examples, in contrast with generalizing networks. We then examine the effect of dataset size on memorization through the required network capacity and training time to interpolate the training set. Counter-intuitively, we find that the number of presentations of a given example required for complete memorization can decrease with dataset size even when input-output pairs share no common structure. Although memorizing networks do not learn a universal solution, they exhibit interesting behaviors which call for further investigations into the underlying mechanisms.

Jeevesh Attri, Florentin Guth

Center for Data Science New York University · Center for Computational Neuroscience Flatiron Institute · Work done while partially affiliated with Center for Data Science, New York University.

Neural Network Memorization Neural Network Training Dynamics Neural Scaling Laws Neural Network Generalization Convolutional Neural Networks

Ask PDF

Sep 9, 2026

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on why this transition occurs, the quantitative structure of when it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Anish Kataria

Princeton University, Princeton, NJ, USA.

Multilayer Perceptrons Neural Network Memorization Neural Network Training Dynamics Neural Scaling Laws Grokking

Ask PDF