cs.LGNov 3, 2026

An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons

Authors: Nancy Baygorrea C., Otávio da Fonseca Martins Gomes

Organizations: Laboratory of Mathematical Sciences, State University of Northern Rio de Janeiro · Centre for Mineral Technology, Postgraduate Program in Geosciences, National Museum, Rio de Janeiro

Abstract

How much label noise has a network memorized, and when? We study a multilayer perceptron and what statistical complexity adds to a description of its predictions. Standard identities decompose predictive information into signal, memorization, and heterogeneity. The memorization term vanishes in the population under the stated noise-independence condition. A Fano-type bound relates it to training accuracies, but the memorized fraction M provides no upper bound. Across 89 MNIST runs with ten pre-specified hypotheses (five supported, four unsupported, one not testable), normalized memorization information rose with M on training samples, remained near zero on held-out samples with fresh noisy labels, and was close to its Fano bound. Unlike raw M, it has a zero population null under the independence condition and chance correction largely removed this difference. When every label was replaced by an incorrect class, test accuracy fell to 0.5%, yet predictions retained information about the true class. Mean statistical complexity peaked before memorization at corruption rates up to 40%. At higher rates, its peak fell within the memorization phase, near the epoch when mean training confidence reached 0.6 and the two could not be separated. As a stopping rule, it matched established rules only at low noise and detected corrupted samples less well than simple label-free scores. An exploratory extension of 280 runs covering Fashion-MNIST, pair-flip noise, and four training variants yielded similar results under uniform noise, but not pair-flip noise, where complexity peaked within two epochs. Conclusions are limited to this architecture, two small datasets, and synthetic noise.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise

    May 18, 2026Linyu Liu, Pinyan LuNeural Network GeneralizationNeural Network Memorization

  2. Random Label Prediction Heads for Studying Memorization in Deep Neural Networks

    Jul 13, 2026Marlon Becker, Jonas Konrad, Luis Garcia Rodriguez +1Neural Network GeneralizationNeural Network Memorization

  3. Fisher information and the geometry of memorization in neural networks

    Oct 12, 2026Daniel T. Bernstein, Chase Goddard, Luca Di Carlo +2Neural Network MemorizationNeural Network Generalization