Memorization in image classification: test set behavior and scaling laws
Authors: Jeevesh Attri, Florentin Guth
Organizations: Center for Data Science New York University · Center for Computational Neuroscience Flatiron Institute · Work done while partially affiliated with Center for Data Science, New York University.
Deep neural networks are capable of both learning generalizing solutions on natural tasks and memorizing arbitrary mappings. We argue that understanding the former requires understanding the latter, which is much less studied. In what ways do networks that generalize or memorize differ in their behavior or their training dynamics? Do memorizing networks simply function as a lookup table where input-output mappings are stored independently? We investigate these questions in the context of CNNs trained on image classification tasks with randomized labels or pixels. We first study the behavior of such memorizing networks on unseen images. We find that independently initialized networks trained on the same random labels learn different predictive functions: they make different predictions on test examples, in contrast with generalizing networks. We then examine the effect of dataset size on memorization through the required network capacity and training time to interpolate the training set. Counter-intuitively, we find that the number of presentations of a given example required for complete memorization can decrease with dataset size even when input-output pairs share no common structure. Although memorizing networks do not learn a universal solution, they exhibit interesting behaviors which call for further investigations into the underlying mechanisms.
Figures & tables
Figure 1: Evolution of label fit, uncertainty, and disagreement across true-label (left), random-label (center), and random-pixel (right) conditions. Solid lines show training-set behavior and dashed lines test-set behavior. Cross-entropy measures label fit (red), entropy measures predictive uncertainty (blue), and Jensen–Shannon divergence measures disagreement across independently initialized models (green), defined as JSD=H(pˉ)−M1∑m=1MH(pm) , where pˉ=M1∑m=1Mpm (see Appendix A.3 ). Quantities are reported in dB, with 10 dB corresponding to uniform prediction (dotted gray line); JSD is rescaled to have a maximum value of 10 dB. Results are averaged over 20 independently initialized 4× -width models per condition (see Appendix A.2 ).
Figure 2: Memorization capacity frontiers across model width and training-set fraction. Rows correspond respectively to fixed and fraction-scaled batch-size/learning-rate policies, and columns to true labels, random labels, and random-pixel inputs. The final training accuracy in the width-training set size plane is estimated by interpolation from its value on grid points.
Figure 3: Training dynamics across dataset sizes, memorization tasks, and training strategies for a 4 × -width model. Panels (a-c) show evaluated training accuracy across epochs for true labels, random labels, and random-pixel inputs at five training-set sizes (indicated by line color). Solid lines denote the fixed batch-size/learning-rate policy, and dashed lines denote the fraction-scaled policy. Panel (d) shows the epoch at which training accuracy first reaches 95% as a function of training-set size, together with fitted curves t95(n)=a+b/n .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Ensemble Jensen–Shannon divergence versus mean individual predictive entropy for true labels, random labels, and random pixels. Blue solid curves show training trajectories and red dashed curves show test trajectories, with points connected in epoch order. Crosses mark the final recorded epoch. The black curve gives the expected ensemble JSD for 20 models with equal confidence and independently chosen uniform preferred classes. The dotted gray line shows the upper bound 10−H , where H is mean individual entropy in dB. The reference is descriptive and does not constitute a test of independence or uniformity.
Anuradha & Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology & Science Pilani, Hyderabad 500078, India.
School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.