Meta-memorization and memorization scaling laws in transformers

Alex Nguyen; Kenneth A. Norman; Gautam Reddy

cs.LGNov 1, 2026

Meta-memorization and memorization scaling laws in transformers

Authors: Alex Nguyen, Kenneth A. Norman, Gautam Reddy

Organizations: Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA · Princeton Neuroscience Institute, Department of Psychology, Princeton University, Princeton, NJ, USA · Department of Physics, Princeton University, Princeton, NJ, USA

Abstract

Larger transformers can memorize more data, with a capacity (in bits) that scales roughly linearly with the number of parameters. Counterintuitively, larger models also memorize faster, requiring fewer repetitions of a sequence during training before it is memorized. We reproduce both memorization scaling laws in transformers trained on random binary sequences, showing that rapid memorization can arise even when data contain no semantic or compositional structure. Rapid memorization is due to a meta-memorization effect, where models trained to memorize a large set of random sequences learn to memorize novel sequences more quickly. The rate $r(t)$ at which models memorize grows with training time $t$ as a power-law $r(t) \sim t^β$ with a fixed exponent $β$ but an increasing upper cutoff across model scales. Meta-memorization enables even one-layer transformers to few-shot memorize a 100-bit sequence. This effect is partly explained by the formation of structured attention heads that convey position-invariant sequence information to MLPs where memorized sequences are stored. Meta-memorization is recapitulated in a simplified model of a dense associative memory where retrieval is constrained by an attentional bottleneck. Our results suggest transformers, through pre-training, acquire reusable mechanisms that enable highly efficient storage and retrieval of memories.

Figures & tables

Figure 1: Experimental setup. We train autoregressive transformers on $K=10^{5}$ random binary sequences of length $N=100$ . (a) Four example sequences. (b) Sequences are assigned ranks and sampled from a Zipfian distribution, producing repetition counts spanning orders of magnitude. For illustrative purposes we only plot up to rank $10^{3}$ . Model size is varied through the number of heads $H$ and model dimension $D$ .

Figure 2: Memorization scaling laws and meta-memorization for 12-layer and 1-layer architectures. (a) For both architectures, final train loss decreases primarily with total parameter count of models even though models vary in both number of heads $H$ and model dimension $D$ . The 12-layer and 1-layer models are trained for $200,000$ and $80,000$ iterations, respectively, with a fixed learning rate. Note that the largest models failed to converge with this learning rate and are not shown in the plot. (b) The maximum rank of memorized sequences ( $\propto$ the memorization capacity $C$ in bits) shown at $60,000$ , $120,000$ and $190,000$ iterations for the 12-layer models and $20,000$ , $50,000$ and $79,000$ iterations for the 1-layer model. The exponent of a power law fit $\gamma$ is $\approx 1$ for both architectures and during training, thus showing that memorization capacity $C$ scales linearly with parameter count $P$ . (c) Larger models memorize sequences faster. Moreover, models memorize data from a second dataset faster after a dataset switch (red dashed line). Due to the rapid learning in 1-layer models, here we show results only for iterations $[0,10^{4}]$ and $[4\times 10^{4},5\times 10^{4}]$ to better illustrate the effect.

Figure 3: Meta-memorization scaling laws. Across models, we find that fewer presentations are required to memorize a rarer sequence (of higher rank) (Figure S4 b). Since higher-rank sequences tend to appear later on during training, this suggests that models become more efficient at memorizing over the course of training. In panels a) and b), the number of presentations ( $n_{k}$ ) vs rank ( $k$ ) data is first fit to a power-law $n_{k}=Ak^{-\alpha}$ for each model. We plot the rescaled number of presentations (i.e., after dividing by $A$ for each model) vs rank for all models. Across model sizes, 12-layer and 1-layer models display power laws with exponents $\alpha\approx 0.7$ and $\alpha\approx 0.85$ , respectively. The exponent is common to all model sizes within a model class, but the scaling extends to higher ranks for larger models. In panels c) and d) we show appearances at saturation vs. sequence rank in data-switching experiments for the largest 1-layer models. Appearances at saturation is the number of presentations $n_{k}$ at which a sequence $k$ is memorized (i.e. learned to 90% accuracy). To avoid visual clutter, we bin the x-axis logarithmically and show the geometric mean of sequence ranks and appearances at saturation within each bin. Error bands show the minimum number of appearances among ranks that fall in each logarithmic bin. We show these pre-switch and post-switch data and observe that after the switch, sequences of equal rank require fewer presentations to be learned. (d) Furthermore, the largest model at $5.7\times 10^{7}$ parameters was able to learn some sequences in one-shot after the switch (the error band reaches 1 appearance after the switch).

Figure 4: Division of labor among feedforward blocks and attention heads. Top : We train models where half-way through training, we resample all training sequences. (a) Accuracy of models after training on second sequences as a function of model size and sequence ranks. (b) Accuracy of models on first sequences as a function of model size and sequence ranks. (c) Accuracy of component-swapped models on first sequences, using MLPs at the end of the first period. Bottom : Linear decodability of 1-layer representations. (d) Linear decodability of 1-layer attention representations of novel sequences unseen during training as a function of training iteration. (e) Linear decodability of MLP representations of novel sequences unseen during training as a function of training iteration. (f) Linear decodability of attention representations predicts accuracy of probe experiments.

Figure 5: Learned attention maps in a trained one-layer transformer. Each panel shows one of 16 heads with $F=64$ features per head, evaluated on the most frequent sequence. Rows are query positions and columns are causal key positions. Vertical bands indicate that heads specialize to fixed positions or position subsets, providing reusable context features to the MLP at each position. See Figure S9 and Figure S10 for attention maps for a different sequence and for a single head.

Figure 6: A phenomenology-driven theory of meta-memorization. (a) In this model, the attention heads present position-invariant information about the sequence to a key-value memory through a finite attentional bottleneck. The key-value memory uses this sequence information to first retrieve sequence index ( $\mu$ ) and a downstream decoder uses sequence index $\mu$ and the current position in the sequence $n$ to predict the $n+1$ th bit. (b) The DAM model recapitulates the finding that switching to new random sequences reveals meta-memorization. Here, $A$ and $B$ are trained continuously under the next-bit objective. Accuracy drops when the sequence set is replaced but recovers faster than during initial training. We show results only for iterations $[0,2\times 10^{3}]$ and $[10^{4},1.2\times 10^{4}]$ to better illustrate the effect. (c) Here, $A$ and $B$ are trained using an InfoNCE-style retrieval objective. The model shows post-switch accuracy comparable to pre-switch accuracy even when $A$ and $B$ are frozen after the switch. In both cases, transfer strengthens with the number of heads $H$ . Note that the noisy accuracy curves (due to finite-batch effects) are smoothed by computing a running average. This explains why the accuracy does not appear to return to random chance in panel b after the data switch, since the first point depicted is a running average of the first 100 steps’ accuracies after the switch. Here, we use $N=50,M=3000,K=500$ , batch size = 50.

Figure 7: Train loss across a wider parameter sweep across 3 hyperparameters: number of attention heads

H

, width per head

F

, and width

W

of MLP for

5\times 10^{4}

iterations. Each color represents models with a fixed model dimension

D=FH

but different

W

. The loss of models with

W=D

is plotted with cross marks. The elbow of the loss for a fixed

D

begins approximately at

W\approx D

, suggesting an optimal allocation of parameters to attention heads and MLPs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Recall accuracy scales with parameter count. Average next-bit accuracy on the top $10000$ sequences increases with total parameters across architectures that vary in $H$ and $D$ .

Figure S2: Train loss and recall accuracy versus total parameter count for 12-layer GPTs trained on binary sequences, sampled uniformly with replacement. Models are trained on $K=10{,}000$ sequences with context length $100$ over $10^{5}$ training iterations. We vary the number of attention heads $H$ and the number of hidden features per head $F$ . Train loss decreases and recall accuracy increases with total parameter count, mostly independently of architecture.

Figure S3: Training loss dynamics across model scales for 12-layer models trained on Zipfian distributed sequences. Larger models generally reach lower loss, even though the largest models show increased loss compared to mid-size models since we did not explicitly tune our optimization hyperparameters.

Figure S4: Meta-memorization in 12-layer and 1-layer models. (a) Saturation iteration vs. sequence rank for 12-layer and 1-layer models. We define a sequence’s saturation iteration as the iteration the model “memorized” it (i.e. attains average accuracy above 90%). We observe that sequences of higher rank are learned later in time, mainly because they are presented less frequently due to the Zipfian data distribution. However, by taking a vertical slice through our figures, we see that larger models memorize faster, since for any given sequence rank, larger models generally require fewer iterations to memorize it. By taking a horizontal slice through our figures, we see that at any given time, larger models will generally have learned more sequences. (b) Appearances at saturation vs. sequence rank for different model sizes. We define appearances at saturation as the number of presentations $n_{k}$ the model takes to “memorize” a sequence $k$ (i.e. learned to 90% accuracy). We observe that higher-ranked sequences require fewer presentations to be memorized. Interestingly, we observe that the very largest 1-layer models require as few as 2 presentations to memorize a sequence (bottom right).

Figure S5: Meta-memorization in data-switching experiments for 12-layer models across sequences of different ranks. We switched the training set of sequences after $4\times 10^{4}$ iterations to a fresh set of random sequences drawn from the same Zipfian distribution. For sequences across different ranks, accuracy immediately returned to chance after the switch, but recovered faster relative to the start of training (See Figure 2 c).

Figure S6: Meta-memorization in data-switching experiments for 1-layer models across sequences of different ranks. We switched the training set of sequences after $4\times 10^{4}$ iterations to a fresh set of random sequences drawn from the same Zipfian distribution. For sequences across different ranks, accuracy immediately returned to chance after the switch, but recovered faster relative to the start of training (See Figure 2 c). Due to the very rapid learning in 1-layer models, here we show results only for iterations $[0,10^{4}]$ and $[4\times 10^{4},5\times 10^{4}]$ to better illustrate the effect.

Figure S7: Fine-tuning accuracy of 1-layer transformers on novel unseen sequences increases as a function of training. Larger models achieve higher fine-tuning accuracy.

Figure S8: 1-layer models with attention weights set to zero throughout training and probing fail to learn novel sequences.

Figure S9: Learned attention maps in a trained one-layer transformer for Sequence 10 (16 heads, 64 features per head). Each panel shows attention maps of one head, evaluated on the that sequence. Rows are query positions and columns are causal key positions.

Figure S10: Attention maps for the fifth head in a trained one-layer transformer across multiple sequences (16 heads, 64 features per head). Each panel shows the attention maps for this head, evaluated on the 10 lowest-ranked sequences. Rows are query positions and columns are causal key positions. We observe vertical bands at similar positions across multiple random sequences.

Neural Network Memorization

Neural network memorization studies how models store and retrieve training examples, how this differs from generalization, and how memory can be measured or controlled.

See all 68 papers

Oct 25, 2026

How Much Information Can We Fit inside a Transformer?

How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required, substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples---e.g., (Paris, capital_of, France) ---drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset that a model memorizes completely, grows exponentially with the transformer dimension. Equivalently, the minimum transformer dimension

d_{\min}

required to memorize

N

facts grows logarithmically in

N

. Across 11 dataset sizes spanning

N = 45

49{,}920

, a lack-of-fit F-test does not reject the logarithmic fit (with an offset), but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code to reproduce our results is available at https://github.com/Magronox/transformer_information_capacity.

Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello +1

Purdue University

Memorization in Language Models Transformer Factual Knowledge in Language Models

Ask PDF

Mar 22, 2026

Mixture of Chapters: Scaling Learnt Memory in Transformers

Transformers lack an explicit architectural mechanism for storing and organizing knowledge acquired during training. We introduce learnable sparse memory banks: a set of latent tokens, randomly initialized and trained end-to-end, that transformer layers query via cross-attention to retrieve stored knowledge. To scale memory capacity without prohibitive attention costs, we propose chapter-based routing inspired by Mixture-of-Experts architectures, partitioning the memory bank into chapters and training a router to select relevant subsets per input. This enables scaling to 262K memory tokens while maintaining tractable computation. We evaluate our approach against standard transformers (in iso-FLOP settings) on pre-training and instruction fine-tuning across relevant benchmarks. Our models surpass iso-FLOP baselines suggesting scope for a new axis of scaling, demonstrating that explicit associative memory provides complementary capacity to what is captured implicitly in model parameters. Additionally, we observe improved knowledge retention under continued training, with robustness to forgetting when transitioning between training phases (e.g., pretraining to instruction fine-tuning).

Tasmay Pankaj Tibrewal, Pritish Saha, Ankit Meda +2

IIT Kharagpur · Fractal AI

Memory-Augmented Language Models Persistent Memory for Language Models Associative Memory

Ask PDF

Mar 16, 2026

Learning to Recall with Transformers Beyond Orthogonal Embeddings

Modern large language models (LLMs) excel at tasks that require storing and retrieving knowledge, such as factual recall and question answering. Transformers are central to this capability because they can encode information during training and retrieve it at inference. Existing theoretical analyses typically study transformers under idealized assumptions such as infinite data or orthogonal embeddings. In realistic settings, however, models are trained on finite datasets with non-orthogonal (random) embeddings. We address this gap by analyzing a single-layer transformer with random embeddings trained with (empirical) gradient descent on a simple token-retrieval task, where the model must identify an informative token within a length-

L

sequence and learn a one-to-one mapping from tokens to labels. Our analysis tracks the ``early phase'' of gradient descent and yields explicit formulas for the model's storage capacity -- revealing a multiplicative dependence between sample size

N

, embedding dimension

d

, and sequence length

L

. We validate these scalings numerically and further complement them with a lower bound for the underlying statistical problem, demonstrating that this multiplicative scaling is intrinsic under non-orthogonal embeddings.

Nuri Mert Vural, Alberto Bietti, Mahdi Soltanolkotabi +1

University of Toronto and Vector Institute · Flatiron Institute · University of Southern California +1

Transformer Memorization in Language Models Representation Geometry in Language Models

Ask PDF

Abstract

Neural Network Memorization

Explore similar work

How Much Information Can We Fit inside a Transformer?

Mixture of Chapters: Scaling Learnt Memory in Transformers

Learning to Recall with Transformers Beyond Orthogonal Embeddings