How Much Information Can We Fit inside a Transformer?

Amirhossein Basareh; Rishabh Sanjay; Daniel de Mello; Abulhair Saparov

cs.AIOct 25, 2026

How Much Information Can We Fit inside a Transformer?

Authors: Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello, Abulhair Saparov

Organizations: Purdue University

Abstract

How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required, substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples---e.g., (Paris, capital_of, France) ---drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset that a model memorizes completely, grows exponentially with the transformer dimension. Equivalently, the minimum transformer dimension $d_{\min}$ required to memorize $N$ facts grows logarithmically in $N$ . Across 11 dataset sizes spanning $N = 45$ to $49{,}920$ , a lack-of-fit F-test does not reject the logarithmic fit (with an offset), but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code to reproduce our results is available at https://github.com/Magronox/transformer_information_capacity.

Figures & tables

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$ -value
Logarithmic + offset	$d=2.756\,\log(N+80.6)-10.09$	$0.997$	$0.67$
Logarithmic	$d=2.17\,\log N-5.39$	$0.979$	$9.1\times 10^{-4}$
Power-law + offset	$d=1.38\,N^{0.259}+0.00$	$0.967$	$1.7\times 10^{-5}$
Power-law	$d=1.38\,N^{0.259}$	$0.967$	$3.0\times 10^{-5}$
Linear	$d=4.83\times 10^{-4}\,N+6.30$	$0.565$	$4.4\times 10^{-16}$

Table 1: Width-scaling fits on Uniform (

L=1

, 11 measured points,

N=45

49{,}920

). All fits are computed in log space, with each fit (except the pure logarithmic fit) constrained so that the fitted

d

is positive for every

N>0

; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

Figure 2: Maximum memorizable dataset size

N_{\max}

vs. number of layers

L

at fixed width

d=8

( Uniform dataset). Grey dots: the three seeds; blue: their mean. Lines: the logarithmic (solid) and power-law (dashed) fits of Table 5 , the two forms that the lack-of-fit test does not reject.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Search space	Notes
Learning rate	$[10^{-6},\,1]$ (log)	depth: $[10^{-7},\,3]$
Use weight decay	$\{\text{False},\,\text{True}\}$	gate; if False, $\text{wd}=0$
Weight decay	$[10^{-8},\,10^{-1}]$ (log)	only sampled when gate is True
Min-LR ratio	$[10^{-8},\,1.0]$ (log)	$\text{min\_lr}=\text{lr}\cdot\text{ratio}$
Schedule-shape:
Decay end step	$[100,\,40{,}000]$	step (after warmup) where decay ends

Appendix

Table 2: Hyperparameter search space. Decay schedule is treated as a stratification variable: a separate optimization study is run for each schedule type and the best is reported. Warmup is fixed (not searched), as initial optimization results showed this would not hurt the best value achieved. Discussed in Section 3 .

Entities ( $n$ )	Facts ( $N$ )	Information (Bits)	Seeds (10 / 20 / 30) $d_{\min}$	Mean $d_{\min}$	Best $d_{\min}$
15	45	176	2 / 2 / 2	2.00	2
30	90	442	3 / 4 / 2	3.00	2
60	180	1,063	3 / 3 / 3	3.00	3
120	360	2,486	4 / 4 / 4	4.00	4
260	780	6,257	4 / 4 / 5	4.33	4
520	1,560	14,075	5 / 5 / 5	5.00	5

Appendix

Table 3: OLMo-3 factual capacity measurements across dataset sizes

N

d_{\min}

is the minimum width achieving 100% memorization accuracy across three random seeds (seeds 10, 20, 30).

Figure 3: Minimum model dimension required for complete memorization vs. dataset size with power-law fit (offset unconstrained) across dataset types ( Uniform , Symmetric , 2Hop ), when we only optimized learning rate and batch size. The minimum transformer dimension required for complete memorization is way higher than under the full search (Figure 1 ), highlighting the impact of hyperparameter optimization.

$N$	$d_{\min}$	Body %	$\text{bits}/P_{\text{body}}$	$\text{bits}/P_{\text{tot}}$
45	3.33	75%	1.0	0.72
90	4.00	66%	1.8	1.15
180	5.00	54%	2.8	1.54
360	6.67	44%	3.9	1.71
780	9.00	32%	5.7	1.80
1,560	10.33	21%	9.8	2.06

Appendix

Table 4: Non-embedding-vs-embedding parameter breakdown for the smallest single-layer model that memorizes a Uniform dataset of size

N

(

d_{\min}

averaged over 3 seeds). Non-embedding:

P=12d^{2}+15d

; embedding:

P_{\text{emb}}=(E+R)\cdot d

; bits

=N\log_{2}E

. Including embedding parameters, capacity rises from 0.72 to

{\approx}2.2

bits per parameter, reaching

{\geq}2.0

from

N=1{,}560

onward. Discussed in Section 5 .

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$
Linear without intercept	$N=498\,L$	$0.690$	$7.8\times 10^{-6}$
Linear with intercept	$N=338\,L+597$	$0.906$	$0.0025$
Power-law, exponent $\geq 1$	$N=498\,L^{1.000}$	$0.690$	$4.0\times 10^{-6}$
Exponential	$N=1197\,e^{0.103\,L}$	$0.741$	$1.2\times 10^{-5}$
Logarithmic	$N=1331\,\log L+709$	$0.992$	$0.73$
Power-law	$N=870\,L^{0.652}$	$0.966$	$0.092$

Appendix

Table 5: Depth-scaling fits on Uniform (fixed width

d=8

, 7 depths, 3 seeds). All fits are computed in log space; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

$N$	45	90	180	360	780	1,560	3,120	6,240
OLMo-3	2.00	3.00	3.00	4.00	4.33	5.00	6.00	6.00
GPT-2	3.33	4.00	5.00	6.67	9.00	10.33	12.67	14.33

Appendix

Table 6: OLMo-3 width scaling experiment: minimum transformer model dimension

d

needed to memorize all

N

facts, with

m=3

relations (averaged over 3 seeds; GPT-2 shown for comparison).

layers $L$	1	2	4	6	8	12	16
max $N$ (mean)	730	1,580	2,560	3,010	3,290	4,160	4,850
smallest seed	720	990	2,310	2,940	3,180	3,990	4,530
largest seed	750	2,010	2,760	3,060	3,420	4,320	5,010

Appendix

Table 7: Depth scaling experiment at transformer model dimension 8: maximum dataset size memorized completely, for each number of layers

L

( Uniform ,

m=3

). Each entry is the mean over seeds 10, 20, 30 for the hyperparameter configuration with the largest mean; the range is the smallest and largest seed.

width $d$	3	4	5	7	8	9	10	13	14	16
$N_{\max}$ (measured)	34	89	214	452	704	918	1,584	4,433	7,379	10,129
$N$ (Table 1 fit)	35	85	158	413	628	939	1,384	4,270	6,173	12,841
difference	$-3\%$	$+4\%$	$+35\%$	$+10\%$	$+12\%$	$-2\%$	$+14\%$	$+4\%$	$+20\%$	$-21\%$

Appendix

Table 8: Largest memorizable dataset size

N_{\max}

L=1

for each width

d

(mean over seeds 10, 20, 30), and the

N

predicted by the width scaling experiment’s logarithmic-with-offset fit

d=2.756\ln(N+80.6)-10.09

of Table 1 .

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$
Logarithmic + offset	$d=3.018\,\ln(N+125.9)-12.31$	$0.996$	$0.085$
Logarithmic	$d=2.374\,\ln N-6.93$	$0.970$	$4.6\times 10^{-7}$
Power-law + offset	$d=1.08\,N^{0.298}+0.00$	$0.989$	$5.5\times 10^{-4}$
Power-law	$d=1.08\,N^{0.298}$	$0.989$	$9.4\times 10^{-4}$
Linear	$d=4.66\times 10^{-3}\,N+2.87$	$0.816$	$3.1\times 10^{-14}$

Appendix

Table 9: Fits to the

N_{\max}

measurements of Table 8 , written as

d=f(N)

as in Table 1 . Fits are computed on

\ln N_{\max}

with the three seeds; the offsets are constrained so that the fitted

d

is positive for every

N>0

; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

entities $n$	9216	4608	3072	2304	1152	576	288	144	96
relations $m$	1	2	3	4	8	16	32	64	96
min. dimension $d$ to memorize all facts	6.00	10.67	15.33	19.00	29.33	38.00	44.00	46.33	45.00

Appendix

Table 10: Minimum transformer model dimension

d

needed to memorize

N=9{,}216

facts as we vary the composition of the dataset, from 9,216 entities with 1 relation to 96 entities with 96 relations (averaged over 3 seeds).

Figure 4: Convergence Speed vs. Dimension. The total number of batches required to achieve 100% training accuracy vs transformer dimension

d

for different entity sizes

E\in\{125,250,500,1000,2000\}

. Plots for both the linear and cosine decay types are shown. The y-axis is on a log scale, showing that as the model approaches the theoretical minimum dimension

d_{min}

, the cost required to train for perfect memorization increases exponentially. The shaded bands represent the variation of the number of batches required over 3 seeds.

Memorization in Language Models

Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.

See all 117 papers

May 12, 2026

Geometric Factual Recall in Transformers

How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, geometric form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer must memorize random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices: subject embeddings encode linear superpositions of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute via ReLU gating, and not as an associative key-value mapping. We extend these results to the multi-hop setting -- chains of relational queries such as ``Who is the mother of the wife of

x

?'' -- providing constructions with and without chain-of-thought that exhibit a provable capacity-depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with precisely the predicted structure. Once trained, the MLP transfers zero-shot to entirely new bijections when subject embeddings are appropriately re-initialized, revealing that it has learned a generic selection mechanism rather than memorized any particular set of facts.

Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1

1New York University · 2Flatiron Institute

Neural Representation Geometry Memorization in Language Models Neural Network Memorization Memorization in Generative Models Factual Knowledge in Language Models

Ask PDF

Mar 16, 2026

Learning to Recall with Transformers Beyond Orthogonal Embeddings

Modern large language models (LLMs) excel at tasks that require storing and retrieving knowledge, such as factual recall and question answering. Transformers are central to this capability because they can encode information during training and retrieve it at inference. Existing theoretical analyses typically study transformers under idealized assumptions such as infinite data or orthogonal embeddings. In realistic settings, however, models are trained on finite datasets with non-orthogonal (random) embeddings. We address this gap by analyzing a single-layer transformer with random embeddings trained with (empirical) gradient descent on a simple token-retrieval task, where the model must identify an informative token within a length-

L

sequence and learn a one-to-one mapping from tokens to labels. Our analysis tracks the ``early phase'' of gradient descent and yields explicit formulas for the model's storage capacity -- revealing a multiplicative dependence between sample size

N

, embedding dimension

d

, and sequence length

L

. We validate these scalings numerically and further complement them with a lower bound for the underlying statistical problem, demonstrating that this multiplicative scaling is intrinsic under non-orthogonal embeddings.

Nuri Mert Vural, Alberto Bietti, Mahdi Soltanolkotabi +1

University of Toronto and Vector Institute · Flatiron Institute · University of Southern California +1

Transformer Memorization in Language Models Representation Geometry in Language Models

Ask PDF

Jul 10, 2026

MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanistic interpretability models of fact-storage in LLMs fail to explain the surprising empirical phenomenon that they store facts at an information-theoretically optimal rate. In this work, we develop a theoretical account of this phenomenon. We develop the first Transformer-compatible fact-storing MLP closed-form construction that satisfies the following three properties empirically observed in LLMs: it (i) attains optimal fact storage scaling, (ii) handles arbitrary input/output geometries, and (iii) works inside Transformers. Key to our work is to analyze the decoding margin of MLPs, whereas prior work only studies MLP fact storage. Under isotropic embeddings, our construction achieves information-theoretically optimal storage capacity scaling and requires

10

104\times

fewer parameters at matched fact count than prior constructions. For arbitrary key and value embeddings, we show that our construction attains the same storage capacity scaling, up to penalization factors depending on the embedding geometries. Moreover, we demonstrate that our constructed MLPs can be used within Transformer blocks for factual recall tasks at optimal capacity scaling, requiring

15

63\times

fewer parameters at matched fact count than prior constructions. Finally, as a proof-of-concept, we show that fact-storing MLPs enable modular fact editing by swapping a Transformer's MLP with a new one.

Roberto Garcia, Jerry Liu, Ronny Junkins +3

Institute for Computational & Mathematical Engineering, Stanford University · Department of Computer Science, Stanford University · Department of Computer Science and Engineering, University at Buffalo

Transformer Memorization in Language Models Factual Knowledge in Language Models Associative Memory

Ask PDF

Abstract

Memorization in Language Models

Explore similar work

Geometric Factual Recall in Transformers

Learning to Recall with Transformers Beyond Orthogonal Embeddings

MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers