How Much Information Can We Fit inside a Transformer?

Amirhossein Basareh; Rishabh Sanjay; Daniel de Mello; Abulhair Saparov

cs.AIOct 25, 2026

How Much Information Can We Fit inside a Transformer?

Authors: Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello, Abulhair Saparov

Organizations: Purdue University

Abstract

How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required, substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples---e.g., (Paris, capital_of, France) ---drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset that a model memorizes completely, grows exponentially with the transformer dimension. Equivalently, the minimum transformer dimension $d_{\min}$ required to memorize $N$ facts grows logarithmically in $N$ . Across 11 dataset sizes spanning $N = 45$ to $49{,}920$ , a lack-of-fit F-test does not reject the logarithmic fit (with an offset), but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code to reproduce our results is available at https://github.com/Magronox/transformer_information_capacity.

Figures & tables

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$ -value
Logarithmic + offset	$d=2.756\,\log(N+80.6)-10.09$	$0.997$	$0.67$
Logarithmic	$d=2.17\,\log N-5.39$	$0.979$	$9.1\times 10^{-4}$
Power-law + offset	$d=1.38\,N^{0.259}+0.00$	$0.967$	$1.7\times 10^{-5}$
Power-law	$d=1.38\,N^{0.259}$	$0.967$	$3.0\times 10^{-5}$
Linear	$d=4.83\times 10^{-4}\,N+6.30$	$0.565$	$4.4\times 10^{-16}$

Table 1: Width-scaling fits on Uniform (

L=1

, 11 measured points,

N=45

49{,}920

). All fits are computed in log space, with each fit (except the pure logarithmic fit) constrained so that the fitted

d

is positive for every

N>0

; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

Figure 2: Maximum memorizable dataset size

N_{\max}

vs. number of layers

L

at fixed width

d=8

( Uniform dataset). Grey dots: the three seeds; blue: their mean. Lines: the logarithmic (solid) and power-law (dashed) fits of Table 5 , the two forms that the lack-of-fit test does not reject.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Search space	Notes
Learning rate	$[10^{-6},\,1]$ (log)	depth: $[10^{-7},\,3]$
Use weight decay	$\{\text{False},\,\text{True}\}$	gate; if False, $\text{wd}=0$
Weight decay	$[10^{-8},\,10^{-1}]$ (log)	only sampled when gate is True
Min-LR ratio	$[10^{-8},\,1.0]$ (log)	$\text{min\_lr}=\text{lr}\cdot\text{ratio}$
Schedule-shape:
Decay end step	$[100,\,40{,}000]$	step (after warmup) where decay ends

Appendix

Table 2: Hyperparameter search space. Decay schedule is treated as a stratification variable: a separate optimization study is run for each schedule type and the best is reported. Warmup is fixed (not searched), as initial optimization results showed this would not hurt the best value achieved. Discussed in Section 3 .

Entities ( $n$ )	Facts ( $N$ )	Information (Bits)	Seeds (10 / 20 / 30) $d_{\min}$	Mean $d_{\min}$	Best $d_{\min}$
15	45	176	2 / 2 / 2	2.00	2
30	90	442	3 / 4 / 2	3.00	2
60	180	1,063	3 / 3 / 3	3.00	3
120	360	2,486	4 / 4 / 4	4.00	4
260	780	6,257	4 / 4 / 5	4.33	4
520	1,560	14,075	5 / 5 / 5	5.00	5

Appendix

Table 3: OLMo-3 factual capacity measurements across dataset sizes

N

d_{\min}

is the minimum width achieving 100% memorization accuracy across three random seeds (seeds 10, 20, 30).

Figure 3: Minimum model dimension required for complete memorization vs. dataset size with power-law fit (offset unconstrained) across dataset types ( Uniform , Symmetric , 2Hop ), when we only optimized learning rate and batch size. The minimum transformer dimension required for complete memorization is way higher than under the full search (Figure 1 ), highlighting the impact of hyperparameter optimization.

$N$	$d_{\min}$	Body %	$\text{bits}/P_{\text{body}}$	$\text{bits}/P_{\text{tot}}$
45	3.33	75%	1.0	0.72
90	4.00	66%	1.8	1.15
180	5.00	54%	2.8	1.54
360	6.67	44%	3.9	1.71
780	9.00	32%	5.7	1.80
1,560	10.33	21%	9.8	2.06

Appendix

Table 4: Non-embedding-vs-embedding parameter breakdown for the smallest single-layer model that memorizes a Uniform dataset of size

N

(

d_{\min}

averaged over 3 seeds). Non-embedding:

P=12d^{2}+15d

; embedding:

P_{\text{emb}}=(E+R)\cdot d

; bits

=N\log_{2}E

. Including embedding parameters, capacity rises from 0.72 to

{\approx}2.2

bits per parameter, reaching

{\geq}2.0

from

N=1{,}560

onward. Discussed in Section 5 .

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$
Linear without intercept	$N=498\,L$	$0.690$	$7.8\times 10^{-6}$
Linear with intercept	$N=338\,L+597$	$0.906$	$0.0025$
Power-law, exponent $\geq 1$	$N=498\,L^{1.000}$	$0.690$	$4.0\times 10^{-6}$
Exponential	$N=1197\,e^{0.103\,L}$	$0.741$	$1.2\times 10^{-5}$
Logarithmic	$N=1331\,\log L+709$	$0.992$	$0.73$
Power-law	$N=870\,L^{0.652}$	$0.966$	$0.092$

Appendix

Table 5: Depth-scaling fits on Uniform (fixed width

d=8

, 7 depths, 3 seeds). All fits are computed in log space; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

$N$	45	90	180	360	780	1,560	3,120	6,240
OLMo-3	2.00	3.00	3.00	4.00	4.33	5.00	6.00	6.00
GPT-2	3.33	4.00	5.00	6.67	9.00	10.33	12.67	14.33

Appendix

Table 6: OLMo-3 width scaling experiment: minimum transformer model dimension

d

needed to memorize all

N

facts, with

m=3

relations (averaged over 3 seeds; GPT-2 shown for comparison).

layers $L$	1	2	4	6	8	12	16
max $N$ (mean)	730	1,580	2,560	3,010	3,290	4,160	4,850
smallest seed	720	990	2,310	2,940	3,180	3,990	4,530
largest seed	750	2,010	2,760	3,060	3,420	4,320	5,010

Appendix

Table 7: Depth scaling experiment at transformer model dimension 8: maximum dataset size memorized completely, for each number of layers

L

( Uniform ,

m=3

). Each entry is the mean over seeds 10, 20, 30 for the hyperparameter configuration with the largest mean; the range is the smallest and largest seed.

width $d$	3	4	5	7	8	9	10	13	14	16
$N_{\max}$ (measured)	34	89	214	452	704	918	1,584	4,433	7,379	10,129
$N$ (Table 1 fit)	35	85	158	413	628	939	1,384	4,270	6,173	12,841
difference	$-3\%$	$+4\%$	$+35\%$	$+10\%$	$+12\%$	$-2\%$	$+14\%$	$+4\%$	$+20\%$	$-21\%$

Appendix

Table 8: Largest memorizable dataset size

N_{\max}

L=1

for each width

d

(mean over seeds 10, 20, 30), and the

N

predicted by the width scaling experiment’s logarithmic-with-offset fit

d=2.756\ln(N+80.6)-10.09

of Table 1 .

Functional form	Best fit	$R^{2}$	Lack-of-fit $p$
Logarithmic + offset	$d=3.018\,\ln(N+125.9)-12.31$	$0.996$	$0.085$
Logarithmic	$d=2.374\,\ln N-6.93$	$0.970$	$4.6\times 10^{-7}$
Power-law + offset	$d=1.08\,N^{0.298}+0.00$	$0.989$	$5.5\times 10^{-4}$
Power-law	$d=1.08\,N^{0.298}$	$0.989$	$9.4\times 10^{-4}$
Linear	$d=4.66\times 10^{-3}\,N+2.87$	$0.816$	$3.1\times 10^{-14}$

Appendix

Table 9: Fits to the

N_{\max}

measurements of Table 8 , written as

d=f(N)

as in Table 1 . Fits are computed on

\ln N_{\max}

with the three seeds; the offsets are constrained so that the fitted

d

is positive for every

N>0

; the last column is the lack-of-fit F-test

p

-value (

p<0.05

rejects the functional form).

entities $n$	9216	4608	3072	2304	1152	576	288	144	96
relations $m$	1	2	3	4	8	16	32	64	96
min. dimension $d$ to memorize all facts	6.00	10.67	15.33	19.00	29.33	38.00	44.00	46.33	45.00

Appendix

Table 10: Minimum transformer model dimension

d

needed to memorize

N=9{,}216

facts as we vary the composition of the dataset, from 9,216 entities with 1 relation to 96 entities with 96 relations (averaged over 3 seeds).

Figure 4: Convergence Speed vs. Dimension. The total number of batches required to achieve 100% training accuracy vs transformer dimension

d

for different entity sizes

E\in\{125,250,500,1000,2000\}

. Plots for both the linear and cosine decay types are shown. The y-axis is on a log scale, showing that as the model approaches the theoretical minimum dimension

d_{min}

, the cost required to train for perfect memorization increases exponentially. The shaded bands represent the variation of the number of batches required over 3 seeds.

Memorization in Language Models

Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.

See all 117 papers

Geometric Factual Recall in Transformers
May 12, 2026Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1Neural Representation Geometry Memorization in Language Models
1New York University · 2Flatiron Institute
Learning to Recall with Transformers Beyond Orthogonal Embeddings
Mar 16, 2026Nuri Mert Vural, Alberto Bietti, Mahdi Soltanolkotabi +1Transformer Memorization in Language Models
University of Toronto and Vector Institute · Flatiron Institute · University of Southern California +1
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Jul 10, 2026Roberto Garcia, Jerry Liu, Ronny Junkins +3Transformer Memorization in Language Models
Institute for Computational & Mathematical Engineering, Stanford University · Department of Computer Science, Stanford University · Department of Computer Science and Engineering, University at Buffalo

May 12, 2026J/K move · Enter open · S save

Geometric Factual Recall in Transformers

Shauli Ravfogel, Gilad Yehudai, Joan Bruna, Alberto Bietti

1New York University · 2Flatiron Institute

Neural Representation Geometry Memorization in Language Models Neural Network Memorization Memorization in Generative Models Factual Knowledge in Language Models

Open paper AskPDF arXiv

How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, geometric form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer must memorize random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices: subject embeddings encode linear superpositions of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute via ReLU gating, and not as an associative key-value mapping. We extend these results to the multi-hop setting -- chains of relational queries such as ``Who is the mother of the wife of

x

?'' -- providing constructions with and without chain-of-thought that exhibit a provable capacity-depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with precisely the predicted structure. Once trained, the MLP transfers zero-shot to entirely new bijections when subject embeddings are appropriately re-initialized, revealing that it has learned a generic selection mechanism rather than memorized any particular set of facts.

Abstract

Memorization in Language Models

Explore similar work

Geometric Factual Recall in Transformers

Learning to Recall with Transformers Beyond Orthogonal Embeddings

MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers