What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

Yekun Chai; Haoyi Xiong

cs.CLOct 4, 2026

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

Authors: Yekun Chai, Haoyi Xiong

Organizations: FloatAI · ETH Zurich · Independent Researcher

Abstract

As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters. We answer them by pricing a repeated token against two references: one epoch on the same data, which gives its value, and fresh data at equal compute, which gives its cost. Against fresh data, the cost of repetition follows a single variable, the number of extra epochs divided by the unique tokens per parameter. Against the same data, a second epoch is worth nearly as much as a fresh one, and repeated tokens fall to half the value of fresh ones after a critical epoch count that grows with the training budget per parameter but hardly with model size. With unique data fixed, the predicted compute-optimal run grows model size and epochs together until loss stops improving, near the critical epoch count. The same variable accounts for the direction of size trends that appear to conflict: larger models tolerate fewer epochs when the corpus is fixed, from about 15 at 127M to 4 at 2B parameters, but not when unique data grow with the model. Counts alone do not determine loss: at identical counts, replaying shards consecutively raises loss by up to 0.46~bits per byte, concentrating repeats on fewer samples also raises it, lower-entropy sources degrade faster with repetition, and re-tokenizing repeats helps only under heavy repetition. These results offer an empirical guide to pretraining when unique data, rather than compute, are the binding constraint.

Figures & tables

Figure 1: Excess loss grows with extra epochs and falls with unique data per parameter; with unique data fixed, the predicted compute-optimal run reaches its lowest loss near the point where a repeat is worth half a fresh token. CommonCrawl grid (§ 4 ). (a) Held-out loss at $U=0.5$ B unique tokens over training tokens $D=RU$ and model size $N$ , from the one-epoch fit plus Eq. ( 3 ), inside the grid’s range (gray, training tokens per parameter, TPP $=D/N$ , below 5); vertical lines mark epochs. Red, the compute-optimal allocation (lowest loss at each compute $C=6ND$ , § 4.4 ); light dashed, lines of equal compute; dark dashed, the same minimization with the one-epoch fit alone, as if every repeated token were worth a fresh one ( $\eta=1$ ); star, the lowest loss at this $U$ , beyond which more compute raises loss (faded). Dotted, the critical epoch count $R_{c}$ of Eq. ( 4 ), beyond which a repeated token is worth less than half a fresh one ( $\eta<\tfrac{1}{2}$ , Eq. 1 ). The optimum holds TPP between 16 and 22, below the fresh-data 25, and reaches its lowest loss at 5.3 epochs, where $R_{c}\approx 5.3$ . (b) Excess loss over fresh data at equal compute (Eq. 2 ) against $z=(R-1)N_{\rm total}/U$ for the 113 of 140 repeated configurations above the 0.002-bpb (bits per byte) resolution threshold; dashed, Eq. ( 3 ). Per-size maps and critical epoch counts are in Figure 4 .

Figure 2: Study design and geometry of multi-epoch pretraining. (a) The data-matched baseline shares

N

and

U

; extra epochs add training tokens

D=RU

and compute. (b) The compute-matched baseline shares

N

and

D

; repeated data replace fresh data. (c) Both quantities on measured 59M losses:

\Delta L

(Eq. 2 ) is the gap to the compute-matched baseline at equal

D

, and

U_{\rm eq}

, the fresh budget with the same loss, gives

\eta\approx 0.85

(Eq. 1 ). (d)

z=(R-1)N_{\rm total}/U

against model size at

R=4

in the three designs: with

U

fixed,

z\propto N_{\rm total}

(§ 5.1 ); with

U/N

(§ 5.2 ) or

D/N

(§ 5.3 ) fixed,

z\propto N_{\rm total}/N

, which falls slowly as the embedding share shrinks.

Figure 3: Repeats lose value with epochs, fastest at small budgets.

\eta

(Eq. 1 ): mean (lines) and range (shading) over five sizes. Sixteen-epoch runs at

D/N\leq 20

are omitted: their loss lies above the shortest one-epoch run, so

U_{\rm eq}

cannot be read off.

Figure 4: Repeated tokens fall to half the value of fresh ones at a critical epoch count that grows with the training budget and hardly with model size. Excess loss over fresh data at equal compute on the CommonCrawl grid, for every size, budget and epoch count (values in Appendix Figure 12 ). Black points, the measured $R_{c}$ , where $\eta=\tfrac{1}{2}$ , interpolated in $\log R$ ; white dashed, the 0.1-bpb level of Eq. ( 3 ).

Figure 5: At fixed unique data, the fitted compute-optimal frontier grows model size and epochs together until loss stops improving. (a,b) Predicted loss over epochs

R

and model size

N

U=0.5

B and 1.5B (fresh-data fit plus Eq. 3 ), within the grid’s range (

1\leq R\leq 16

5\leq D/N\leq 300

); since

C=6NRU

, the dotted diagonals are iso-FLOP lines, labeled in FLOPs. The solid line minimizes loss along each of them; the dashed line ignores the cost of repetition. Points are grid runs with

R\geq 2

and

U

within a factor 1.41 of the panel’s, colored by their loss shifted to this

U

by the fit. Stars mark the lowest predicted loss at each

U

, at the grid’s largest model for 1.5B; beyond it, more compute raises loss. (c) Epochs on the frontier; fitted up to the stars,

R_{\rm opt}\propto C^{0.47}

U=0.5

B and

C^{0.48}

at 1.5B; faded beyond them.

Figure 6: At fixed $U$ , larger models reach their loss minimum after fewer epochs, while a second epoch keeps its value. CommonCrawl, $U=1$ B, separately annealed runs. (a) Held-out loss; open markers, additional runs at $R=64$ (Appendix K ). (b) The full sweep: loss above each model’s lowest measured loss, $\Delta L$ (bpb), for every model size and epoch count (hatched, not run). White dots, parabola minima in $\log R$ for the five sizes whose loss turns upward; stars, lowest measured loss for the three smallest models (14M, 31M, 59M), which have not turned upward by $R=64$ . (c) $\eta$ at $R=2$ (filled) and $R=4$ (open), inverting a power law through each size’s one-epoch endpoints at $U=1$ B and 4B (Appendix Table 10 ).

Figure 7: When unique data grow with model size, the loss-minimizing epoch count barely changes. Full-corpus repetition at

U=2.5N

(so

D/N=2.5R

) for seven exact non-embedding sizes from 23M to 1.2B; test loss of separately annealed runs. (a) Epochs at the loss minimum, from a least-squares quadratic in

\log R

through all points of each size, and power laws

R^{*}\propto N^{\gamma}

through them. (b,c) Test loss (points) and a six-parameter surface fitted per source (

\log L

quadratic in

\log R

, with coefficients linear in

\log N

); diamonds mark the surface minimum for each size.

Figure 8: At

D=20N

, the cost of heavy partial repetition shrinks with size but persists. Excess loss with half of the budget from a subset repeated

R_{\rm sub}=8

(filled) or 2 (open) times; means with 95%

t

intervals over three paired replicates.

Design	Baseline	$z$ with size	Predicted	Observed
Fixed $U$ (§ 5.1 )	data-matched	$\propto N_{\rm total}$	minimum at fewer epochs	15.3 to 3.6 epochs, 127M to 2B
Fixed $U/N$ (§ 5.2 )	data-matched	$\propto N_{\rm total}/N$	minimum nearly fixed	3.7–4.4 epochs on CommonCrawl
Fixed $D/N$ , $f$ , $R_{\rm sub}$ (§ 5.3 )	compute-matched	$\propto N_{\rm total}/N$	22–39% less at 1.2B	33% and 21% less
Fixed $U$ , T5 ( Xue et al., 2023 )	compute-matched	$\propto N_{\rm total}$	larger models degrade more	larger models degrade more
$R=4$ , $D\approx 20N$ ( Muennighoff et al., 2023 )	compute-matched	$\approx 0.6$	$\Delta L\approx 0.01$ bpb	nearly as good as fresh data
Small subset, many repeats ( Hernandez et al., 2022 )	compute-matched	large	large excess loss	marked degradation

Table 1: The direction of each size trend matches how

z

changes with model size.

z=(R-1)N_{\rm total}/U

;

N_{\rm total}/N

falls from 2.1 at 23M to 1.08 at 1.2B. Predictions for the published results use Eq. ( 3 ).

Figure 9: Large loss increases appear under consecutive repetition, not per-epoch shuffling, at fixed

(N,U,R)

. Sweeps on CommonCrawl at

U=1

B. (a) Schematic of the three orders of the same data, drawn at shard level. (b) Loss of shuffled and consecutive runs minus that of the spaced run with the same

(N,U,R)

. (c) Unseen-minus-seen loss gap: consecutive order nearly removes the gap while raising loss, so a smaller gap does not by itself indicate better generalization.

Figure 10: Lower-entropy sources exhibit steeper post-minimum degradation. Seven SlimPajama sources trained separately (59M, $U=0.2$ B), each evaluated on its own held-out set. (a) Loss above each source’s minimum, in $10^{-3}$ bpb; rows are ordered by unigram entropy $H$ (bits/token, right), and boxes mark the loss-minimizing $R$ ( $R=1$ omitted). (b) Rise from minimum to $R=16$ and (c) unseen-minus-seen gap growth ( $R=1\to 16$ ) vs. unigram entropy; lines are least-squares fits with 95% confidence bands.

Figure 11: Re-tokenizing repeats raises loss at

R\leq 4

and lowers it at

R\geq 8

. (a) Final test loss: means and replicates. (b) Paired difference from always-off with pointwise 95%

t

intervals; light bars show simultaneous Bonferroni intervals for the policy contrasts.

R\leq 4

and

R\geq 8

use separate linear scales.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Single-run sweeps
Experiment	Models	Controlled	Varied	Sources
$D/N\times R$ grid (§ 4 )	23M–361M	$N$ , $D$	$D/N\leq 300$ , $R\leq 16$ ( $U=D/R$ )	CC
Fixed $U$ (§ 5.1 )	14M–2B	$U$	$N,R\leq 32$ ; $R=64$ (14M–59M)	CC
$U=2.5N$ (§ 5.2 )	23M–1.2B	$U/N=2.5$	$N$ , $R\leq 12$	CC, GitHub
Order (§ 6.2 )	59M–511M	$N$ , $U$ , $R$	order	CC
Source entropy (§ 6.2 )	59M	$N$ , $U$	source, $R\leq 16$	7 sources

Appendix

Table 2: The grid maps the landscape; the other designs test whether the map transfers. Sizes are

12Ld^{2}

proxies for the fixed-budget, fixed-

U

, partial-repetition, order and source-entropy sweeps, and exact non-embedding counts elsewhere. Paired studies share data order and initialization across the allocations they compare. CC: CommonCrawl. Mixed: all seven SlimPajama subsets in their natural proportions. 5 sources: CC, GitHub, arXiv, Books and Wikipedia.

Field	45M model	185M model
Exact non-embedding parameters	45,084,900	184,623,360
Tied embedding parameters	32,164,480	51,463,168
Exact total parameters	77,249,380	236,086,528
Rounded total size	77.2M	236.1M
Width / depth	640 / 10	1024 / 16
Width/depth ratio	64	64

Appendix

Table 3: Architectures of the paired 45M and 185M models.

F

counts analytical training FLOPs per target.

Study/model	LR tuple	$B$	$G$	Fixed budget / updates
Source / 45M	0.01/0.01/0.05	131,072	1	$D=805{,}306{,}368$ ; $K=6{,}144$
$D/N$ / 45M	0.02/0.02/0.05	131,072	1	$D=201{,}326{,}592$ ; $K=1{,}536$
$D/N$ / 185M	0.02/0.02/0.05	524,288	4	$D=824{,}180{,}736$ ; $K=1{,}572$
$U/N$ / 45M	0.02/0.02/0.05	131,072	1	$U=50{,}331{,}648$ ; $K=384R$
$U/N$ / 185M	0.02/0.02/0.05	524,288	4	$U=206{,}045{,}184$ ; $K=393R$

Appendix

Table 4: Batch and update settings of the paired runs.

B

is global targets/update;

G

is accumulation. Each GPU receives 16,384 targets per microbatch on eight GPUs. LR order is matrix/scalar/tied embedding. Partial-repetition rows have fixed

D

; full-corpus rows have fixed

U

and

D=RU

, with

R\in\{1,2,4,6,8,12\}

Main CommonCrawl model-size sweep
$d$	$L$	$d/L$	heads (KV)	$N$	$N_{\rm count}$ (M)	batch (tok)
384	8	48.00	6 (3)	14M	12.990	262,144
512	10	51.20	8 (4)	31M	28.859	524,288
640	12	53.33	10 (5)	59M	54.102	524,288
768	18	42.67	12 (6)	127M	116.848	524,288
1024	20	51.20	16 (8)	252M	230.779	524,288

Appendix

Table 5: The proxy family.

N

is the

12Ld^{2}

size label and

N_{\rm count}

the exact non-embedding count in millions; batch is in target tokens per update. All models have head dimension 64 and MLP width

4d

. The mixed-corpus ladder of the fixed-budget sweep (Figure 17 d) shares the 31M and 59M rows; its 252M model is also used in the 252M partial-repetition panel.

$R$	$L_{\rm Muon}$	$L_{\rm AdamW}$	$\Delta L_{\rm Muon}$	$\Delta L_{\rm AdamW}$
1	1.112899	1.152695	0.000	0.000
2	1.114845	1.153794	1.945	1.100
4	1.124383	1.157711	11.484	5.016
8	1.160769	1.189120	47.869	36.426
16	1.279985	1.283226	167.086	130.531
32	1.867810	1.760766	754.911	608.072

Appendix

Table 6: Optimizer sensitivity at one model size and token budget. Absolute mixed-corpus losses

L

are in bpb;

\Delta L=L(R)-L(1)

is computed relative to each optimizer’s own single-epoch endpoint, in

10^{-3}

bpb.

Contrast	Stratum	Mean (bpb)	95% CI
Concentration	Source mean	$+0.0039$	$[+0.0033,+0.0045]$
	CommonCrawl	$+0.0049$	$[+0.0039,+0.0059]$
	GitHub	$+0.0034$	$[+0.0018,+0.0049]$
	Books	$+0.0034$	$[+0.0023,+0.0046]$
Two sizes, $(50\%,8)$	Source mean	$-0.0138$	$[-0.0192,-0.0083]$
Two sizes, $(50\%,2)$	Source mean	$-0.0030$	$[-0.0093,+0.0032]$

Appendix

Table 7: Summary of paired contrasts. Concentration:

(33.3\%,4)

minus

(50\%,2)

at matched counts in the 45M study, six replicates (

df=5

). Two sizes: 185M minus 45M excess loss at matched

D/N

, three replicates (

df=2

Figure 12: The grid at every size, with values (cf. Figure 4 ). (a) Excess loss of $R$ epochs over $U=D/R$ against one epoch over $D$ fresh tokens at the same $N$ and $D$ , in units of $10^{-3}$ bpb. Black line, $R_{c}$ from the fresh-data fit plus Eq. ( 3 ) (solid lines in panel c); dashed, the 0.1-bpb level set of Eq. ( 3 ). (b) All 140 repeated configurations against $z$ ; open markers fall below the 0.002-bpb resolution threshold and are not fitted. (c) Critical epoch count $R_{c}$ , at which $\eta=\tfrac{1}{2}$ (Eq. 1 ): measured crossings, interpolated in $\log R$ (points); the fresh-data fit plus Eq. ( 3 ) (lines); and Eq. ( 4 ) fitted to the points (dashed).

Figure 13: Counting embeddings removes the size term. Grid excess loss (bpb) against

z=(R-1)N/U

with

N

the non-embedding (left) or total (right) parameter count.

Figure 14: The grid re-sliced by epochs and by budget. Excess loss of $R$ epochs over one epoch at the same $N$ and $D$ , in units of $10^{-3}$ bpb, (a) at fixed $R$ and (b) at fixed $D/N$ ; Figure 12 a gives the slices at fixed model size. Solid and dashed lines are the 0.01- and 0.1-bpb contours of Eq. ( 3 ); gray numbers fall below the 0.002-bpb resolution threshold.

Figure 15: An additive repetition term fits the grid; an effective-data form cannot reach the high-

z

runs. (a) Residuals in

\log L

of the joint additive fit (black) and of the joint effective-data fit of Qin et al. (2026) (orange) against

z

. (b,c) Test loss at fixed

D

against

R

for three budgets at 23M and 361M: measured runs (points) and the additive fit (lines). Shading spans every effective-data law with

0\leq\eta\leq 1

on the one-epoch backbone; its lower edge is

\eta=1

(fresh data) and its dotted upper edge

\eta=0

(one epoch over

U

), extrapolated where

U<5N

Figure 16: Iso-loss contours at fixed unique data. Test loss over epochs and model size at

U=0.5

B (a,b) and 1.5B (c,d). (a,c) Linear interpolation, in

\log N

and

\log R

, of grid runs whose

U

lies within a factor 1.41 of the target, each shifted to the target

U

by the fit. (b,d) The fit used for Figure 5 : one-epoch backbone plus Eq. ( 3 ), shown where

5\leq D/N\leq 300

. Contours every 0.01 bpb. Solid orange, the efficient frontier; dashed, the frontier if repeats were fresh; stars, the lowest measured or fitted loss. Runs differ from the fit by 0.011 and 0.002 bpb on average at the two

U

. The lowest measured loss is at 361M and

R=4

at both

U

; the fit places its minimum at

R\approx 5

–6.

Figure 17: At the 59M proxy size, a second epoch retains most of a fresh epoch’s value (c); at a fixed token budget, repetition raises loss at every model size (d). (a) Fixed-corpus repetition keeps model size

N

and unique tokens

U

fixed, so extra epochs add training tokens

D=RU

and compute. (b) Partial repetition keeps

N

and

D

fixed; a fraction

f

of training tokens comes from a subset seen

R_{\rm sub}

times, so

U=D(1-f+f/R_{\rm sub})

. (c) 59M proxy size: one-epoch fresh-data endpoints (circles) and a fixed

U=1.2

B ladder labeled by

R

(squares); shading marks the excess loss between them. The diamond is 25B repeated twice; dashed guides map it to

U_{\rm eq}=46.2

B on the reference

L=E+AD^{-\alpha}

fitted to the 6–50B endpoints (hollow circle), which gives

\eta

(Eq. 1 ); the inset resolves the 50B endpoints. (d) Excess loss of

R

epochs over

U=D/R

versus one epoch at the same model and

D

(3.2B; 4B for 252M/472M at

R\leq 8

; open markers give the two largest ladders at

D=3.2

R=16

and 32); the axis is linear below

0.002\text{ bpb}

(gray band). Five proxy sizes from 31M to 472M.

Figure 18: Partial repetition raises mean loss (a–c), and matched aggregate counts do not give matched loss (d). Paired study; 45M non-embedding parameters,

D\approx 805

M. (a–c) Excess loss relative to the same-source all-fresh run per allocation

(f,R_{\rm sub})

, where

f

denotes the fraction of training tokens drawn from the repeated subset: lines join source means with pointwise 95%

t

intervals (

n=3

); dots are paired replicates. (d) Concentration penalty: concentrated

(f=33.3\%,R_{\rm sub}=4)

minus spread

(f=50\%,R_{\rm sub}=2)

. Both allocations have equal training-token, distinct-target and total-extra-exposure counts; the concentrated one repeats a subset one third as large and is not shown in (a–c). Bars show per-source and equal-source mean penalties with 95%

t

intervals over six paired replicates (points). All 18 source–replicate differences are positive, and a sign-flip test on the six replicate means gives

p=0.031

, the smallest attainable.

Size	$N_{\rm nonemb}$	CC	GH	CC	GH	CC	GH
		Full corpus $R^{*}$		Partial $(50\%,8)$		Partial $(50\%,2)$
23M	23,087,168	3.72	3.06	–	–	–	–
45M	45,084,900	3.79	2.84	0.0322 $\pm$ 0.0021	0.0337 $\pm$ 0.0060	0.0038 $\pm$ 0.0033	0.0043 $\pm$ 0.0013
78M	77,898,384	4.03	2.75	–	–	–	–
185M	184,623,360	4.22	2.68	0.0249 $\pm$ 0.0012	0.0296 $\pm$ 0.0007	0.0024 $\pm$ 0.0023	0.0041 $\pm$ 0.0010
361M	360,563,600	4.30	2.66	0.0245 $\pm$ 0.0012	0.0270 $\pm$ 0.0023	0.0027 $\pm$ 0.0014	0.0029 $\pm$ 0.0004

Appendix

Table 8: Scale studies at fixed budgets per parameter. Full corpus: epochs at the loss minimum at

U=2.5N

(least-squares quadratic in

\log R

through all points of a size; one replicate). Partial repetition: test excess loss over the same-replicate fresh run at

D=20N

, mean

\pm

95%

t

half-width over three paired replicates. CC: CommonCrawl; GH: GitHub.

Figure 19: At matched

D/N

, the 185M model pays less for heavy partial repetition; at matched

U/N

, both sizes are best near four epochs. Two sources, three paired replicates per source; exact non-embedding parameters. (a) Matched

D/N

: excess loss under partial repetition for 45M vs. 185M models under light (

f=50\%,R_{\rm sub}=2

) and heavy (

f=50\%,R_{\rm sub}=8

) reuse. For heavy reuse, the larger-minus-smaller contrast is

-0.0138\text{ bpb}

, 21% of the smaller configuration’s penalty. (b,c) Matched

U/N

: test loss of separately annealed full-corpus runs; lines join replicate means. Diamonds mark the repeat count selected on development data.

$U=1$ B
$N$	$L(1)$	$R=2$	$R=4$	$R=8$	$R=16$	$R=32$
14M	1.2029	$-31.4$	$-51.1$	$-61.9$	$-71.6$	$-70.6$
31M	1.1642	$-36.6$	$-61.2$	$-76.1$	$-85.8$	$-90.9$
59M	1.1343	$-41.0$	$-67.5$	$-83.0$	$-90.0$	$-94.2$
127M	1.1060	$-45.9$	$-74.0$	$-88.2$	$-91.1$	$-87.3$
252M	1.0837	$-48.4$	$-76.5$	$-84.9$	$-76.1$	$-55.5$

Appendix

Table 9: Fixed-

U

ladders relative to each model’s own one-epoch run (CommonCrawl sweeps, CommonCrawl held-out bpb,

12Ld^{2}

proxy sizes).

L(1)

is the one-epoch loss; other columns give

L(R)-L(1)

10^{-3}

bpb, negative when the extra epochs help. The 511M/

R=4

entry is described in Appendix K ; dashes mark horizons not trained.

$N$	$L_{1}(1\text{B})$	$L_{1}(4\text{B})$	$\eta$ , $R=2$	$\eta$ , $R=4$
59M	1.1343	1.0605	0.95 [0.76, 1.05]	0.81 [0.75, 0.83]
252M	1.0837	0.9955	0.93 [0.75, 1.03]	0.72 [0.64, 0.75]
511M	1.0951	0.9788	1.01 [0.82, 1.12]	0.72 [0.65, 0.75]
1B	1.0696	0.9553	0.94 [0.76, 1.05]	0.59 [0.51, 0.63]

Appendix

Table 10: Value of a repeated token across proxy sizes (CommonCrawl,

U=1

12Ld^{2}

proxy sizes).

\eta

inverts a power law through each size’s one-epoch endpoints

L_{1}

U=1

B and

U=4

B; brackets give the range for exponents

a/2

and

2a

. The 511M/

R=4

entry is described in Appendix K .

Figure 20: Held-out-size check of the surface. Eq. ( 6 ) is fit to the 57 configurations with

N\leq 511

M and predicts all 16 held-out 1B/2B endpoints. (a) Measured versus predicted loss. (b) Measured-minus-predicted residuals versus

z

on a symmetric-log axis that includes the three

R=1

z=0

points. Filled symbols mark the ten low-reuse points, including the three at

R=1

, and hollow symbols the six high-reuse points.

Figure 21: The count-based surface captures the shape of the measured landscape at

U=1

B, but not its detail. (a) Piecewise-linear interpolation of the measurements (circles) in log coordinates. (b) The 73-point surface of Eq. ( 6 ), evaluated at its reported parameters on the same color scale and ending at the measured

R=32

boundary. Solid curves minimize fitted loss along fixed compute

6NUR

where the minimum is interior; the dashed line is a 20-tokens-per-parameter fresh reference.

Form	Penalty	Parameters	$R^{2}$	MAE (bpb)
Benefit-only	none	6	0.360	0.0884
Separable	$\lambda(R-1)^{p}(N/U)^{q}$	9	0.985	0.1142
Load-augmented	$\lambda z^{p}$	8	0.982	0.0311

Appendix

Table 11: Penalty forms on the 73 sweep endpoints.

R^{2}

is in-sample; MAE is over the 16 held-out-size 1B/2B endpoints (Figure 20 ). Parameter counts include the shared benefit terms; diagnostics are in Appendix G .

Figure 22: Partial repetition by domain and model size at fixed training-token budget (sweeps). Signed excess loss $L(f,R_{\rm sub})-L(0,1)$ relative to all-fresh training of the same domain, model and budget; the axis $R$ denotes $R_{\rm sub}$ . (a–e) Five domains at 59M, $D=2$ B. (f–h) CommonCrawl at 252M, 1B and 2B, $D=8$ B; the 252M model uses four KV heads, unlike its main-sweep counterpart. Dots are measured configurations; stars mark the lowest observed interior loss; crosses mark unmeasured configurations, whose color is interpolated. The solid contour marks 0.002 bpb and dashed lines the per-panel 0.01-bpb tolerance fits; contour intervals are nonlinear.

Figure 23: The excess loss of partial repetition builds up during training (development loss). Each point subtracts the same-source, same-replicate fresh run at the same training progress; the learning rate decays linearly over the shaded final 20%. (a) 45M partial-repetition study: mean over three sources and three replicates. (b) Concentration penalty

(33.3\%,4)

minus

(50\%,2)

per source and mean, three replicates. (c) Two-size study at matched

D/N

(

D\approx 4.5N

), mean over two sources and three replicates; lighter 45M, darker 185M.

Figure 24: Approximate IsoFLOP profiles at

U=1

B. Points are grouped by nearest half-octave in

\log_{2}C

, with at least three per bin and up to

8.33\%

actual-budget variation. (a) Retained observations and fitted curves at bin-center compute. Outlined circles mark six interior grid minima and triangles two boundary-limited minima. (b) Selected model size versus bin-center compute. The six-point slope is

0.697

. The dotted fit uses the three interior minima at

R=4

(slope

0.995

); the dashed line is a slope-

1/2

reference.

Primary contrast	Mean bpb	95% CI
20%-off minus On	$-0.0028$	$[-0.0042,-0.0014]$
50%-off minus 20%-off	$+0.0122$	$[+0.0114,+0.0129]$

Appendix

Table 12: Primary tokenization contrasts and actual budgets. Contrasts average the paired policy difference over the five horizons within each replicate, with 95% paired-

t

CIs (

n=3

). Right: training targets

D

and padded positions

P

relative to the deterministic segmentation

E

, as ranges over replicates and horizons.

Explore similar work

Jul 28, 2026cs.LG

Bridging Compute- and Data-Optimal Pretraining

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function,

η

, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit

η

for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of

η

implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

Tian Qin, Kimia Hamidieh, David Alvarez-Melis

Harvard University · MIT CSAIL

Pretraining Epoch Bridging Language

Ask PDF

Oct 4, 2026cs.CL

Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from sixteen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regression to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.

Ziyue WANG, T. Kanamori

Institute of Science Tokyo

Model Size

Ask PDF

Jun 2, 2026cs.LG

q0: Primitives for Hyper-Epoch Pretraining

Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text. But pretraining a single model saturates within a few passes, long before the compute budget is exhausted. We argue this calls for a conceptual shift from training a single model toward exploring a population of models and aggregating their predictions. We introduce hyper-epoch pretraining (q0), which turns a multi-epoch budget into a population of diverse models whose combined predictions reach a lower validation loss than a single refined model. q0 reduces to three core primitives. A cyclic schedule with anti-correlated learning rate and weight decay collects diverse models from a few parallel trajectories. Chain distillation trains each model against its predecessor so that model quality compounds across the population. A learned prior, fit on a held out set, selects and weights members for any inference budget. On a 1.8B-parameter model trained on 100M FineWeb tokens, q0 matches a strong 256-epoch ensemble baseline using only ~56 epochs (~4.6x fewer), or ~67 epochs (~3.8x fewer) when matched to the baseline's ensemble size, and continues to improve beyond it. These gains reach cumulative ~12.9x data efficiency under the Slowrun setting and transfer to downstream benchmarks. Crucially, the optimal allocation shifts with the budget, so we give prescriptive recipes for how to spend a given epoch budget to maximize generalization, from a single epoch up to the largest budgets.

Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1

Q Labs · Princeton University

Epoch Pretraining

Ask PDF