Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

Ziyue WANG; T. Kanamori

cs.CLOct 4, 2026

Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

Authors: Ziyue WANG, T. Kanamori

Organizations: Institute of Science Tokyo

Abstract

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from sixteen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regression to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.

Figures & tables

Figure 1: Repetition preferences change with model size. Here

r

counts passes over a fixed 100M-token target corpus; M denotes millions of model parameters. The vertical axis is held-out target cross-entropy in nats minus each size’s lowest measured mean. Stars mark these minima. Separate markers identify the three retained candidates selected from 35M–80M curves before 520M training. Bars show across-seed standard deviations (80M: three seeds; 520M: seven). Smaller-model losses use a selection split; 520M uses a separate test split. Lines connect measurements. Section 6.1 evaluates the candidates.

Figure 2: Measured loss versus repetition count

r

at 35M, 50M, and 80M. Each curve subtracts its own lowest mean held-out target loss, in nats. Markers show tested counts on the original size-dependent grids; stars mark the measured minima. The Wikipedia inset enlarges the shallow minima. Lines connect measurements. Appendices B.1 and B.2 report losses and variation across seeds.

Figure 3: Proof-Pile-2 at 520M. Left: mean loss reduction from 16 to 8, with standard-error bars, for two independent six-seed experiments. All six paired differences are positive in each experiment. Right: six seed curves from the first experiment, all satisfying

L(8)<L(4)<L(16)

; squares and labels give mean losses.

Prediction and evaluation grid	$r$	Excess loss	$r$	Excess loss
	PubMed		Caselaw
Original choice, seven counts	8	0.038	12	0
Original choice, twelve counts	8	0.044	12	0
Same function on integer $r$ , twelve counts	10	0.003	11	0.008

Table 1: 200M point choices and their measured excess loss in nats. The seven-count choices were fixed before training. Integer choices apply the unchanged prediction functions after refinement. Zero denotes selection of the lowest measured mean on the stated grid.

Figure 4: Predictions fixed from 35M/50M/80M and measured 200M loss versus repetition count

r

, each relative to its value at 12. No 200M parameters are fitted. Main panels show the valley; insets show the full range. Bars are sample SDs of three paired seed differences. Dashed lines mark predicted continuous minima, stars measured minima.

Figure 5: New 200M measurements predicted from seven counts. Equation ( 9 ), including its offset, is fitted only at

r\in\{4,8,12,16,24,32,48\}

. Diamonds mark the five excluded counts

\{6,9,10,11,14\}

used for RMSE. Main panels show the valley and insets the full range. Loss is absolute test loss in nats; bars are sample SDs across three seeds.

	New-count RMSE		Selection loss
Model	PubMed	Caselaw	PubMed	Caselaw
Adapted Lovelace model, Eq. ( 19 )	0.238	0.224	0.003	0.008
Same repetition terms, free $\eta$	0.159	0.184	0.003	0.008
Saturated extension, Eq. ( 9 )	0.108	0.074	0.006	0.018

Table 2: Matched fits to seven counts at 35M/50M/80M/200M/520M, including offsets. RMSE uses the five excluded 200M counts. Selection loss is the measured excess loss of the chosen count over the twelve-count minimum. All values are in nats.

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Target pool	100M English-Wikipedia-derived tokens: two 50M-token subpools, one clean-prose construction and one redundancy-injected construction
Generic training corpus	FineWeb-derived training stream; generic tokens fill the fraction $1-h=0.2$ of each main training configuration
Tokenizer and context	Fixed 32,768-token byte-level byte-pair encoding (BPE); context length 1024
Target evaluation	Mean over the two clean held-out target splits; each constituent split uses at most 2,097,152 tokens
Held-out generic loss	Fixed held-out generic data, used only for comparisons between configurations within this study
Candidate sets	35M: $\{1,4,12,24,36,48,60\}$ ; 50M: $\{1,4,8,16,24,32,48\}$ ; 80M: $\{1,4,8,12,16,24,36\}$

Appendix

Table A.1: Configuration of the Wikipedia-derived loss curves.

Item	35M/50M/80M experiments	520M experiments
Target corpus and tokenizer	Fixed 100M-token Proof-Pile-2 formal-mathematics/code target, validation split, and 32,768-token tokenizer	Same corpus, split, and tokenizer
Generic training corpus	Fixed FineWeb-derived training stream	Same corpus
Target evaluation	Fixed held-out target split; at most 2,097,152 tokens	Same split and maximum
Held-out generic loss	Fixed held-out generic data used for within-corpus comparisons	Same evaluation data
Candidate sets	35M: $\{6,12,24,48,64\}$ ; 50M: $\{4,8,16,32,48\}$ ; 80M: $\{4,8,12,16,24,36,48\}$	First experiment: $\{4,8,16\}$ ; second experiment: $\{8,16\}$
Seeds	Four per 35M model/count combination; three per 50M and 80M combination	Six paired seeds in each nonoverlapping seed set

Appendix

Table A.2: Cross-scale comparability of the Proof-Pile-2 experiments.

$r$	1	4	12	24	36	48	60
Mean $L_{35\mathrm{M}}(r)$	6.274	4.307	3.229	3.005	2.941	2.927	2.929
Seed SD	0.026	0.019	0.011	0.004	0.003	0.002	0.004
Seeds	4	4	4	3	4	4	4

Appendix

Table B.1: Completed 35M Wikipedia-derived loss curve. The

r=24

configuration has three completed seeds; all other repetition counts have four. “Seed SD” denotes the standard deviation across random seeds.

$r$	1	4	8	16	24	32	48
Mean $L_{50\mathrm{M}}(r)$	6.256	4.118	3.330	3.014	2.930	2.910	2.943
Seed SD	0.036	0.018	0.005	0.004	0.004	0.002	0.002
Seeds	3	3	3	3	3	3	3

Appendix

Table B.2: Completed 50M Wikipedia-derived loss curve. Every repetition count has three seeds.

$r$	1	4	8	12	16	24	36
Mean $L_{80\mathrm{M}}(r)$	6.225	3.818	3.155	2.977	2.912	2.905	3.044
Seed SD	0.015	0.062	0.013	0.003	0.001	0.005	0.009
Seeds	3	3	3	3	3	3	3

Appendix

Table B.3: Completed 80M Wikipedia-derived loss curve. Every repetition count has three seeds. Loss differences are calculated before rounding the displayed means.

Scale	$r$	Mean loss	Scale	$r$	Mean loss
35M	6	0.913	50M	4	1.021
35M	12	0.783	50M	8	0.795
35M	24	0.804	50M	16	0.788
35M	48	0.950	50M	32	0.978
35M	64	1.070	50M	48	1.243

Appendix

Table B.4: Completed Proof-Pile-2 loss curves at 35M and 50M.

Scale	$r$	Mean loss	Seed SD
80M	4	0.898	0.009
80M	8	0.762	0.004
80M	12	0.796	0.005
80M	16	0.866	0.007
80M	24	1.101	0.007
80M	36	1.497	0.015

Appendix

Table B.5: Completed Proof-Pile-2 loss curve at 80M. Each count has three seeds; “Seed SD” denotes the standard deviation across them.

Rule	Output at 80M	Interpretation
Direct reuse of 50M selection	$r=16$	Reuse the 50M minimum
Two-point log-linear extrapolation	24.22, mapped to $r=24$	Extrapolate the two selected counts, then use the nearest evaluated count
Nearby candidate set	$\{16,24,36\}$	Extrapolated center and its adjacent evaluated counts

Appendix

Table B.6: Predictions produced from the completed smaller-model experiments.

Figure B.1: Individual paired loss reductions

L(16)-L(8)

for the two independent 520M Proof-Pile-2 experiments. The twelve points are grouped by experiment, with six nonoverlapping seeds each; all are positive. Dashed lines mark the experiment means. Figure 3 summarizes each experiment with its mean and SE.

Seed	$L(8)$	$L(4)$	$L(16)$	$L(16)-L(8)$
1342	0.820	0.864	1.357	0.537
1343	0.846	0.942	1.374	0.528
1344	0.878	0.884	1.368	0.490
1345	0.856	0.941	1.377	0.521
1346	0.822	0.915	1.354	0.532
1347	0.834	0.955	1.373	0.539

Appendix

Table B.7: Seed-level results for the first 520M experiment.

Seed	$L(8)$	$L(16)$	$L(16)-L(8)$
1348	0.852	1.373	0.521
1349	0.848	1.364	0.516
1350	0.854	1.362	0.509
1351	0.853	1.350	0.497
1352	0.846	1.366	0.520
1353	0.880	1.369	0.489

Appendix

Table B.8: Seed-level results for the second 520M experiment.

Seed	$L(16)-L(4)$	$L(4)-L(8)$	Loss order
1342	0.493	0.044	$L(8)<L(4)<L(16)$
1343	0.432	0.095	$L(8)<L(4)<L(16)$
1344	0.484	0.007	$L(8)<L(4)<L(16)$
1345	0.436	0.084	$L(8)<L(4)<L(16)$
1346	0.439	0.093	$L(8)<L(4)<L(16)$
1347	0.418	0.121	$L(8)<L(4)<L(16)$

Appendix

Table B.9: Seed-level contrasts involving

r=4

in the first experiment.

Change	50M	80M	520M first experiment	520M second experiment
$16\to 8$	$-0.007$	0.104	0.524	0.509
$8\to 4$	$-0.227$	$-0.136$	$-0.074$	–

Appendix

Table B.10: Proof-Pile-2 differences

G_{b\to a}(N)=L_{N}(b)-L_{N}(a)

; positive values favor the lower count. The second 520M experiment uses new seeds and evaluates only 8 and 16.

$N$	$a\to b$	Target loss			Generic loss
		Matched	Additional	Final	Matched	Additional	Final
35M	$6\to 12$	$-0.079$	$-0.051$	$-0.130$	$-0.288$	$-0.245$	$-0.533$
35M	$12\to 24$	$-0.009$	$0.029$	$0.020$	$-0.137$	$-0.161$	$-0.298$
35M	$24\to 48$	$0.002$	$0.145$	$0.147$	$-0.062$	$-0.106$	$-0.168$
50M	$4\to 8$	$-0.139$	$-0.088$	$-0.227$	$-0.339$	$-0.326$	$-0.665$
50M	$8\to 16$	$-0.034$	$0.027$	$-0.007$	$-0.193$	$-0.202$	$-0.395$

Appendix

Table B.11: Decomposition at matched target-token exposure on Proof-Pile-2. “Matched” compares schedules at the same exposure, “Additional” measures further training within the longer run, and “Final” is their sum. Negative values favor the longer configuration.

Configuration	Target fraction	Repetitions	Target loss	Generic loss
High target fraction	$\approx 0.8$	$\approx 32$	0.978	4.262
Middle target fraction	$\approx 0.4$	$\approx 16$	0.748	3.904
Low target fraction	$\approx 0.2$	$\approx 8$	0.748	3.835

Appendix

Table B.12: Final losses in the fixed-horizon control. Actual target-token fractions and repetitions differ from the nominal values by less than 1%.

$r$	PubMed	Caselaw
4	$\mathbf{3.775}\pm 0.151$	$3.802\pm 0.138$
8	$\mathbf{2.339}\pm 0.010$	$\mathbf{2.707}\pm 0.062$
12	$\mathbf{2.552}\pm 0.013$	$\mathbf{2.649}\pm 0.014$
16	$3.194\pm 0.022$	$\mathbf{3.307}\pm 0.042$
24	$4.720\pm 0.017$	$5.141\pm 0.023$
32	$5.243\pm 0.027$	$5.775\pm 0.017$

Appendix

Table C.1: Completed 520M test curves on the additional corpora. Entries are seven-seed mean

\pm

standard deviation in nats. The preselected retained counts are bold.

Comparison	PubMed	Caselaw
Reuse the 80M winner	0.854	2.492
Keep the three lowest-loss 80M counts	0.213	0.658
Use the fixed set $\{8,12,32\}$	0	0
Exchange the selected sets between corpora	0	0

Appendix

Table C.2: 520M loss reduction from using the best count in each preselected set, relative to the best count in each baseline choice or set. All values are in nats.

$r$	PubMed mean $\pm$ SD	Caselaw mean $\pm$ SD	Stage
4	$2.752\pm 0.060$	$3.229\pm 0.062$	Original
6	$2.474\pm 0.002$	$2.784\pm 0.009$	Planned
8	$2.375\pm 0.002$	$2.610\pm 0.009$	Original
9	$2.348$ (SD $<0.001$ )	$2.567\pm 0.006$	Exploratory
10	$2.334$ (SD $<0.001$ )	$2.537\pm 0.004$	Planned
11	$\mathbf{2.332}\pm 0.003$	$2.523\pm 0.003$	Exploratory

Appendix

Table C.3: All completed 200M target-test results, in nats. Values are means and sample standard deviations across three seeds, not confidence intervals. “Planned” denotes the additional counts prepared before target results; “exploratory” denotes counts chosen after the main results. The lowest measured mean in each corpus is bold.

Corpus	Frozen fit	$r_{*}(N)$	$r_{7}$	$\Delta_{7}$	$\Delta_{12}$	Local RMSE
PubMed	Primary, $q=1$	9.659	8	0.038	0.044	0.021
	Free $q$	7.730	8	0.038	0.044	0.075
	Saturated	14.827	16	0.100	0.106	0.048
	35M/50M, $q=1$	10.534	12	0	0.006	0.015
Caselaw	Primary, $q=1$	11.431	12	0	0	0.019
	Free $q$	10.112	12	0	0	0.044

Appendix

Table C.4: All frozen point predictions.

r_{*}(N)

is the continuous function minimum;

r_{7}

is the count selected on the original grid.

\Delta_{7}

and

\Delta_{12}

are the measured excess losses of this same

r_{7}

relative to the lowest mean on

G_{7}

and

G_{12}

. Local RMSE compares predicted and measured

L(r)-L(12)

\{8,9,10,11,12,14,16\}

Target (M)	Sources (M)	PubMed	Caselaw
520	35, 50, 80	$\{4,8,12\}$	$\{8,12,16\}$
520	35, 50	$\{8,12,16\}$	$\{8,12,16\}$
520	35, 80	$\{4,8,12\}$	$\{8,12,16\}$
520	50, 80	$\{4,8,12\}$	$\{8,12,16\}$
80	35, 50	$\{16,24,32\}$	$\{16,24,32\}$

Appendix

Table C.5: Retained sets using all three available seeds at each listed source size. Every row retains the lowest target mean loss in both corpora. The full sensitivity analysis also includes seed subsets.

Corpus	Target (M)	Pruned set	Fixed set	Fixed minus pruned
PubMed	80	2.311	2.354	0.043
Caselaw	80	2.541	2.603	0.062
PubMed	520	2.339	2.339	0
Caselaw	520	2.649	2.649	0

Appendix

Table C.6: Lowest target mean loss available within each set, in nats. The fixed comparison set is

\{8,12,32\}

. The 80M rows retain the original selection-split evaluation; the 520M rows use test loss. Three source seeds per size are used.

Corpus	Sizes	$b\to a$	$\delta(b)-\delta(a)$	$m_{s}$	$G_{b\to a}$
Wikipedia	35M $\to$ 80M	$36\to 24$	0.203	0.064	0.139
Proof-Pile-2	50M $\to$ 80M	$16\to 8$	0.110	0.007	0.104

Appendix

Table D.1: Decomposition of the observed reversals, in nats. Both counts are measured at each size. Entries are computed before rounding.

Target corpus	Nominal $(B_{\mathrm{dec}},P_{\mathrm{inc}})$	Realized $(B_{\mathrm{dec}},P_{\mathrm{inc}})$	80M continuous minimum	Evaluated choice
Wikipedia-derived	$(4.431,\,0.011)$	$(4.431,\,0.011)$	$25.82\to 25.32$	$24\to 24$
Proof-Pile-2	$(1.690,\,0.018)$	$(1.683,\,0.017)$	$9.67\to 9.48$	$8\to 8$

Appendix

Table E.1: Post hoc fit to the completed 35M and 50M curves. The arrow in the continuous-minimum column shows the result using nominal parameter counts followed by the result using realized parameter counts.

Target corpus	Ordered bootstrap resamples	Available fits	Unavailable fits	Same 80M choice
Wikipedia-derived	6912	6885	27	6885/6885 select $r=24$
Proof-Pile-2	6912	6912	0	6912/6912 select $r=8$

Appendix

Table E.2: Stability of the fitted 80M choices under the paired bootstrap of completed smaller-model curves.

	Fit RMSE		Held-out RMSE
Corpus	Cap 4	Cap 8	Cap 4	Cap 8
Wikipedia-derived	0.020	0.020	—	—
Proof-Pile-2	0.015	0.015	—	—
PubMed	0.089	0.088	0.107	0.113
Caselaw	0.076	0.071	0.093	0.095

Appendix

Table E.3: Structural fit error in nats. Held-out columns withhold each of five interior repetition counts at all four new-corpus scales. The wider exponent limits improve reconstruction slightly but do not improve this interpolation diagnostic.

Corpus	$r_{*}$	$\alpha$	$\nu$	$s_{N}$	$\lambda_{N}$
Proof-Pile-2	6.026	1.938	2.383	0.275	0.233
PubMed	9.490	2.279	3.734	0.602	0.175
Caselaw	9.984	1.536	4.000	0.779	0.207

Appendix

Table E.4: Parameters for evaluating the central 520M regret expression directly. They summarize the fitted shape and do not use the size offset.

Corpus	Fitted $r_{*}$	$k_{N}$ (cap 4)	Retained-set regret	Unit-grid regret
PubMed	9.46–9.49	5.124	0.066–0.069	0.006–0.007
Caselaw	9.98–10.32	4.784	0.060–0.094	$<0.003$

Appendix

Table E.5: 520M quantities derived from the fitted two-effect law. Regret is relative to the unmeasured continuous model minimum, in nats. Ranges span the two exponent-bound choices, not confidence intervals.

Figure E.1: The same two-effect family fitted to all four corpora. Points are completed-run means. Solid lines show the central fit; dotted lines show the wider-bound sensitivity fit, both using Equation 9 . Agreement at the largest size here is reconstruction using those outcomes.

Figure E.2: Compute envelopes and repetition-loss differences from the same empirical model. (a,b) Fits for 35M, 50M, and 80M on the common selection split; the black curve is their lower envelope within the tested repetition ranges, using

6ND

as a compute proxy. (c,d) Independent 520M test losses relative to

r=8

on PubMed and

r=12

on Caselaw, compared with fitted differences anchored at the same respective counts. Stars mark untested continuous model minima; dotted curves show wider exponent limits. Shading spans the candidate intervals selected before 520M training. The curve model was fitted after the experiments.

Corpus	Lovelace shape	Only $\eta$ freed	Saturating extension
Wikipedia-derived	0.066	0.034	0.020
Proof-Pile-2	0.087	0.068	0.015
PubMed	0.258	0.167	0.089
Caselaw	0.263	0.202	0.076

Appendix

Table E.6: Fits to all available pre-200M configuration means. Entries are RMSE in nats. The last column reproduces the existing central fit from Section E.2 ; all three columns use target-scale observations and describe fitted curves.

Corpus	Lovelace shape	Only $\eta$ freed	Saturating extension
PubMed	0.336	0.236	0.107
Caselaw	0.371	0.340	0.093

Appendix

Table E.7: RMSE on repetition counts omitted at every model size, in nats. The saturating-extension column is the earlier leave-count-out analysis from Section E.2 , included as a reference.

	$\eta=0$			Free $\eta$
Corpus	$q$	RMSE	$q\leq 0$	$q$	RMSE	$q\leq 0$
Wikipedia-derived	0.996	0.071	0.099	1.579	0.046	0.099
Proof-Pile-2	1.280	0.034	0.163	1.360	0.034	0.060
PubMed	1.309	0.015	0.093	1.292	0.015	0.077
Caselaw	1.175	0.021	0.088	1.170	0.021	0.067

Appendix

Table E.8: Sign check using the 35M, 50M, and 80M curves. Error columns are RMSE in nats; “

q\leq 0

” denotes the best nonpositive profile value.

Corpus	Fitted sizes (M)	$\eta=0$	Free $\eta$
PubMed	35, 50	[1.103, 1.384]	[0.894, 1.082]
	35, 50, 80	[1.285, 1.334]	[1.263, 1.320]
	All	[0.728, 0.772]	[0.647, 0.718]
Caselaw	35, 50	[0.860, 0.998]	[0.688, 0.941]
	35, 50, 80	[1.132, 1.213]	[1.123, 1.212]
	All	[0.845, 0.887]	[0.819, 0.887]

Appendix

Table E.9: Empirical 2.5th–97.5th percentile ranges of

q

from resampling complete seed curves. “All” includes 520M. Each row uses 1,000 refits for

\eta=0

and 300 for free

\eta

Corpus	Fitted sizes (M)	Best $q$	RMSE	Best $q\leq 0$ RMSE
Wikipedia-derived	35, 50	0.659	0.016	0.016
	35, 50, 80	0.826	0.020	0.041
Proof-Pile-2	35, 50, 80	$4.000^{*}$	0.012	0.055
	All	3.523	0.015	0.065
PubMed	35, 50, 80	$4.000^{*}$	0.003	0.074
	All	$4.000^{*}$	0.089	0.215

Appendix

Table E.10: Sign check in the original saturating function family. RMSEs are in nats; an asterisk marks the upper bound

q=4

. “All” means all pre-200M sizes available for that corpus.

Corpus	Source $\to$ target (M)	Predicted $r$	Measured best $r$	Excess loss
Wikipedia-derived	$35,50\to 80$	24	24	0
Proof-Pile-2	$35,50\to 80$	8	8	0
PubMed	$35,50\to 80$	16	16	0
Caselaw	$35,50\to 80$	24	24	0
Proof-Pile-2	$35,50\to 520$	4	8	0.074
PubMed	$35,50\to 520$	4	8	1.435

Appendix

Table E.11: Retrospective held-size predictions from the simple model. Excess loss is the selected count’s target mean loss minus the minimum over the target’s evaluated counts, in nats.

	PubMed		Caselaw
$r$	Predicted	Measured	Predicted	Measured
8	$-0.006$	$0.038$	$0.058$	$0.095$
10	$-0.020$	$-0.003$	$0.007$	$0.022$
11	$-0.013$	$-0.006$	$(-0.001,0)$	$0.008$
16	$0.097$	$0.100$	$0.058$	$0.073$
48	$1.575$	$2.896$	$1.425$	$2.926$

Appendix

Table E.12: Frozen primary-model predictions and measured mean loss differences relative to

r=12

, in nats. Values smaller in magnitude than 0.001 are shown as such to preserve their sign.

Corpus	Predicted interval	Measured counts inside	Measured low-loss counts
PubMed	$[7.730,11.973]$	$\{8,9,10,11\}$	$\{9,10,11,12\}$
Caselaw	$[9.314,13.936]$	$\{10,11,12\}$	$\{11,12,14\}$

Appendix

Table E.13: The primary frozen model’s interval with continuous model regret at most 0.020 nats, compared with measured counts at most 0.020 above the

G_{12}

minimum. Membership uses unrounded endpoints. The continuous intervals are predictions, not confidence intervals.

Corpus	Earlier fit	$\widehat{r}_{*}$	Choice	Excess loss	Full RMSE	Local RMSE
PubMed	Bound 4	11.214	11	0	0.375	0.048
	Bound 8	9.525	10	0.003	0.598	0.171
Caselaw	Bound 4	13.181	14	0.018	0.317	0.070
	Bound 8	11.639	12	0	0.598	0.068

Appendix

Table E.14: Unchanged earlier empirical coefficients evaluated at 200M. The choice and its measured excess loss refer to

G_{12}

; the RMSE columns compare

L(r)-L(12)

. The local grid is the seven measured counts between 8 and 16.

Check	PubMed	Caselaw
Joint fit, all 40 means: training RMSE	0.115	0.089
Joint fit, all 12 target means: 200M training RMSE	0.129	0.083
Joint fit, seven target counts: absolute prediction RMSE on five added counts	0.108	0.074
Leave one count out at every size: RMSE over 30 predictions	0.127	0.118
Same leave-count-out fits: RMSE on ten 200M interior counts	0.111	0.095
Same leave-count-out fits: RMSE on five added 200M counts	0.089	0.077

Appendix

Table E.15: Checks of the same saturated empirical family. All errors are in nats. Training errors, prediction errors, and relative-loss errors are separated explicitly; the held-count rows use training-only intercepts. The joint fits use exponent bound 4.

$r$	Lovelace	$+\eta$	Saturated	Lovelace	$+\eta$	Saturated
	PubMed			Caselaw
6	0.219	0.203	0.069	0.162	0.208	0.108
9	0.226	0.103	$-0.084$	0.206	0.148	$-0.006$
10	0.235	0.115	$-0.105$	0.222	0.157	$-0.032$
11	0.242	0.134	$-0.122$	0.236	0.169	$-0.056$
14	0.267	0.207	$-0.144$	0.279	0.225	$-0.106$

Appendix

Table E.16: Prediction minus measured loss on the five 200M counts excluded from the matched fits, in nats. Intercepts are estimated only from training counts. “

+\eta

” changes only the scale-amplitude exponent; “saturated” is the joint empirical extension.

Work	Selected decision	From inexpensive evidence to intended use	Run construction and repetition
Kaplan et al. (2020) ; Hoffmann et al. (2022)	Model size and token allocation	Model size, data, and compute	Processed tokens are treated as fresh
Repeated-data scaling ( Hernandez et al., 2022 ; Xue et al., 2023 ; Muennighoff et al., 2025 ; Xu et al., 2026 )	Useful epochs or finite-data allocation	Unique data, repeats, and model size	Repetition is explicit in an effective-data law
Liu et al. (2025)	Data-mixture weights	Small proxy models to a larger run	Proxy regression over independently trained mixtures
Sedova et al. (2026)	Mixture and repetition-aware minimum	Model size, token count, target size, and mixture	Candidate losses come from intermediate states of one run
Zhou et al. (2026)	Mixture at a target token horizon	Token horizon and model size	Repetition is matched across training runs
DataDecide; AutoScale ( Magnusson et al., 2025 ; Kang et al., 2025 )	Corpus ranking or domain weights	Model size or data scale	Completed corpus runs or domain-weight extrapolation

Appendix

Table F.1: Comparison with nearby scaling-law and data-selection studies. The third column states how evidence is used to choose a more expensive configuration.

Question	Observation	Supported conclusion
Does a better 35M curve fit ensure transfer?	Conditions with small leave-one-out fit error still lose 0.032–0.035 nat at 50M.	Fit quality at one scale does not determine the transferred count (Section G.4 ).
Does gradient alignment predict the loss increase?	The fitted change is 0.001 nat per standard deviation, with an approximate 95% interval of $[-0.002,0.005]$ .	No large association is observed in this twelve-condition design; an exactly zero effect is not established (Section G.3 ).
Do simple text-overlap and duplication statistics predict the loss increase?	Lexical overlap is driven by two high-leverage conditions; longer $n$ -gram overlap is nearly constant; 8-gram redundancy is correlated with the corpus construction.	None supplies a reliable prediction rule here (Section G.3 ).
Does changing the generic corpus remove the failure?	All twelve conditions select 48 at 35M and 32 at 50M, with 0.032–0.042 nat loss increases from reuse.	The tested generic-corpus substitutions do not remove the transfer failure (Section G.3 ).
Do additional natural corpora extend the result?	Their gradient-alignment values cover the constructed range, but they have no loss curves across repetition counts.	They contextualize the measurement only (Section G.6 ).
Does higher repetition degrade performance on both corpus categories?	Held-out generic loss continues to improve in the matched-exposure comparisons.	The target-loss increase is not accompanied by an increase on the measured generic data (Section B.4 ).

Appendix

Table G.1: Questions addressed by the secondary Wikipedia-derived analyses.

Condition	$\kappa$	Gradient alignment	Lexical overlap	Longer $n$ -gram overlap	8-gram redundancy	Seeds
Semantic 0.00	0.00	0.209	0.228	0.005	0.008	4
Semantic 0.25	0.25	0.217	0.178	0.005	0.008	2
Semantic 0.50	0.50	0.222	0.179	0.005	0.008	2
Semantic 0.75	0.75	0.235	0.180	0.005	0.008	2
Semantic 1.00	1.00	0.239	0.178	0.005	0.008	4
Entity 0.00	0.00	0.393	0.189	0.005	0.007	4

Appendix

Table G.2: Measured properties of the twelve generic corpora. “Semantic” varies document proximity; “entity” varies target-preferred entity surface forms.

Condition	Seeds	35M selection	50M selection	Loss increase from reusing 48
Semantic 0.00	4	48	32	0.042
Semantic 0.25	2	48	32	0.034
Semantic 0.50	2	48	32	0.033
Semantic 0.75	2	48	32	0.032
Semantic 1.00	4	48	32	0.037
Entity 0.00	4	48	32	0.036

Appendix

Table G.3: Transfer results for the twelve generic corpora. All conditions select 48 at 35M and 32 at 50M.

Term	Coefficient	Standard error	$t$ statistic
Intercept	-0.135	0.135	-1.00
Construction strength $\kappa$	0.001	0.002	0.75
Gradient alignment $m_{c}$	0.019	0.016	1.18
Lexical overlap	0.233	0.088	2.64
Longer $n$ -gram overlap	24.54	23.31	1.05
8-gram redundancy	-0.078	0.037	-2.11

Appendix

Table G.4: Weighted regression of transfer loss on the constructed corpus properties.

Figure G.1: The descriptive lexical-overlap association is driven by two conditions outside the narrow range occupied by the other ten. Labels identify the two semantic conditions with unusually high overlap; hollow markers denote the duplication controls. The vertical axis uses the same nat loss differences as Table G.3 . The figure is a sensitivity diagnostic, not evidence for a lexical mechanism.

Rule	Count at 50M	Loss increase	Count at 80M	Loss increase
Fixed $r=4$	4	1.208	4	0.914
Fixed $r=8$	8	0.419	8	0.251
Fixed $r=16$	16	0.104	16	0.008
Fixed $r=24$	24	0.020	24	0
Fixed $r=32$	32	0	36 †	0.139
Reuse 35M selection	48	0.033	36	0.139

Appendix

Table G.5: Simple rules applied to the measured Wikipedia-derived curves. Loss increase is relative to the lowest measured mean at each scale:

r=32

at 50M and

r=24

at 80M.

Figure G.2: Leave-one-out fit error at 35M versus the measured 35M-to-50M loss increase. Conditions with the smallest fit errors still transfer

r=48

, which has higher 50M loss than

r=32

. The dashed line marks the

6.4\times 10^{-3}

-nat fit-error threshold; circles and squares distinguish the semantic and entity families, with hollow markers for duplication controls. The plot concerns this fitted baseline; it is not a claim that every possible scaling model must fail.

$K$ minibatches	1	8	32	64	128
Target-to-target alignment	0.059	0.320	0.619	0.770	0.868
Unpreconditioned cosine	0.41	0.85	0.96	0.98	0.99
FineWeb-to-target alignment	0.025	0.124	0.229	0.278	0.298

Appendix

Table G.6: Gradient-alignment estimates at different accumulation depths for one saved 35M model state. Target-to-target uses independent target minibatches; FineWeb-to-target uses the main generic corpus.

Figure G.3: Measured target-to-target gradient alignment as accumulation depth increases, together with

1/(1+19.5/K)

. This checks measurement noise; it does not test whether a count selected at one model size remains preferable at another.

Corpus	Mean alignment	Seed SD	Relation to the constructed ranges
Project Gutenberg	0.105	0.005	Below semantic range
OpenAssistant-1	0.184	0.003	Below semantic range
FineWeb generic	0.222	0.007	Inside semantic range
Unmodified base corpus	0.232	0.004	Near semantic range
Harvard USPTO Patent Dataset (HUPD) claims	0.271	0.013	Between constructed ranges
Raw Wikipedia	0.307	0.004	Between constructed ranges

Appendix

Table G.7: Target–generic gradient alignment at the fixed Wikipedia-derived reference model state defined in Section G.5 , using

K=32

and five estimation seeds. “Seed SD” denotes the standard deviation across estimation seeds.

Explore similar work

Jun 23, 2026cs.LG

Internal Data Repetition Destroys Language Models

Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition. Earlier controlled studies predated Chinchilla-style scaling laws and could only measure the cost of repetition indirectly. We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss. We show that under this modernized paradigm, repetition damage is systematic in three ways. First, holding compute allocated to repeated data constant, eval loss peaks at an intermediate repeat count

\Rep

; repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. Second, the location of this peak is well-fit by a power law in model size; this scaling law reveals that the most damaging number of repeated data grows more quickly than compute. Finally, when repeated documents consume 10% of the FLOPs budget in a controlled exact-document repetition setting, the compute-equivalent loss can be large: on FineWeb-Edu-Dedup, the most damaging repeat count for a Qwen3-style 344M-parameter model at

\OT=1

matches the loss of a no-repetition run using 67% of the FLOPs. We demonstrate that these phenomena are not language-model-specific, and can be analytically understood in a simple statistical model: a misspecified linear regression with verbatim duplicates reproduces the same qualitative loss peak, quantifying how such peaks can arise from a statistical tradeoff between memorization and generalization. Our findings add precision to the study of duplication in language models, allowing practitioners to quantify the wasted compute incurred by the presence and repeat structure of duplicates in pretraining corpora.

Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6

1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1

Repetition Scaling Laws

Ask PDF

May 12, 2026cs.LG

Scaling Laws for Mixture Pretraining Under Data Constraints

As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality-filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15-20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints.

Anastasiia Sedova, Skyler Seto, Natalie Schluter +1

Apple · ITU

Pretraining Scaling Laws Mixtures Target Domain Text Corpora

Ask PDF

Oct 4, 2026cs.CL

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters. We answer them by pricing a repeated token against two references: one epoch on the same data, which gives its value, and fresh data at equal compute, which gives its cost. Against fresh data, the cost of repetition follows a single variable, the number of extra epochs divided by the unique tokens per parameter. Against the same data, a second epoch is worth nearly as much as a fresh one, and repeated tokens fall to half the value of fresh ones after a critical epoch count that grows with the training budget per parameter but hardly with model size. With unique data fixed, the predicted compute-optimal run grows model size and epochs together until loss stops improving, near the critical epoch count. The same variable accounts for the direction of size trends that appear to conflict: larger models tolerate fewer epochs when the corpus is fixed, from about 15 at 127M to 4 at 2B parameters, but not when unique data grow with the model. Counts alone do not determine loss: at identical counts, replaying shards consecutively raises loss by up to 0.46~bits per byte, concentrating repeats on fewer samples also raises it, lower-entropy sources degrade faster with repetition, and re-tokenizing repeats helps only under heavy repetition. These results offer an empirical guide to pretraining when unique data, rather than compute, are the binding constraint.

Yekun Chai, Haoyi Xiong

FloatAI · ETH Zurich · Independent Researcher

Epoch Pretraining

Ask PDF