cs.LGOct 11, 2026

Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Authors: Sewon Min

Organizations: UC Berkeley & Allen Institute for AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.

Figures & tables

Figure 1: (a) Batch-tuned infinite-compute frontier for a 153M model on the 1B-token data pool. Batch tuning and the repetition-tuned recipe progressively improve the validation loss. (b) Separating GPU availability from training time yields four distinct compute regimes. (c) An example with a 1.5B model on the 1B-token pool: even when time-constrained, a smaller batch size with fewer GPUs can achieve lower validation cross-entropy (CE) loss than a larger batch size with more GPUs, especially as compute becomes abundant.
Figure 2: Batch size has an interior optimum on the 1B-token pool. At each (model size, batch size) pair, we tune (learning rate, weight decay, epoch count), and report the lowest validation CE.
 CFC_{\mathrm{F}} (equiv. EE )Batch size (million tokens)
Bi et al.0.90×10190.90\times 10^{19} ( E=1E=1 )0.463
Bi et al.7.20×10197.20\times 10^{19} ( E=8E=8 )0.914
Bi et al.18.0×101918.0\times 10^{19} ( E=20E=20 )1.233
Shuai et al.0.90×10190.90\times 10^{19} ( E=1E=1 )0.551
Shuai et al.7.20×10197.20\times 10^{19} ( E=8E=8 )0.681
Shuai et al.18.0×101918.0\times 10^{19} ( E=20E=20 )0.747
Table 1: Batch-size comparison for the 1.5B model on the 1B-token pool. CF≈6NEDuC_{\mathrm{F}}\approx 6NED_{\mathrm{u}} denotes compute consumed; batches are in millions of tokens.
Figure 3: Training and validation losses across batch sizes. The large batch ( B=512B=512 ) sees a larger train-validation gap and earlier validation degradation, consistent with weaker stochastic regularization.
Figure 4: Data-loader ablation. (a) Best validation loss and its selected epoch for each batch and loader, with learning rate and weight decay tuned for each. (b) Validation and train losses for B=512B=512 ; stars indicate each method’s best observed validation checkpoint.
Figure 5: Weight-tying and embedding-decay ablation with Repack . (a) Best validation loss and its selected epoch for each batch size and method, with learning rate and weight decay tuned for each. (b) Validation and train losses for B=64B=64 ; stars indicate each recipe’s best validation checkpoint.
Figure 6: Comparison between the original and repetition-tuned recipes ( Repack , tied embeddings, positive embedding decay). The repetition-tuned recipe improves both validation CE (top) and HellaSwag BPB (bottom) across model sizes and batch sizes.
Figure 7: The repetition-tuned recipe improves as the unique-data pool grows. We report the best observed validation CE on 0.333B-, 1B-, and 3B-token pools. Orange diamonds show the tuned original recipe on the 1B-token pool, and purple squares show its untuned B=512B=512 baseline. Labels give the selected batch size and epoch.
Figure 8: The time-constrained frontier jointly selects the training recipe, batch size, and the number of repetitions on the 1B-token data pool. The black curve shows the best validation CE attainable at each optimizer-step deadline; BT⋆B_{T}^{\star} indicates the batch that supplies the frontier.
Figure 9: Joint GPU-time-constrained frontier for the original and repetition-aware recipes on the 1B-token data pool. Each cell shows the lowest observed validation loss achieved using exactly the indicated number of GPUs within the deadline. Colors range from blue (lower loss) to red (higher loss), normalized separately for each model size on a log scale. The black boundary indicates where the best observed unconstrained loss is reached; beyond it, neither additional time nor additional GPUs improve validation loss.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

ModelNuntiedN_{\mathrm{untied}}NtiedN_{\mathrm{tied}}Layersdmodeld_{\mathrm{model}}Headsdffd_{\mathrm{ff}}
153M153,104,896101,724,6721251282,048
474M473,990,144371,229,696161,024164,096
1.5B1,484,851,2001,279,330,304162,048168,192
Appendix
Table 2: Model architectures. Untied counts define the nominal model labels; tied counts are the trainable parameter counts used as NN in our repetition-aware scaling laws. dmodeld_{\mathrm{model}} is the residual width and dffd_{\mathrm{ff}} is the SwiGLU feed-forward width.
Batch sizeLRWDEval. epochPool APool BPool CAcross-pool SD
325×10−45{\times}10^{-4}0.3203.000523.000763.000930.00021
641×10−31{\times}10^{-3}0.3202.977132.977332.975940.00075
1281×10−31{\times}10^{-3}0.3162.998392.999362.996280.00157
2562×10−32{\times}10^{-3}0.3123.036893.042923.041670.00318
Appendix
Table 3: Impact of different random seeds in the 1B-token unique data pools. SD indicates standard deviation.
Figure 10: Unique-data equivalent of repeated-data training. Curves show validation CE for unique-data training at three batch sizes. Horizontal lines mark the best repeated 1B-token results with and without tuning BB . With tuning, repeated-data training matches 6×6\times – 8×8\times unique data at B=64B=64 and ∼16×\sim 16\times at B=1024B=1024 ; without tuning, it matches ∼8×\sim 8\times at B=1024B=1024 .
Figure 11: HellaSwag and aggregate downstream performance. (a) Performance versus batch size. Each best available epoch was selected based on validation CE. (b) Performance across epochs at B=64B=64 .
   HellaSwag8-task average
StepsMethodVal. CE ↓\downarrowAcc. ↑\uparrowBPB ↓\downarrowAcc. ↑\uparrowBPB ↓\downarrow
1,907B=512B=512 (E4)3.22135.020.937442.701.4018
1,907B=1,024B=1{,}024 (E8)3.22636.410.938642.551.4265
1,907B=1,024B=1{,}024 w. DiLoCo (E8)3.21738.090.933143.171.3831
2,861B=512B=512 (E6)—————
2,861B=1,024B=1{,}024 (E12)3.16037.640.919143.481.3673
Appendix
Table 5: Matched-optimizer-step comparison of baselines and DiLoCo. All methods use Repack . Bold marks the best available result within each step budget.
 Conventional throughput-first H100Latency-first H100
ModelmmDPTPPPGPUsTstepT_{\rm step}Tok./s/GPUmmDPTPPPGPUsTstepT_{\rm step}Tok./s/GPU
1.5B812811128∼0.60\sim 0.60 s54.654.6 k11024111024∼0.22\sim 0.22 s18.618.6 k
8B251211512∼1.00\sim 1.00 s8.198.19 k11024212048∼0.60\sim 0.60 s3.413.41 k
32B11024212048∼1.95\sim 1.95 s1.051.05 k11024428192∼0.70\sim 0.70 s0.730.73 k
70B11024414096∼2.10\sim 2.10 s0.490.49 k110248432768∼0.56\sim 0.56 s0.230.23 k
Appendix
Table 6: H100 projections for throughput-first and latency-first compute-parallelism configurations at the same global batch size, B=1,024B=1{,}024 sequences, with gradient accumulation a=1a=1 . Throughput and latency are simulated using a model that accounts for communication overhead. See § F for details.
Figure 12: Optimal batch size at each GPU count and deadline on the 1B-token pool, both for the original (left) and the repetition-aware recipe (right). Colors indicate the batch size of the observed configuration with the lowest validation CE that uses exactly the indicated number of H100 GPUs and finishes within the optimizer-step-equivalent deadline, with microbatch size held fixed. The black boundary indicates the region where the globally optimal loss under unconstrained GPU and time budgets is reached; beyond it, moving right (more time) or up (more GPUs) does not improve the validation loss.
BBLRWDEECE
165 ⁣× ⁣10−45\!\times\!10^{-4}0.1203.017
325 ⁣× ⁣10−45\!\times\!10^{-4}0.3203.001
6410−310^{-3}0.3202.977
12810−310^{-3}0.3162.998
2562 ⁣× ⁣10−32\!\times\!10^{-3}0.3123.037
5122 ⁣× ⁣10−32\!\times\!10^{-3}0.383.124
Appendix
Table 7: Hyperparameters for the batch-size sweep in Figure 2 (§ 3 ), all on a 1B-token pool. The 1.5B, B=64B=64 row also supplies our observed result in the batch-size comparison in Table 1 . All runs use the original recipe.
EELRWDCE
110−310^{-3}0.14.000
210−310^{-3}0.33.469
410−310^{-3}0.33.224
82 ⁣× ⁣10−3\bm{2\!\times\!10^{-3}}0.33.124
1210−310^{-3}0.33.198
Appendix
Table 8: Examples of selected hyperparameters across training horizons for three model sizes on the 1B-token pool, corresponding to Figure 2 (§ 3 ). All evaluated epochs through the first checkpoint after the validation loss saturates or reverses are shown. Bold marks the best checkpoint. All runs use the original recipe.
EELRWDCE
15 ⁣× ⁣10−45\!\times\!10^{-4}0.13.420
25 ⁣× ⁣10−45\!\times\!10^{-4}0.13.260
45 ⁣× ⁣10−45\!\times\!10^{-4}0.13.147
85 ⁣× ⁣10−45\!\times\!10^{-4}0.13.066
125 ⁣× ⁣10−45\!\times\!10^{-4}0.13.034
165 ⁣× ⁣10−45\!\times\!10^{-4}0.13.021
Appendix
Table 9: Hyperparameters for every plotted checkpoint in Figure 3 (§ 3 ), with the 1.5B model and a 1B-token pool. Each epoch uses its own selected LR/WD pair. All runs use the original recipe.
LoaderLRWDEECE
Fixed10−310^{-3}0.3163.011
Shuffle10−310^{-3}0.3202.977
Repack10−310^{-3}0.3242.971
Appendix
Table 10: Configurations for the data-loader comparison table in Figure 4 (a) (§ 4 ), with the 1.5B model and a 1B-token pool. All runs use untied embeddings and no embedding decay.
EELRWDCE
110−310^{-3}0.34.000
210−310^{-3}0.33.467
410−310^{-3}0.33.240
810−310^{-3}0.33.205
1210−310^{-3}0.33.248
Appendix
Table 11: Hyperparameters for every plotted checkpoint in Figure 4 (b) (§ 4 ), with the 1.5B model, a 1B-token pool, and B=512B=512 . All runs use untied embeddings and no embedding decay.
ConfigurationWD EmbLRWDEECE
Shuffle , untied010−310^{-3}0.3202.977
Repack , untied010−310^{-3}0.3242.971
Repack , tied010−310^{-3}0.3242.971
Repack , untied0.310−310^{-3}0.3322.951
Repack , tied0.310−310^{-3}0.3322.946
Appendix
Table 12: Configurations for the weight-tying/embedding-decay comparison table in Figure 5 (a) (§ 4 ), with the 1.5B model and a 1B-token pool.
EELRWDCE
110−310^{-3}0.33.397
210−310^{-3}0.33.229
410−310^{-3}0.33.112
810−310^{-3}0.33.027
1210−310^{-3}0.32.994
1610−310^{-3}0.32.978
Appendix
Table 13: Hyperparameters for every plotted checkpoint in Figure 5 (b) (§ 4 ), with the 1.5B model, a 1B-token pool, B=64B=64 , and Repack .
BBLRWDEECE
6410−310^{-3}0.3322.946
12810−310^{-3}0.3202.979
25610−310^{-3}1403.019
51210−310^{-3}1283.030
Appendix
Table 14: Hyperparameters for the repetition-tuned recipe across all model sizes and batch sizes, in Figure 6 (§ 4 ), on a 1B-token pool. The same checkpoints were used for the validation loss and the Hellaswag BPB. Original-recipe configurations are in Table 7 .
DuD_{\mathrm{u}}BBLRWDEECE
0.3336410−310^{-3}1283.255
16410−310^{-3}0.3322.946
312810−310^{-3}0.3802.767
Appendix
Table 15: Hyperparameters for every repetition-tuned point in the data-pool-size comparison of Figure 7 (§ 4 ). DuD_{\mathrm{u}} is the unique-data pool size in billions of tokens.
More in this topic

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

Explore similar work

CardsList
Oct 4, 2026

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters. We answer them by pricing a repeated token against two references: one epoch on the same data, which gives its value, and fresh data at equal compute, which gives its cost. Against fresh data, the cost of repetition follows a single variable, the number of extra epochs divided by the unique tokens per parameter. Against the same data, a second epoch is worth nearly as much as a fresh one, and repeated tokens fall to half the value of fresh ones after a critical epoch count that grows with the training budget per parameter but hardly with model size. With unique data fixed, the predicted compute-optimal run grows model size and epochs together until loss stops improving, near the critical epoch count. The same variable accounts for the direction of size trends that appear to conflict: larger models tolerate fewer epochs when the corpus is fixed, from about 15 at 127M to 4 at 2B parameters, but not when unique data grow with the model. Counts alone do not determine loss: at identical counts, replaying shards consecutively raises loss by up to 0.46~bits per byte, concentrating repeats on fewer samples also raises it, lower-entropy sources degrade faster with repetition, and re-tokenizing repeats helps only under heavy repetition. These results offer an empirical guide to pretraining when unique data, rather than compute, are the binding constraint.

Yekun Chai, Haoyi Xiong

FloatAI · ETH Zurich · Independent Researcher

Language Model PretrainingLanguage Model Scaling LawsData Repetition in Language Model Training
AskPDF
May 19, 2026

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.

Jingwen Liu, Ezra Edelman, Surbhi Goel +1

Columbia University · University of Pennsylvania · Kempner Institute, Harvard University

Deep Learning OptimizationStatistical Learning TheoryData-Efficient LearningSmall-Data Learning
AskPDF
Jul 28, 2026

Bridging Compute- and Data-Optimal Pretraining

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, ηη, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit ηη for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of ηη implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

Tian Qin, Kimia Hamidieh, David Alvarez-Melis

Harvard University · MIT CSAIL

Language Model PretrainingLanguage Model Scaling LawsScaling LawsData-Efficient Learning
AskPDF