A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
Organizations: University of Texas at Austin · Carnegie Mellon University
Abstract
In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.
Figures & tables
| Symbol | Meaning | Symbol | Meaning |
|---|---|---|---|
| Model width | Population feature dimension | ||
| Training samples | Samples per update | ||
| Ensemble members | Training epochs | ||
| Updates per epoch | Total updates |
| Condition | Optimal stopping | Ensemble size | Excess loss |
|---|---|---|---|
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Models and data | Horizon and replication |
|---|---|---|
| Data-size sweep Fig. 1 , top | , (85M); M; | About 8,000 updates ( – B tokens), or 11–101 epochs; one run per strategy at each |
| Model-size grid Figs. 1 , bottom, 5 , 8 | , plus at ; M; | 40 epochs; one run per cell; 18 cells spanning 11M–680M parameters |
| Matched-budget comparison Figs. 2 , 9 | Base , ; ensembles versus wider/deeper single models; M; | 40-epoch training budget; ensembles in Fig. 2 evaluated from saved model states through epoch 31 |
| Ensemble-size scaling Fig. 7 | , ; M; ; shared or independent data orders | 40 epochs; 20 trained members per strategy; 10 random member orderings per strategy |
| Ensembles across model sizes Fig. 6 | Six replicated cells (11M–170M); M; ; | 40 epochs; at least four independent members per cell; up to 680M total counted parameters |
| Additional joint-fit data | Seven-cell model-size ladder at M; | 50 epochs; one run per configuration |
| Data | FineWeb; GPT-2 BPE tokenizer (vocabulary 50,257); 10M held-out validation tokens |
|---|---|
| Architecture | Decoder-only Transformer, depth , width , head dimension 64 ( heads); rotary embeddings, QK-norm, RMSNorm without weights, no biases, no dropout; SwiGLU MLP with hidden width ; untied input and output embeddings; learned residual, input-injection, and U-Net skip gains; sliding-window attention on three of every four layers |
| Parameterization | CompleteP with base , ; truncated-normal initialization, std 0.02 at base width |
| Sequence length | 2048 tokens |
| Batch size | 131,072 tokens (64 sequences) per update |
| Optimizer | AdamW, , |
| Learning rates | matrices 0.01, embeddings 0.0375, output head 0.0005, scalar gains 0.025 |
| Quantity | Definition or procedure |
|---|---|
| Validation loss | Cross-entropy in nats per token on a held-out FineWeb split. |
| Epoch and step | Each update processes 131,072 tokens; one epoch is updates (763 at M; 152 at M). |
| Evaluation cadence | Every 76 updates in the data sweep and 152 updates in the model grids, corresponding to about 10M and 19.9M training tokens. |
| Saved model state | A checkpoint stores the model parameters at a training step. The optimal checkpoint is the evaluated state with minimum validation loss. |
| Optimal stopping | is the evaluated step with minimum validation loss. is the corresponding epoch count. |
| Data-sweep aggregation | One minimum per strategy at each corpus size; the reported minimum loss and optimal epoch average the two strategies’ minima. |
| Fit | Model and observations |
|---|---|
| Joint law | on 35 observations: the 18-cell M grid, seven-cell M ladder, and ten data-sweep settings. Shared exponent , floor , . |
| Model-size laws | One minimum per cell on the 18-cell M grid. Fit against , or against with separate exponents. Optimal-epoch fits use the same cells. |
| Data-size laws | Ten corpus sizes at fixed , using the strategy-averaged minima and optimal epochs defined in Table 5 . |
| Ensemble/model-size laws | A separate power law in total counted parameters for each , using the six replicated cells and the same loss floor. |
| Ensemble stopping | Fit over for six model configurations , with a shared exponent and separate prefactors . The dashed guide in Figure 6 B uses the geometric mean of these prefactors. |
| Ensemble-size law | Equation ( 13 ) fits an infinite-ensemble asymptote for each strategy at each checkpoint; this is distinct from the shared loss floor. |
| Quantity | Fitted dependence | Exponent(s) | Points | |
|---|---|---|---|---|
| 18 | ||||
| 18 | ||||
| 10 | ||||
| 18 | ||||
| 18 | ||||
| 10 |
| Comparison | Result on the measured configurations |
|---|---|
| Depth/width loss fit | , , each stable to under leave-one-out. The fitted sensitivity ratio is . |
| Matched-size example | At M, attains loss , versus for . |
| Depth/width stopping fit | , , so . Its rounded matches the fit using alone. Dependence solely on would give . |
| Resource scaling | Optimal epochs fall by a fitted factor of per decade of data scaling, versus per decade of model scaling. |
| Ensemble/model-size fits | Exponents , , for , versus for single models. The fitted law lies about nats below the single-model law over the comparison range. |
| Asymptotic exponent | Continuum fit | Simulation fit | |
|---|---|---|---|
| 1.5 | 0 | 0.167 | 0.172 [0.117, 0.234] |
| 2 | 0.704 | 0.633 [0.535, 0.727] | |
| 2.5 | 1 | 0.860 | 0.776 [0.705, 0.865] |
Language Model Scaling Laws
Language model scaling laws describe how model size, training data, compute, and inference effort jointly shape loss, capabilities, and efficiency.
See all 151 papers