cs.LGOct 7, 2026
SaveSequential Pretraining Favors Large Models
Organizations: Purdue University · University of Illinois Urbana–Champaign
Abstract
Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.
Figures & tables
Figure 1: Conceptual illustration of primacy bias and Exposure Therapy. Small models may become constrained by parameters recruited for early data, making later distributions harder to fit. Additional capacity can allow optimization to lead to alternative solutions, while Exposure Therapy encourages small models toward parameters that remain compatible with later learning.
Figure 2: Sensitivity to harmful early data decreases with model scale. Primacy sensitivity compares final target performance under target-first and OOD-first training. Positive values indicate that the early OOD phase impaired later target learning. Columns indicate different datasets; curves within each panel correspond to different seeds. Small models exhibit significant primacy sensitivity, while the gap attenuates sharply with increasing capacity.
Figure 3: Large models recruit a less-overlapping route for downstream learning. Solid curves with circular markers indicate primacy sensitivity , using the left axis; dashed curves with square markers show cross-phase representational overlap on the right axis. The reduction in sensitivity roughly aligns with the reduction in overlap between the units recruited during early OOD training and those recruited after subsequent target training. This correspondence supports the hypothesis that additional capacity allows larger models to route downstream learning through a less-overlapping set of units, rather than repurposing the units most strongly shaped by the early OOD distribution. At high activation thresholds, this signal becomes less accurate and rises in larger models.
Figure 4: Capacity-constrained foundation models remain sensitive to data encountered early in pretraining. The English primacy sensitivity gap reports the percentage increase in final English loss relative to the mean English-first loss at the same model scale. During the first half of training, models see either English or an OOD corpus consisting of code, mathematics, German, or Finnish; all models then train on a majority English distribution during the second half. Points denote individual runs, colors indicate the OOD corpus, and horizontal bars denote group means. Across 100M, 500M, and 1B parameter models, OOD-first pretraining produces substantially higher final English loss. Training data and optimization budget increase with model size, keeping models capacity-constrained relative to their respective training problems.
Figure 5: Less overparameterized models require stronger Exposure Therapy. Each point pairs a model’s parameter count with the smallest tested exposure dose required to recover of the corresponding target-first model’s performance. We evaluate Transformer classifiers on Yahoo Answers, Yelp, and Emotion. Dashed lines denote dataset-specific linear fits. Across all three datasets, the required exposure dose increases as model capacity decreases.
Figure 6: Exposure Therapy improves both later-distribution and aggregate performance in 1B parameter models. Top: held-out English loss after sequential pretraining on FineWeb and TinyStories; lower is better. Bottom: aggregate loss, computed across the Phase-1 OOD corpus and the English target distribution; lower is better. Within each OOD-corpus group, bars show English-first, OOD-first, and OOD-first training with a Exposure Therapy rate. Black points denote individual runs. OOD-first training consistently increases subsequent English loss, while ET reduces this impairment across every tested OOD and target distribution. At the same time, OOD-first training generally benefits aggregate performance by learning both useful distributions, and ET further reduces aggregate loss across all tested conditions. ET therefore improves acquisition of the later distribution without sacrificing the useful structure learned earlier.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 Training with Exposure Therapy
Figure 8: Exposure Therapy improves sequential pretraining across model scales. Columns show 100M, 500M, and 1B parameter models. The first two rows report, respectively, held-out English loss and aggregate loss when FineWeb is the English target distribution; the bottom two rows report the corresponding quantities for TinyStories. Within each OOD-corpus group, gray bars denote English-first training, solid colored bars denote OOD-first training, and dotted colored bars denote OOD-first training with a Exposure Therapy rate and 10% exposure set size. Black points denote individual runs. Lower values are better. Across all three model scales and both English target distributions, OOD-first training increases subsequent English loss relative to the English-first reference, while Exposure Therapy consistently reduces this impairment. Exposure Therapy also reduces aggregate loss relative to unregularized OOD-first training across the tested conditions.
Figure 9: Exposure Therapy remains effective under reverse-order pretraining. Columns show 100M, 500M, and 1B parameter models. The top row reports held-out code/math/German/Finnish target loss, while the bottom row reports aggregate loss over FineWeb and the paired code, mathematics, German, or Finnish distribution; lower is better. Within each distribution, gray bars denote target-first training, solid colored bars denote English-first training, and dotted bars denote English-first training with Exposure Therapy. Black points denote individual runs. Across model scales, ET generally improves late-distribution learning relative to unregularized English-first training and consistently improves aggregate performance, demonstrating that its benefits persist when broad English data is encountered before more specialized distributions.
Figure 10: Exposure Therapy under alternative Phase-2 training conditions. Results are shown for 100M-parameter models across code, mathematics, German, and Finnish target distributions. Gray bars denote target-first training, solid colored bars denote English-first training, and dotted bars denote English-first training with Exposure Therapy. Black points show individual runs. Top panels report held-out target loss and bottom panels report aggregate loss over the target and English distributions; lower is better. (a) Phase 2 uses the same target-dominated mixture as the main experiments, but English and target updates are randomly shuffled rather than deterministically interleaved. Exposure Therapy reduces both target and aggregate loss across the tested distributions. (b) Phase 2 contains only the target distribution, removing continued English updates. Exposure Therapy produces little change in target loss for code and mathematics, but improves target loss for German and Finnish and reduces aggregate loss across all four distributions.