Sequential Pretraining Favors Large Models

Mohnish Harwani; Yujia Zheng

cs.LGOct 7, 2026

Sequential Pretraining Favors Large Models

Authors: Mohnish Harwani, Yujia Zheng

Organizations: Purdue University · University of Illinois Urbana–Champaign

Abstract

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Figures & tables

Figure 1: Conceptual illustration of primacy bias and Exposure Therapy. Small models may become constrained by parameters recruited for early data, making later distributions harder to fit. Additional capacity can allow optimization to lead to alternative solutions, while Exposure Therapy encourages small models toward parameters that remain compatible with later learning.

Figure 2: Sensitivity to harmful early data decreases with model scale. Primacy sensitivity $\Delta_{\mathrm{prim}}$ compares final target performance under target-first and OOD-first training. Positive values indicate that the early OOD phase impaired later target learning. Columns indicate different datasets; curves within each panel correspond to different seeds. Small models exhibit significant primacy sensitivity, while the gap attenuates sharply with increasing capacity.

Figure 3: Large models recruit a less-overlapping route for downstream learning. Solid curves with circular markers indicate primacy sensitivity $\Delta_{\mathrm{prim}}$ , using the left axis; dashed curves with square markers show cross-phase representational overlap on the right axis. The reduction in sensitivity roughly aligns with the reduction in overlap between the units recruited during early OOD training and those recruited after subsequent target training. This correspondence supports the hypothesis that additional capacity allows larger models to route downstream learning through a less-overlapping set of units, rather than repurposing the units most strongly shaped by the early OOD distribution. At high activation thresholds, this signal becomes less accurate and rises in larger models.

Figure 4: Capacity-constrained foundation models remain sensitive to data encountered early in pretraining. The English primacy sensitivity gap reports the percentage increase in final English loss relative to the mean English-first loss at the same model scale. During the first half of training, models see either English or an OOD corpus consisting of code, mathematics, German, or Finnish; all models then train on a majority English distribution during the second half. Points denote individual runs, colors indicate the OOD corpus, and horizontal bars denote group means. Across 100M, 500M, and 1B parameter models, OOD-first pretraining produces substantially higher final English loss. Training data and optimization budget increase with model size, keeping models capacity-constrained relative to their respective training problems.

Figure 5: Less overparameterized models require stronger Exposure Therapy. Each point pairs a model’s parameter count with the smallest tested exposure dose required to recover $90\%$ of the corresponding target-first model’s performance. We evaluate Transformer classifiers on Yahoo Answers, Yelp, and Emotion. Dashed lines denote dataset-specific linear fits. Across all three datasets, the required exposure dose increases as model capacity decreases.

Figure 6: Exposure Therapy improves both later-distribution and aggregate performance in 1B parameter models. Top: held-out English loss after sequential pretraining on FineWeb and TinyStories; lower is better. Bottom: aggregate loss, computed across the Phase-1 OOD corpus and the English target distribution; lower is better. Within each OOD-corpus group, bars show English-first, OOD-first, and OOD-first training with a $30\%$ Exposure Therapy rate. Black points denote individual runs. OOD-first training consistently increases subsequent English loss, while ET reduces this impairment across every tested OOD and target distribution. At the same time, OOD-first training generally benefits aggregate performance by learning both useful distributions, and ET further reduces aggregate loss across all tested conditions. ET therefore improves acquisition of the later distribution without sacrificing the useful structure learned earlier.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{A}\subset\mathcal{D}_{\mathrm{g}},\qquad|\mathcal{A}|=s,

Appendix

Algorithm 1 Training with Exposure Therapy

Figure 8: Exposure Therapy improves sequential pretraining across model scales. Columns show 100M, 500M, and 1B parameter models. The first two rows report, respectively, held-out English loss and aggregate loss when FineWeb is the English target distribution; the bottom two rows report the corresponding quantities for TinyStories. Within each OOD-corpus group, gray bars denote English-first training, solid colored bars denote OOD-first training, and dotted colored bars denote OOD-first training with a $30\%$ Exposure Therapy rate and 10% exposure set size. Black points denote individual runs. Lower values are better. Across all three model scales and both English target distributions, OOD-first training increases subsequent English loss relative to the English-first reference, while Exposure Therapy consistently reduces this impairment. Exposure Therapy also reduces aggregate loss relative to unregularized OOD-first training across the tested conditions.

Figure 9: Exposure Therapy remains effective under reverse-order pretraining. Columns show 100M, 500M, and 1B parameter models. The top row reports held-out code/math/German/Finnish target loss, while the bottom row reports aggregate loss over FineWeb and the paired code, mathematics, German, or Finnish distribution; lower is better. Within each distribution, gray bars denote target-first training, solid colored bars denote English-first training, and dotted bars denote English-first training with Exposure Therapy. Black points denote individual runs. Across model scales, ET generally improves late-distribution learning relative to unregularized English-first training and consistently improves aggregate performance, demonstrating that its benefits persist when broad English data is encountered before more specialized distributions.

Panel of Figure 10 — Figure 10: Exposure Therapy under alternative Phase-2 training conditions. Results are shown for 100M-parameter models across code, mathematics, German, and Finnish target distributions. Gray bars denote target-first training, solid colored bars denote English-first training, and dotted bars denote English-first training with Exposure Therapy. Black points show individual runs. Top panels report held-out target loss and bottom panels report aggregate loss over the target and English distributions; lower is better. (a) Phase 2 uses the same target-dominated mixture as the main experiments, but English and target updates are randomly shuffled rather than deterministically interleaved. Exposure Therapy reduces both target and aggregate loss across the tested distributions. (b) Phase 2 contains only the target distribution, removing continued English updates. Exposure Therapy produces little change in target loss for code and mathematics, but improves target loss for German and Finnish and reduces aggregate loss across all four distributions.

Explore similar work

May 12, 2026cs.LG

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.

Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2

Carnegie Mellon University

Model Fine-Tuning Retention Forgetting Early Layers Exposure

Ask PDF

Jun 16, 2026cs.AI

Small Initialization Matters for Large Language Models

Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple

γ

-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.

Liangkai Hang, Junjie Yao, Zhiyu Li +3

School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.

Capacity Developmental Trajectories

Ask PDF

May 28, 2026cs.LG

Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures.

Jing Huang, Daniel Wurgaft, Rachit Bansal +6

Stanford University · Kempner Institute at Harvard University · MIT +1

Model Size Capacity Scaling Interference Impact Analysis

Ask PDF