cs.LGMar 17, 2026
SaveThe Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
Organizations: DatologyAI
Abstract
Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks overfitting to the domain and forgetting general knowledge. We study a simple strategy, specialized pretraining (SPT), where a small domain dataset, typically reserved for finetuning, is repeated starting from pretraining as a fraction of the total tokens. Across three specialized domains (ChemPile, MusicPile, and ProofPile), SPT improves domain performance and preserves general capabilities after finetuning compared to standard pretraining. In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x. These gains grow when the target domain is underrepresented in the pretraining corpus: on domains far from web text, a 1B SPT model outperforms a 3B standard pretrained model. Beyond these empirical gains, we derive overfitting scaling laws to guide practitioners in selecting the optimal domain-data repetition for a given pretraining compute budget. Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility. SPT yields better specialized domain performance (via reduced overfitting across repeated exposures) and better general domain performance (via reduced forgetting during finetuning), ultimately achieving stronger results with fewer parameters and less total compute when amortized over inference. To get the most out of domain data, incorporate it as early in training as possible.
Figures & tables
Figure 1: Specialized pretraining (SPT) mixes the finetuning dataset into pretraining as a small fraction of tokens, repeating it 10–50× over the course of training. Compared to general pretraining (dashed), SPT (solid) achieves lower domain test loss (blue) and less forgetting of general knowledge (gold) throughout finetuning. For narrow domains, these gains can overcome differences in model scale.
Figure 2: Specialized pretraining ( SPT ) outperforms finetuning-only across domains. We pretrain models with a small fraction ( ) of domain-specific tokens mixed into general web data, then finetune on the domain dataset. We plot the best post-finetuning domain loss across pretraining budgets for MusicPile, ChemPile, and ProofPile (300M tokens each). Even small domain mixtures ( – , blue curves) consistently outperform pretraining on general data alone ( , gray) at all token scales.
Figure 3: The finetuner’s tax. Training a 1B model with specialized pretraining ( SPT ) costs more upfront than finetuning a 3B model on domain data alone, but the smaller model is cheaper to serve. The break-even point arrives after approximately 1 trillion inference tokens, after which SPT saves both compute and money while often delivering comparable or better performance.
Figure 4: Specialized pretraining ( SPT ) is more effective than scaling tokens or parameters. We compare models that include domain data during pretraining ( SPT FT ) against models pretrained only on general web data ( NPT FT ). Left: Relative gain of SPT FT over NPT FT . Center: Compute multiplier showing how much faster SPT reaches NPT ’s best performance. Right: Percentage of the 1B vs. 3B parameter performance gap closed by SPT . Values above 100% indicate the 1B SPT outperforms 3B NPT . SPT consistently improves model quality, training speed, and parameter efficiency (i.e. SPT is a pareto improvement in training efficiency) across all three examined domains.
Figure 5: SPT reduces forgetting and improves downstream task performance. (a) For ChemPile, we plot Dolma loss (general knowledge) against domain loss for the best post-finetuning checkpoint at each pretraining budget (40B to 200B tokens) and mixture percentage . Larger SPT mixtures achieve lower domain loss and lower general loss, indicating less catastrophic forgetting. (b) We compare NPT (gray) and 2% SPT (blue) on downstream tasks matched to each domain: MusicTheoryBench for MusicPile, ChemBench General Chemistry subset for ChemPile, and MATH for ProofPile. All tasks are evaluated in 4-choice MCQA format. For each pretraining budget, we report the best accuracy across finetuning runs. SPT outperforms NPT across most settings.
Figure 6: SPT regularizes finetuning. Domain training loss (x-axis) versus domain test loss (y-axis) across the pretraining and finetuning stages for MusicPile, with mixture fractions . Dotted lines show the pretraining stage; solid lines show finetuning. During finetuning, SPT models achieve lower test loss than NPT models at the same training loss, indicating that SPT reduces overfitting. Notably, the model has already seen the domain data over during pretraining, yet it still overfits less during finetuning than the NPT model seeing the same data for the first time. This regularization is a consequence of diffused exposure to specialized data during pretraining.
Figure 7: Benefits of SPT increase as pretraining and finetuning domains diverge. We vary the percentage of Japanese monolingual text in the pretraining mix for an English Japanese translation task, and plot of SPT FT over NPT FT . With less Japanese monolingual data (leftwards on x-axis), the distributional gap between pretraining and finetuning data grows, and the gain from SPT increases, plateauing at approximately 5%.
Figure 8: Effect of domain data size and model size on . Left: Relative gain of SPT over NPT as a function of domain dataset size ({3M, 30M, 300M} tokens from MusicPile) at three pretraining scales (40B, 120B, 200B tokens). For the 300M-token dataset, gains are consistently high across all scales. For smaller datasets, SPT is less helpful at longer training horizons due to overfitting from excessive repetition; in these regimes, specialized continued pretraining (star) is preferable. Right: Domain test loss vs. model size for NPT ( ) and SPT ( ) on MusicPile-300M. Relative gain grows with scale, driven not only by decreasing NPT loss but also a widening SPT – NPT gap.
Figure 9: The optimal mixture shifts with compute budget. Domain test loss on MusicPile throughout SPT for mixture fractions . At small compute budgets (left region), high domain mixtures ( ) achieve the lowest test loss. As training progresses, these mixtures overfit and their loss increases, while moderate mixtures ( ) become optimal (middle region). Past 240B tokens, lower mixtures ( ) are best. Regardless of budget, SPT with an appropriate outperforms NPT ( ).
Figure 10: Replay is not a substitute for early domain exposure. MusicPile validation-loss trajectories during replay-based continued pretraining. Blue lines ( SPT CPT ) consistently achieve lower domain test loss than purple lines ( NPT CPT ) across all replay rates (lighter shades indicate higher replay: ). While higher replay slows overfitting, it does not close the gap between SPT and NPT , confirming that when domain data is seen has a lasting impact on the model’s performance on specialized domains.
Figure 11: Repetition scaling laws as the sum of two powers. As the domain data is repeated throughout pretraining during specialized pretraining, for high mixture percentages, the model may overfit and the test loss goes back up. A single power law cannot model this overfitting stage. Instead, we fit a power law separately, for the domain training loss and the domain train-test gap. The latter also follows a power law with a positive exponent. Then the test loss can be modeled using the sum of two powers. Furthermore, we model all coefficients as a function of the mixture percentage, according to Equation 4 .
Figure 12: Predicting loss after finetuning. We measure the difference in domain-specific test loss right after pretraining versus the best test loss after finetuning ( ) as a function of the pretraining tokens. We observe that this difference follows a power law relationship.
Figure 13: Forecasting domain loss and optimal mixture from small-scale runs. We demonstrate two ways our scaling laws can guide the choice of mixture percentage for SPT. (Left) We predict the entire domain test loss trajectory of SPT by fitting our scaling laws (Equation 1 ) using only the curves from . The forecast correctly predicts that SPT begins to overfit around 70B tokens. (Right) We extrapolate post-finetuning test loss past 120B pretraining tokens and correctly predict that SPT surpasses SPT around 280B tokens. Together, these forecasts allow practitioners to identify the right mixture fraction without running the full training sweep.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Gamma and log-linear fits over learned coefficients.
Figure 15: All power law fits over the training loss, train-test gap on MusicPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 16: All power law fits over the training loss, train-test gap on ChemPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 17: All power law fits over the training loss, train-test gap on ProofPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 18: Test follows a power law relationship with respect to the pretraining steps.
Figure 19: Comparison of Pearson correlations between the Japanese overlap sweep and cross-domain analysis. C2ST AUC flips from to .
Figure 20: Japanese overlap sweep: each metric plotted against . All metrics show strong correlations ( ) in the expected direction.
Figure 21: Cross-domain analysis: each metric plotted against for ChemPile, MusicPile, and ProofPile. Correlations are much weaker than the Japanese sweep, and C2ST AUC flips sign.