Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks overfitting to the domain and forgetting general knowledge. We study a simple strategy, specialized pretraining (SPT), where a small domain dataset, typically reserved for finetuning, is repeated starting from pretraining as a fraction of the total tokens. Across three specialized domains (ChemPile, MusicPile, and ProofPile), SPT improves domain performance and preserves general capabilities after finetuning compared to standard pretraining. In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x. These gains grow when the target domain is underrepresented in the pretraining corpus: on domains far from web text, a 1B SPT model outperforms a 3B standard pretrained model. Beyond these empirical gains, we derive overfitting scaling laws to guide practitioners in selecting the optimal domain-data repetition for a given pretraining compute budget. Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility. SPT yields better specialized domain performance (via reduced overfitting across repeated exposures) and better general domain performance (via reduced forgetting during finetuning), ultimately achieving stronger results with fewer parameters and less total compute when amortized over inference. To get the most out of domain data, incorporate it as early in training as possible.
Figures & tables
Figure 1: Specialized pretraining (SPT) mixes the finetuning dataset into pretraining as a small fraction of tokens, repeating it 10–50× over the course of training. Compared to general pretraining (dashed), SPT (solid) achieves lower domain test loss (blue) and less forgetting of general knowledge (gold) throughout finetuning. For narrow domains, these gains can overcome differences in model scale.
Figure 2: Specialized pretraining ( SPT ) outperforms finetuning-only across domains. We pretrain models with a small fraction ( δ ) of domain-specific tokens mixed into general web data, then finetune on the domain dataset. We plot the best post-finetuning domain loss across pretraining budgets for MusicPile, ChemPile, and ProofPile (300M tokens each). Even small domain mixtures ( δ=1 – 5% , blue curves) consistently outperform pretraining on general data alone ( δ=0% , gray) at all token scales.
Figure 3: The finetuner’s tax. Training a 1B model with specialized pretraining ( SPT ) costs more upfront than finetuning a 3B model on domain data alone, but the 3× smaller model is cheaper to serve. The break-even point arrives after approximately 1 trillion inference tokens, after which SPT saves both compute and money while often delivering comparable or better performance.
Figure 4: Specialized pretraining ( SPT ) is more effective than scaling tokens or parameters. We compare models that include domain data during pretraining ( SPT → FT ) against models pretrained only on general web data ( NPT → FT ). Left: Relative gain of SPT → FT over NPT → FT . Center: Compute multiplier showing how much faster SPT reaches NPT ’s best performance. Right: Percentage of the 1B vs. 3B parameter performance gap closed by SPT . Values above 100% indicate the 1B SPT outperforms 3B NPT . SPT consistently improves model quality, training speed, and parameter efficiency (i.e. SPT is a pareto improvement in training efficiency) across all three examined domains.
Figure 5: SPT reduces forgetting and improves downstream task performance. (a) For ChemPile, we plot Dolma loss (general knowledge) against domain loss for the best post-finetuning checkpoint at each pretraining budget (40B to 200B tokens) and mixture percentage δ . Larger SPT mixtures achieve lower domain loss and lower general loss, indicating less catastrophic forgetting. (b) We compare NPT (gray) and 2% SPT (blue) on downstream tasks matched to each domain: MusicTheoryBench for MusicPile, ChemBench General Chemistry subset for ChemPile, and MATH for ProofPile. All tasks are evaluated in 4-choice MCQA format. For each pretraining budget, we report the best accuracy across finetuning runs. SPT outperforms NPT across most settings.
Figure 6: SPT regularizes finetuning. Domain training loss (x-axis) versus domain test loss (y-axis) across the pretraining and finetuning stages for MusicPile, with mixture fractions δ=0%,2%,5% . Dotted lines show the pretraining stage; solid lines show finetuning. During finetuning, SPT models achieve lower test loss than NPT models at the same training loss, indicating that SPT reduces overfitting. Notably, the δ=5% model has already seen the domain data over 33× during pretraining, yet it still overfits less during finetuning than the NPT model seeing the same data for the first time. This regularization is a consequence of diffused exposure to specialized data during pretraining.
Figure 7: Benefits of SPT increase as pretraining and finetuning domains diverge. We vary the percentage of Japanese monolingual text in the pretraining mix for an English → Japanese translation task, and plot Rgain of SPT → FT over NPT → FT . With less Japanese monolingual data (leftwards on x-axis), the distributional gap between pretraining and finetuning data grows, and the gain from SPT increases, plateauing at approximately 5%.
Figure 8: Effect of domain data size and model size on Rgain . Left: Relative gain of SPT over NPT as a function of domain dataset size ({3M, 30M, 300M} tokens from MusicPile) at three pretraining scales (40B, 120B, 200B tokens). For the 300M-token dataset, gains are consistently high across all scales. For smaller datasets, SPT is less helpful at longer training horizons due to overfitting from excessive repetition; in these regimes, specialized continued pretraining (star) is preferable. Right: Domain test loss vs. model size for NPT ( δ=0% ) and SPT ( δ=2% ) on MusicPile-300M. Relative gain grows with scale, driven not only by decreasing NPT loss but also a widening SPT – NPT gap.
Figure 9: The optimal mixture shifts with compute budget. Domain test loss on MusicPile throughout SPT for mixture fractions δ∈{0%,2%,5%,10%} . At small compute budgets (left region), high domain mixtures ( δ=10% ) achieve the lowest test loss. As training progresses, these mixtures overfit and their loss increases, while moderate mixtures ( δ=5% ) become optimal (middle region). Past 240B tokens, lower mixtures ( δ=2% ) are best. Regardless of budget, SPT with an appropriate δ outperforms NPT ( δ=0% ).
Figure 10: Replay is not a substitute for early domain exposure. MusicPile validation-loss trajectories during replay-based continued pretraining. Blue lines ( SPT → CPT ) consistently achieve lower domain test loss than purple lines ( NPT → CPT ) across all replay rates (lighter shades indicate higher replay: 0%→10%→20% ). While higher replay slows overfitting, it does not close the gap between SPT and NPT , confirming that when domain data is seen has a lasting impact on the model’s performance on specialized domains.
Figure 11: Repetition scaling laws as the sum of two powers. As the domain data is repeated throughout pretraining during specialized pretraining, for high mixture percentages, the model may overfit and the test loss goes back up. A single power law cannot model this overfitting stage. Instead, we fit a power law separately, for the domain training loss and the domain train-test gap. The latter also follows a power law with a positive exponent. Then the test loss can be modeled using the sum of two powers. Furthermore, we model all coefficients as a function of the mixture percentage, according to Equation 4 .
Figure 12: Predicting loss after finetuning. We measure the difference in domain-specific test loss right after pretraining versus the best test loss after finetuning ( ΔTest ) as a function of the pretraining tokens. We observe that this difference follows a power law relationship.
Figure 13: Forecasting domain loss and optimal mixture from small-scale runs. We demonstrate two ways our scaling laws can guide the choice of mixture percentage for SPT. (Left) We predict the entire domain test loss trajectory of 10% SPT by fitting our scaling laws (Equation 1 ) using only the curves from δ∈[0,5] . The forecast correctly predicts that 10% SPT begins to overfit around 70B tokens. (Right) We extrapolate post-finetuning test loss past 120B pretraining tokens and correctly predict that 2% SPT surpasses 5% SPT around 280B tokens. Together, these forecasts allow practitioners to identify the right mixture fraction without running the full training sweep.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: Gamma and log-linear fits over learned coefficients.
Figure 15: All power law fits over the training loss, train-test gap on MusicPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 16: All power law fits over the training loss, train-test gap on ChemPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 17: All power law fits over the training loss, train-test gap on ProofPile-300M over the course of SPT. Finally, test loss as the sum of two powers.
Figure 18: Δ Test follows a power law relationship with respect to the pretraining steps.
Figure 19: Comparison of Pearson correlations between the Japanese overlap sweep and cross-domain analysis. C2ST AUC flips from r=+0.90 to r=−0.98 .
Figure 20: Japanese overlap sweep: each metric plotted against Rgain . All metrics show strong correlations ( ∣r∣>0.85 ) in the expected direction.
Figure 21: Cross-domain analysis: each metric plotted against Rgain for ChemPile, MusicPile, and ProofPile. Correlations are much weaker than the Japanese sweep, and C2ST AUC flips sign.
Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.
Yuduo Li, Xiaofeng Shi, Qian Kou +2
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.
To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited amount of target data. Typically, generic data is only mixed in during fine-tuning to prevent catastrophic forgetting of the generic domain. We surprisingly find that replaying the generic data during fine-tuning can actually improve performance on the (less related) target task. Concretely, in a controlled pre-training environment with 4M target tokens, 4B total tokens, and 150M parameter models, generic replay increases target data efficiency by up to 1.87× for fine-tuning and 2.06× for mid-training. We further analyze data schedules that introduce target data during pre-training and find that replay helps more when there is less target data present in pre-training. We demonstrate the success of replay in practice for fine-tuning 8B parameter models, improving agentic web navigation success by 4.5% and Basque question-answering accuracy by 2%.