Language models are typically trained with a single choice of architecture on diverse domains and data sources. Modern transformers have been extensively refined for such general-purpose learning, but are they equally effective across different subsets of the pretraining data? We investigate this question by training small language models from scratch on representative subsets of pretraining data such as web text, code, and mathematics. We introduce variants of transformers in which standard activation functions are replaced with learned splines tuned on specific datasets, providing a simple way to modify their inductive biases. Using a mix-and-match of architectures and datasets, we evaluate cross-domain transfer, i.e. how well does an architecture tuned for one domain perform on data from another? We find that the tuned transformer variants improve training efficiency or generalization on all seven of our tested datasets, but the gains are often domain-specific. For example, architectures tuned on a natural language dataset such as FineWeb perform well on TinyStories, whereas those tuned on mathematics perform poorly on natural language. We identify several mechanisms underlying these improvements, including wider loss basins and more abundant "lottery tickets". Our results show that different subsets of pretraining data can benefit from different learning mechanisms, highlighting a limitation of the prevailing trend towards a small set of general-purpose architectures and optimizers. This points to an opportunity to improve data efficiency by developing domain-specific learning strategies that retain the ability to train general-purpose models.
Figures & tables
Figure 1: Our experimental setup quantifies the compatibility of inductive biases across subsets of data. We specialize a transformer variant for a source domain D1 ( left ) by replacing nonlinearities with parameterized splines, pre-trained in a warm-up phase. We then reuse these frozen components to train a model on a target dataset D2 ( right ). Only very few parameters are transferred.
Figure 2: Typical training curves during the warm-up phase (on Csn-Python here), where we optimize both the weights of transformers and their nonlinearities, initialized as constant functions (left). We train multiple models in a staggered schedule. They share the same nonlinearities, providing a regularizing effect that makes them more likely to transfer later to other models.
Figure 3: Validation performance of models trained on three datasets representative of natural language, code, and mathematics. Dots represent models trained with different seeds or nonlinearities (see legend on the left). Coloured outlines represent approximate distributions. The architectures specialized for each target dataset ( D1=D2 ) almost always improve over the baselines, whereas the transfer across domains ( D1=D2 ) varies a lot. See Figure 8 for similar results on other datasets.
Figure 4: Validation loss on pairs of datasets for language and code. Each marker corresponds to a specific seed and nonlinearity, used to train models on either of two datasets. The patterns formed by the markers indicate whether the improvements on one dataset correlate with improvements on the other. With some exceptions, most panels show a good correlation, indicating a compatibility between language and code datasets. See the discussion in Section 4.1 .
Figure 5: Validation accuracy while training generalist models on Math-subset . The difference with the baseline persists with up to ∼ 20 tokens/parameter and increased model depth.
Figure 6: Generalist vs. specialist models on Math (i.e. trained on all vs. individual question types). We observe larger improvements from learned nonlinearities in specialist models. This means that each type of math question can benefit from question-specific inductive biases. Note that we adjust the amount of training tokens such that the overall accuracy is similar between generalist and specialist baselines. See Figure 10 for a detailed view including the full set of labels on the Y axis.
Figure 7
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Complement to Figure 3 with results on all datasets. Consistent observations are made per domain. See discussion in Section 4 .
Figure 9: Counter-example to Figure 4 where we observe inverse correlations in the performance on pairs of datasets. We consider here the Shakespeare datasets with a character-level tokenization. This likely requires different inductive biases than standard subword tokenization because of longer-range interactions, different typical entropy in the predictions, etc. As a result, the architectures optimized for this dataset ( Shakespeare-char , vertical axis) work badly on others ( FineWeb , Enwik8 ) and vice versa.
Figure 10: Larger version of Figure 6 with all Y-axis labels.
Figure 11: Learned nonlinearities for various datasets (MLP/attention).
Figure 12: Learned layer-specific nonlinearities for Enwik8 .
Figure 13: Comparison of nonlinearities learned with models of 4 vs. 12 layers for different Math question types.
Figure 14: More examples like Figure 7 . The baseline with standard LR decay ( - - ) often improves late during training when the LR reaches very small values. In comparison, our model converges more easily both with ( – ) and without ( – ) LR decay.
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be key for pushing AI beyond merely scaling current designs. Method. We present a method to optimize a transformer architecture for a given dataset, which we use as a tool to study optimal task-specific inductive biases. This method replaces the most important non-linearities (GeLUs,;softmax) with functions learned on held-out data. We then train the resulting architectures on other datasets, as a way to evaluate the compatibility between pairs of tasks. Findings. On algorithmic toy tasks, we identify new architectures with dramatic improvements in learning speed, in- and out-of-distribution generalization, and stability across seeds. The new designs prove very task-specific however, and indicate that these tasks require inductive biases very different from those of standard transformers. On code and language modeling datasets, we also find architectures with consistent, yet smaller improvements. These designs transfer much better across datasets and domains (English & computer code). Implications. Our results show that standard transformers are rarely a local optimum in the space of architectures. Simple alternatives can perform much better but sacrifice universality. This suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D' specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets.
Skyler Seto, Pierre Ablin, Anastasiia Filippova +4
Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks overfitting to the domain and forgetting general knowledge. We study a simple strategy, specialized pretraining (SPT), where a small domain dataset, typically reserved for finetuning, is repeated starting from pretraining as a fraction of the total tokens. Across three specialized domains (ChemPile, MusicPile, and ProofPile), SPT improves domain performance and preserves general capabilities after finetuning compared to standard pretraining. In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x. These gains grow when the target domain is underrepresented in the pretraining corpus: on domains far from web text, a 1B SPT model outperforms a 3B standard pretrained model. Beyond these empirical gains, we derive overfitting scaling laws to guide practitioners in selecting the optimal domain-data repetition for a given pretraining compute budget. Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility. SPT yields better specialized domain performance (via reduced overfitting across repeated exposures) and better general domain performance (via reduced forgetting during finetuning), ultimately achieving stronger results with fewer parameters and less total compute when amortized over inference. To get the most out of domain data, incorporate it as early in training as possible.
Christina Baek, Ricardo Pio Monti, David Schwab +31