Language models are typically trained with a single choice of architecture on diverse domains and data sources. Modern transformers have been extensively refined for such general-purpose learning, but are they equally effective across different subsets of the pretraining data? We investigate this question by training small language models from scratch on representative subsets of pretraining data such as web text, code, and mathematics. We introduce variants of transformers in which standard activation functions are replaced with learned splines tuned on specific datasets, providing a simple way to modify their inductive biases. Using a mix-and-match of architectures and datasets, we evaluate cross-domain transfer, i.e. how well does an architecture tuned for one domain perform on data from another? We find that the tuned transformer variants improve training efficiency or generalization on all seven of our tested datasets, but the gains are often domain-specific. For example, architectures tuned on a natural language dataset such as FineWeb perform well on TinyStories, whereas those tuned on mathematics perform poorly on natural language. We identify several mechanisms underlying these improvements, including wider loss basins and more abundant "lottery tickets". Our results show that different subsets of pretraining data can benefit from different learning mechanisms, highlighting a limitation of the prevailing trend towards a small set of general-purpose architectures and optimizers. This points to an opportunity to improve data efficiency by developing domain-specific learning strategies that retain the ability to train general-purpose models.
Figures & tables
Figure 1: Our experimental setup quantifies the compatibility of inductive biases across subsets of data. We specialize a transformer variant for a source domain D1 ( left ) by replacing nonlinearities with parameterized splines, pre-trained in a warm-up phase. We then reuse these frozen components to train a model on a target dataset D2 ( right ). Only very few parameters are transferred.
Figure 2: Typical training curves during the warm-up phase (on Csn-Python here), where we optimize both the weights of transformers and their nonlinearities, initialized as constant functions (left). We train multiple models in a staggered schedule. They share the same nonlinearities, providing a regularizing effect that makes them more likely to transfer later to other models.
Figure 3: Validation performance of models trained on three datasets representative of natural language, code, and mathematics. Dots represent models trained with different seeds or nonlinearities (see legend on the left). Coloured outlines represent approximate distributions. The architectures specialized for each target dataset ( D1=D2 ) almost always improve over the baselines, whereas the transfer across domains ( D1=D2 ) varies a lot. See Figure 8 for similar results on other datasets.
Figure 4: Validation loss on pairs of datasets for language and code. Each marker corresponds to a specific seed and nonlinearity, used to train models on either of two datasets. The patterns formed by the markers indicate whether the improvements on one dataset correlate with improvements on the other. With some exceptions, most panels show a good correlation, indicating a compatibility between language and code datasets. See the discussion in Section 4.1 .
Figure 5: Validation accuracy while training generalist models on Math-subset . The difference with the baseline persists with up to ∼ 20 tokens/parameter and increased model depth.
Figure 6: Generalist vs. specialist models on Math (i.e. trained on all vs. individual question types). We observe larger improvements from learned nonlinearities in specialist models. This means that each type of math question can benefit from question-specific inductive biases. Note that we adjust the amount of training tokens such that the overall accuracy is similar between generalist and specialist baselines. See Figure 10 for a detailed view including the full set of labels on the Y axis.
Figure 7
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Complement to Figure 3 with results on all datasets. Consistent observations are made per domain. See discussion in Section 4 .
Figure 9: Counter-example to Figure 4 where we observe inverse correlations in the performance on pairs of datasets. We consider here the Shakespeare datasets with a character-level tokenization. This likely requires different inductive biases than standard subword tokenization because of longer-range interactions, different typical entropy in the predictions, etc. As a result, the architectures optimized for this dataset ( Shakespeare-char , vertical axis) work badly on others ( FineWeb , Enwik8 ) and vice versa.
Figure 10: Larger version of Figure 6 with all Y-axis labels.
Figure 11: Learned nonlinearities for various datasets (MLP/attention).
Figure 12: Learned layer-specific nonlinearities for Enwik8 .
Figure 13: Comparison of nonlinearities learned with models of 4 vs. 12 layers for different Math question types.
Figure 14: More examples like Figure 7 . The baseline with standard LR decay ( - - ) often improves late during training when the LR reaches very small values. In comparison, our model converges more easily both with ( – ) and without ( – ) LR decay.