When told that robins have sesamoid bones, people are far more prone to extend the property to sparrows than to whales. This behavior, called category-based property induction, is well documented in humans, but whether and how it develops in language models trained only to predict the next token is unclear. To shed light on this, we construct a hierarchical taxonomy of synthetic entities grouped by the properties they share. We then inject facts from this latent taxonomy into the continued pre-training of Pythia and OLMo 2 models from 70M to 7B parameters, and track the development of inductive behaviors. We find that taxonomically aligned induction emerges from next-token prediction alone: a property learned for one entity transfers more strongly within its taxonomic category than across categories. Internally, entities that share a deeper category become more similar in activation space, and clustering recovers the full taxonomic tree from entity representations. Contrary to the usual expectation that scale helps, we find that induction weakens with model size when the data budget is fixed, suggesting that larger models memorize individual facts rather than compress them into shared structure. Induction also strengthens with more pre-training before the taxonomy injection, transfers to novel in-context properties, and can be overridden by a latent taxonomy implied by the prompt. Together, these results show that a model's inductive generalization reflects the latent organization of its training data, with implications for data curation and for predicting how far a learned fact will spread.
Figures & tables
Figure 1: Overview. (a) Example natural (top) and synthetic (bottom) taxonomies of depth D=2 ; experiments use D=3 . (b) We continue pre-training Pythia and OLMo 2 with synthetic sequences mixed in, and (c) track inductive generalization (Pythia 160M shown; see Figure 2 ).
Figure 2: Logp(property∣entity) over continued pre-training of Pythia 160M and OLMo 2 1B. With shared properties (left), memorization (dashed) > induction within category (green) > induction across categories (red). Without shared properties (right), within- and across-category induction collapse onto each other. Shaded regions: standard error of the mean across three random seeds.
Figure 3: Hierarchical recovery of the taxonomy from entity representations of OLMo 2 1B. Dendrograms before vs. after continued pre-training on entities and properties. Bars below each dendrogram mark the ground-truth L3 and L2 categories (clustering details in Appendix, subsection C.6 ).
Figure 4: More mature checkpoints acquire induction more strongly. Shown is the induction advantage averaged over the last 1,000 steps of synthetic data injection in continued pre-training from various checkpoints (x-axis). Circles show induction advantage for individual evaluation templates, while horizontal lines show the average. All reported values are averages from three random seeds.
Figure 5: At a fixed data budget, scale hurts. Shown is the induction advantage averaged over the last 1,000 data injection steps vs. model size. Circles indicate induction advantage for individual evaluation templates, while horizontal lines show the average induction advantage. All reported values are averages from three random seeds.
Figure 6: Left: induction within vs. across categories during two-phase training (Taxonomy conflict: Yes ), measured against both the old taxonomy T1 (dashed) and the new taxonomy T2 (solid). Right: induction advantage reached under T2 when the model was first trained on a conflicting taxonomy ( Yes ) vs. trained on T2 from the same base checkpoint without the prior phase ( No ).
Figure 7: In-weights taxonomy shapes in-context induction. Left: schematics of the two setups; models are trained only on shared properties, so probe and control properties appear solely in the prompt. Top: inductive generalization of the probe P (green) vs. each control Ci (red) to target E2; one shot lists the probe and all control properties. Bottom: generalization of P to the in-context closest target E3 vs. the in-weights closest target E2; one shot lists all properties in the in-context taxonomy. Shots are capped by each model’s context window; sentences use randomly sampled evaluation templates (Box C.5 ); shaded regions show SEM over five random seeds.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Pythia
OLMo 2
Optimization
Optimizer
AdamW
AdamW
Learning rate
≈3×10−4
≈4×10−4
LR scheduler
constant
constant
Warmup steps
0
0
Adam β1
0.9
0.9
Appendix
Table 1: Hyperparameters used for continued pre-training with injected synthetic taxonomy data. Learning rates are loaded from the optimizer checkpoints and are therefore slightly below the reported values due to their original (global) cosine schedules. A constant learning rate (LR) scheduler was chosen as it approximates the global cosine schedule well on our small training window of 10k training steps.
Figure 9Figure 10
Figure 8: Schematic of the experiment on interference across taxonomies. Models are first trained on taxonomy T1 and then on a conflicting taxonomy T2 that reuses the same entity names but regroups them. Control denotes standard training without taxonomy conflict (Taxonomy conflict: No ).
Figure 9: Mean pairwise cosine similarity between entity representations over continued pre-training, grouped by the deepest level of the ground-truth taxonomy that the two entities share. Similarity increases monotonically with shared depth (L1 < L2 < L3). Shaded regions indicate the standard error of the mean across three random seeds.
Figure 10: Hierarchical recovery of the taxonomy from entity representations of Pythia 160M, before (top) and after (bottom) continued pre-training on entities and properties. Bars below each dendrogram mark the ground-truth L3 and L2 categories.
Figure 11: Logp(property∣entity) over the last 6,000 continued pre-training steps of Pythia 160M (left) and OLMo 2 1B (right), after synthetic data injection stops at step 5,000. Shaded regions indicate the standard error of the mean across three random seeds.
Figure 12: Memorization over training (left) and early induction advantage, averaged over steps 0–1,000 (right), versus model size for the Pythia family. Scale increases the speed of memorization while reducing correct inductive generalization. Circles indicate induction advantage for individual evaluation templates, while horizontal lines show the average. All reported values are averages across three random seeds, and the shaded regions show the standard error of the mean across those seeds.
Figure 13: Comparison for Pythia 160M between the batch size used in all our experiments (32 sequences; 65k tokens) and the batch size of the original Pythia training run ( Biderman et al., 2023 ) (1,024 sequences; 2M tokens). Shaded regions indicate the standard error of the mean across three random seeds.
Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts, which can be recalled but do not support such generalization in a given task. We study this distinction by constructing new relations from existing ones. Take the relation that maps a place to its continent (Paris -> Europe) and assign each of its answers an arbitrary new label (Europe -> crab, Asia -> zebra). The new relation then maps Paris -> crab. We fine-tune the model on the new relation for a subset of subjects, without telling it the original relation or the relabeling rule, and test whether it assigns the corresponding labels to held-out subjects that share an answer in the original relation. Having learned Paris -> crab, does it answer Munich -> crab? Across eleven language models and 70 relations, how linearly readable the original relation is in pretrained representations strongly predicts this generalization. Within each model, the two correlate across relations at Pearson r = 0.87 to 0.92. This association persists when controlling, separately and jointly, for the number of fine-tuning subjects and answers, the model's accuracy on the original relation, and how often the subjects occur in pretraining text. It is also robust to re-randomizing the label assignment and fine-tuning seed, and to replacing the animal labels with labels from thirteen other semantic categories. When we map a fine-tuned model's wrong label back to the original answer it was assigned to, Munich -> zebra becomes Munich -> Asia. Answers recovered this way are more often among the linear probe's top answers for the subject than among the pretrained model's own. Whether a new relation generalizes depends less on whether its facts can be recalled than on how readily their shared structure can be read out linearly.
Faridoun Mehri, Hinrich Schütze
Center for Information and Language Processing, LMU Munich Munich Center for Machine Learning (MCML)
Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (ρ=.81 across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (R2=.68-.84 across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.
Emmy Liu, Kaiser Sun, Millicent Li +4
Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1