cs.CLApr 9, 2026

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Authors: Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig

Organizations: Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University · Department of Computer Science, University of Southern California

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (ρ=.81ρ= .81 across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (R2=.68R^2 = .68-.84.84 across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.

Figures & tables

Figure 1: Emergence order across model families and sizes, smoothed with a Gaussian kernel ( σ=1.0\sigma=1.0 ). Dots represent the point at which the model reaches a fixed 50% accuracy threshold. While the absolute emergence time varies across models, the ordering shows regularity.
TaskInputOut
simple_icl: present_to_gerundrunrunning
fact_extraction: extract_numberPassage: “John gave 5 apples to Mary on Tuesday.” How many apples?5
logical_ops: conditionalIf it rains, the ground gets wet. It rains. Is the ground wet?yes
Compositional tasks
comp: gerund_upperrunRUNNING
comp: translate_sp_eng_reverseholaolleh
Table 1: Example simple and compositional tasks. Compositional tasks require chaining multiple primitive skills.
Figure 2: Emergence order heatmap of selected tasks across models (absolute threshold = 0.8). Tasks sorted by consensus emergence order. Consistent color gradients across columns indicate stable ordering.
 OLMo2OLMo3LLM360Pythia
 1B7B13B7BAmberCrystal410M1.4B2.8B12B
O2-1B—.80.72.75.88.76.84.92.78.84
O2-7B —.93.93.88.92.71.72.77.84
O2-13B  —.93.82.89.64.66.76.83
O3-7B   —.83.92.70.72.79.85
Amber    —.87.77.84.87.90
Table 2: Spearman rank correlation ( ρ\rho ) of emergence orderings between model pairs (absolute threshold = 80%). All 45 correlations are significant ( p<10−7p<10^{-7} ).
 All tasksSim. only
ModelR2R^{2}MAER2R^{2}MAE
Pythia-410M.681.195.717.301
OLMo2-1B.723.070.602.289
Pythia-1.4B.778.086.755.193
Amber (7B).751.082.725.205
Crystal (7B).676.133.568.315
Table 3: Leave-one-out prediction of held-out composite task trajectories (26 tasks). All tasks includes simple and composite tasks. Sim. only includes only simple tasks. Restricting to elementals degrades MAE for every model (mean Δ\Delta MAE =+.135=+.135 ), indicating a composition bottleneck.
Figure 3: Example composite task predictions for OLMo2-7B between 0-1T tokens.
Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

TaskNInputOutput
String Operations
copying20gTpigTHKgTpigTHK
token_reversal20cattac
string_analogy10abc →\to abd, ijk →\to ?ijl
simple_icl:uppercase26bB
simple_icl:lowercase26Bb
Appendix
Table 4: All elemental tasks in the evaluation suite with representative examples.
TaskNInputOutput
Morphology ×\times String Operation
compositional:gerund_lower178RUNrunning
compositional:gerund_upper178runRUNNING
compositional:gerund_reverse178rungninnur
compositional:gerund_upper_reverse178runGNINNUR
compositional:plural_lower165CHILDchildren
Appendix
Table 5: All compositional tasks in the evaluation suite with representative examples.
Figure 4: Complete trajectories for Pythia-410M over 300B tokens.
Figure 5: Complete trajectories for OLMo2-1B over 1T tokens.
Figure 6: Complete trajectories for Pythia-1.4B over 300B tokens.
Figure 7: Complete trajectories for OLMo-2 7B over 1T tokens.
Figure 8: Complete trajectories for OLMo-3 7B over 1T tokens.
Figure 9: Complete trajectories for Amber (7B) over 1T tokens.
Figure 10: Complete trajectories for CrystalCoder (7B) over 1T tokens.
Figure 11: Complete trajectories for Pythia-12B over 300B tokens. Note that this model exhibits some instabilities compared to others.
Figure 12: Complete trajectories for OLMo2-13b over 1T tokens. Note that this model exhibits some instabilities compared to others.
Definitionnn pairsMean ρ\rhoMin ρ\rhoMax ρ\rho
Absolute threshold ( θ=0.5\theta=0.5 )360.8600.5970.955
Absolute threshold ( θ=0.8\theta=0.8 , stable for 3 consecutive checkpoints)360.7900.5990.961
Relative threshold ( α=0.5\alpha=0.5 , fraction of max performance)360.5280.0850.866
Relative threshold ( α=0.8\alpha=0.8 , fraction of max performance)360.5000.0770.773
Appendix
Table 6: Summary of emergence ordering consistency under different definitions. Absolute thresholds yield substantially higher cross-model correlations than relative thresholds.
Model AModel Bnn tasksρ\rhopp
Within-family (OLMo-2)
OLMo2-1BOLMo2-7B1060.8894.5×10−374.5\times 10^{-37}
OLMo2-1BOLMo2-13B1050.8324.5×10−284.5\times 10^{-28}
OLMo2-7BOLMo2-13B1040.8652.9×10−322.9\times 10^{-32}
Within-family (Pythia)
Pythia-1.4BPythia-410M960.9101.3×10−371.3\times 10^{-37}
Appendix
Table 7: Pairwise Spearman rank correlations: Absolute threshold ( θ=0.5\theta=0.5 ). Mean ρ=0.860\rho=0.860 .
Model AModel Bnn tasksρ\rhopp
Within-family (OLMo-2)
OLMo2-1BOLMo2-7B1060.7184.5×10−184.5\times 10^{-18}
OLMo2-1BOLMo2-13B1050.7214.0×10−184.0\times 10^{-18}
OLMo2-7BOLMo2-13B1040.9342.0×10−472.0\times 10^{-47}
Within-family (Pythia)
Pythia-1.4BPythia-410M960.8246.1×10−256.1\times 10^{-25}
Appendix
Table 8: Pairwise Spearman rank correlations: Absolute threshold ( θ=0.8\theta=0.8 , stable for 3 consecutive checkpoints). Mean ρ=0.790\rho=0.790 .
Model AModel Bnn tasksρ\rhopp
Within-family (OLMo-2)
OLMo2-1BOLMo2-7B1060.5798.2×10−118.2\times 10^{-11}
OLMo2-1BOLMo2-13B1050.4333.9×10−63.9\times 10^{-6}
OLMo2-7BOLMo2-13B1040.5634.8×10−104.8\times 10^{-10}
Within-family (Pythia)
Pythia-1.4BPythia-410M960.8665.3×10−305.3\times 10^{-30}
Appendix
Table 9: Pairwise Spearman rank correlations: Relative threshold ( α=0.5\alpha=0.5 , fraction of max performance). Mean ρ=0.528\rho=0.528 .
Model AModel Bnn tasksρ\rhopp
Within-family (OLMo-2)
OLMo2-1BOLMo2-7B1060.4918.9×10−88.9\times 10^{-8}
OLMo2-1BOLMo2-13B1050.3591.7×10−41.7\times 10^{-4}
OLMo2-7BOLMo2-13B1040.7074.6×10−174.6\times 10^{-17}
Within-family (Pythia)
Pythia-1.4BPythia-410M960.7162.3×10−162.3\times 10^{-16}
Appendix
Table 10: Pairwise Spearman rank correlations: Relative threshold ( α=0.8\alpha=0.8 , fraction of max performance). Mean ρ=0.500\rho=0.500 .
ModelRepresentationLayerk headsσ\sigmaλ\lambda
LLM360
amberhidden states21—6.025680.0001
crystalhidden states8—6.258220.0001
Pythia
pythia_410mhidden states3—0.339910.001
pythia_1.4bhidden states12—5.936390.001
Appendix
Table 11: Function vector hyperparameters selected per model. The full residual stream was chosen as the representation for all models besides OLMo2-13B, in which the top 10 heads by causal indirect effect in layer 10 were chosen. σ\sigma and λ\lambda are the parameters used in ridge regression.
Figure 13: Predictions of held-out compositional tasks for Pythia-410M.
Figure 14: Predictions of held-out compositional tasks for OLMo2-1B.
Figure 15: Predictions of held-out compositional tasks for Pythia-1.4B.
Figure 16: Predictions of held-out compositional tasks for OLMo2-7B.
Figure 17: Predictions of held-out compositional tasks for OLMo3-7B.
Figure 18: Predictions of held-out compositional tasks for Amber.
Figure 19: Predictions of held-out compositional tasks for CrystalCoder.
Figure 20: Predictions of held-out compositional tasks for Pythia-12B.
Figure 21: Predictions of held-out compositional tasks for OLMo2-13B.

Explore similar work

CardsList

  1. Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

    Aug 4, 2026Jo-Ku Cheng, Nikolaos Aletras, Marco ValentinoLanguage Model PretrainingRepresentation Geometry in Language Models

    School of Computer Science, University of Sheffield, United Kingdom

  2. Small Initialization Matters for Large Language Models

    Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model PretrainingNeural Network Initialization

    School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.

  3. Tracing Morphology, Semantics, and Divergence in Language Acquisition

    Oct 13, 2026Forough Shirin Abkenar, Haoshuo Fu, Liang Feng +2Language Modeling

    Futurewei Technologies, USA