What do Language Models Learn and When? The Implicit Curriculum Hypothesis
Authors: Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig
Organizations: Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University · Department of Computer Science, University of Southern California
Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (ρ=.81 across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (R2=.68-.84 across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.
Figures & tables
Figure 1: Emergence order across model families and sizes, smoothed with a Gaussian kernel ( σ=1.0 ). Dots represent the point at which the model reaches a fixed 50% accuracy threshold. While the absolute emergence time varies across models, the ordering shows regularity.
Task
Input
Out
simple_icl: present_to_gerund
run
running
fact_extraction: extract_number
Passage: “John gave 5 apples to Mary on Tuesday.” How many apples?
5
logical_ops: conditional
If it rains, the ground gets wet. It rains. Is the ground wet?
yes
Compositional tasks
comp: gerund_upper
run
RUNNING
comp: translate_sp_eng_reverse
hola
olleh
Table 1: Example simple and compositional tasks. Compositional tasks require chaining multiple primitive skills.
Figure 2: Emergence order heatmap of selected tasks across models (absolute threshold = 0.8). Tasks sorted by consensus emergence order. Consistent color gradients across columns indicate stable ordering.
OLMo2
OLMo3
LLM360
Pythia
1B
7B
13B
7B
Amber
Crystal
410M
1.4B
2.8B
12B
O2-1B
—
.80
.72
.75
.88
.76
.84
.92
.78
.84
O2-7B
—
.93
.93
.88
.92
.71
.72
.77
.84
O2-13B
—
.93
.82
.89
.64
.66
.76
.83
O3-7B
—
.83
.92
.70
.72
.79
.85
Amber
—
.87
.77
.84
.87
.90
Table 2: Spearman rank correlation ( ρ ) of emergence orderings between model pairs (absolute threshold = 80%). All 45 correlations are significant ( p<10−7 ).
All tasks
Sim. only
Model
R2
MAE
R2
MAE
Pythia-410M
.681
.195
.717
.301
OLMo2-1B
.723
.070
.602
.289
Pythia-1.4B
.778
.086
.755
.193
Amber (7B)
.751
.082
.725
.205
Crystal (7B)
.676
.133
.568
.315
Table 3: Leave-one-out prediction of held-out composite task trajectories (26 tasks). All tasks includes simple and composite tasks. Sim. only includes only simple tasks. Restricting to elementals degrades MAE for every model (mean Δ MAE =+.135 ), indicating a composition bottleneck.
Figure 3: Example composite task predictions for OLMo2-7B between 0-1T tokens.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Task
N
Input
Output
String Operations
copying
20
gTpigTHK
gTpigTHK
token_reversal
20
cat
tac
string_analogy
10
abc → abd, ijk → ?
ijl
simple_icl:uppercase
26
b
B
simple_icl:lowercase
26
B
b
Appendix
Table 4: All elemental tasks in the evaluation suite with representative examples.
Task
N
Input
Output
Morphology × String Operation
compositional:gerund_lower
178
RUN
running
compositional:gerund_upper
178
run
RUNNING
compositional:gerund_reverse
178
run
gninnur
compositional:gerund_upper_reverse
178
run
GNINNUR
compositional:plural_lower
165
CHILD
children
Appendix
Table 5: All compositional tasks in the evaluation suite with representative examples.
Figure 4: Complete trajectories for Pythia-410M over 300B tokens.
Figure 5: Complete trajectories for OLMo2-1B over 1T tokens.
Figure 6: Complete trajectories for Pythia-1.4B over 300B tokens.
Figure 7: Complete trajectories for OLMo-2 7B over 1T tokens.
Figure 8: Complete trajectories for OLMo-3 7B over 1T tokens.
Figure 9: Complete trajectories for Amber (7B) over 1T tokens.
Figure 10: Complete trajectories for CrystalCoder (7B) over 1T tokens.
Figure 11: Complete trajectories for Pythia-12B over 300B tokens. Note that this model exhibits some instabilities compared to others.
Figure 12: Complete trajectories for OLMo2-13b over 1T tokens. Note that this model exhibits some instabilities compared to others.
Definition
n pairs
Mean ρ
Min ρ
Max ρ
Absolute threshold ( θ=0.5 )
36
0.860
0.597
0.955
Absolute threshold ( θ=0.8 , stable for 3 consecutive checkpoints)
36
0.790
0.599
0.961
Relative threshold ( α=0.5 , fraction of max performance)
36
0.528
0.085
0.866
Relative threshold ( α=0.8 , fraction of max performance)
36
0.500
0.077
0.773
Appendix
Table 6: Summary of emergence ordering consistency under different definitions. Absolute thresholds yield substantially higher cross-model correlations than relative thresholds.
Table 8: Pairwise Spearman rank correlations: Absolute threshold ( θ=0.8 , stable for 3 consecutive checkpoints). Mean ρ=0.790 .
Model A
Model B
n tasks
ρ
p
Within-family (OLMo-2)
OLMo2-1B
OLMo2-7B
106
0.579
8.2×10−11
OLMo2-1B
OLMo2-13B
105
0.433
3.9×10−6
OLMo2-7B
OLMo2-13B
104
0.563
4.8×10−10
Within-family (Pythia)
Pythia-1.4B
Pythia-410M
96
0.866
5.3×10−30
Appendix
Table 9: Pairwise Spearman rank correlations: Relative threshold ( α=0.5 , fraction of max performance). Mean ρ=0.528 .
Model A
Model B
n tasks
ρ
p
Within-family (OLMo-2)
OLMo2-1B
OLMo2-7B
106
0.491
8.9×10−8
OLMo2-1B
OLMo2-13B
105
0.359
1.7×10−4
OLMo2-7B
OLMo2-13B
104
0.707
4.6×10−17
Within-family (Pythia)
Pythia-1.4B
Pythia-410M
96
0.716
2.3×10−16
Appendix
Table 10: Pairwise Spearman rank correlations: Relative threshold ( α=0.8 , fraction of max performance). Mean ρ=0.500 .
Model
Representation
Layer
k heads
σ
λ
LLM360
amber
hidden states
21
—
6.02568
0.0001
crystal
hidden states
8
—
6.25822
0.0001
Pythia
pythia_410m
hidden states
3
—
0.33991
0.001
pythia_1.4b
hidden states
12
—
5.93639
0.001
Appendix
Table 11: Function vector hyperparameters selected per model. The full residual stream was chosen as the representation for all models besides OLMo2-13B, in which the top 10 heads by causal indirect effect in layer 10 were chosen. σ and λ are the parameters used in ridge regression.
Figure 13: Predictions of held-out compositional tasks for Pythia-410M.
Figure 14: Predictions of held-out compositional tasks for OLMo2-1B.
Figure 15: Predictions of held-out compositional tasks for Pythia-1.4B.
Figure 16: Predictions of held-out compositional tasks for OLMo2-7B.
Figure 17: Predictions of held-out compositional tasks for OLMo3-7B.
Figure 18: Predictions of held-out compositional tasks for Amber.
Figure 19: Predictions of held-out compositional tasks for CrystalCoder.
Figure 20: Predictions of held-out compositional tasks for Pythia-12B.
Figure 21: Predictions of held-out compositional tasks for OLMo2-13B.
Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at ≈33% sparsity.
Jo-Ku Cheng, Nikolaos Aletras, Marco Valentino
School of Computer Science, University of Sheffield, United Kingdom
Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple γ-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.
Liangkai Hang, Junjie Yao, Zhiyu Li +3
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.
Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.