Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models
Organizations: University of Maryland · Pr(Ai)2R Group · Stanford University · Bar-Ilan University ·
Abstract
How does data shape language model (LM) behavior throughout pretraining? We investigate this question through a case study on entity comparison, e.g., Between France and Brazil, which country is larger?, across nine comparison tasks spanning five entity types. We begin with controlled experiments in which we train small LMs (163M parameters) on mixtures of natural text from pretraining corpora and synthetic task data. Our analysis identifies three distinct phases of learning: (1) an early phase where the LM selects entities by frequency, (2) a middle phase where the LM selects entities by position in the prompt (first vs. last mentioned entity), and (3) a late phase where the LM selects the entity that is the correct answer. We show that the emergence of these three phases is causally determined by statistical properties of the training data. With small amounts of task-specific synthetic data, we observe only the first two phases and the model fails to learn the task; with large amounts, the model jumps directly from the frequency-based heuristic to solving the task correctly. Moreover, changing the entity frequency distribution from a naturally occurring Zipfian distribution (a small number of entities are very common and the vast majority are rare) to a uniform one weakens the frequency phase and speeds up learning of the task. Lastly, we show that the three phases of training dynamics from our controlled experiments appear in the open source model OLMo-2 32B. Together, our findings demonstrate that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.
Figures & tables
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Entity | Domain | Example prompt |
|---|---|---|
| Country | Area | By total area, among the two countries France and Brazil, the larger country is |
| U.S. state | Area | By total area, among the two U.S. states Texas and Ohio, the larger state is |
| U.S. state | Population | By population, among the two U.S. states California and Vermont, the more populous state is |
| Person | Birth year | Between the two people Socrates and Barack Obama, the one born more recently is |
| Movie | Release year | Between the two films Casablanca and Inception, the one released more recently is |
| Movie | Box office | By worldwide box office gross, among the two films The Dark Knight and Pulp Fiction, the higher-grossing film is |
| Name | Definition | Example |
|---|---|---|
| Value-based features | ||
| value_left | ||
| value_right | ||
| value_left_log | ||
| value_right_log | ||
| value_ratio | ||
| Task | Ckpt | Position drop acc. | First position | Second position | Always correct | Always incorrect |
|---|---|---|---|---|---|---|
| People birth year | 84B | 32.7 | 0.1 | 72.5 | 19.5 | 7.9 |
| Movie box office | 1T | 7.9 | 42.3 | 0.3 | 43.2 | 14.2 |
| University founding year | 1T | 8.2 | 0.9 | 34.1 | 34.6 | 30.4 |
| University student population | 5T | 8.8 | 37.2 | 0.3 | 53.2 | 9.3 |
| Country area | 5T | 0.1 | 8.2 | 4.6 | 78.1 | 9.1 |
| Run | Synth tokens / epoch | Natural tokens / epoch |
|---|---|---|
| 0% | 0 | 1,000,000,288 |
| 0.01% | 100,352 | 999,899,936 |
| 0.1% | 1,000,448 | 998,999,840 |
| 1% | 10,000,384 | 989,999,104 |
| Condition | Gini coefficient | Max/min ratio | P 95 /P 5 ratio |
|---|---|---|---|
| OLMo-mix-1124 (reference) | 0.772 | 8,600 | 470 |
| Natural Zipfian + Synthetic Zipfian | 0.722 | 6,800 | 300 |
| Natural Zipfian + Synthetic Uniform | 0.671 | 320 | 69 |
| Natural Uniform + Synthetic Zipfian | 0.071 | 65 | 1 |
| Natural Uniform + Synthetic Uniform | 0.031 | 11 | 1 |
| OLMo-2 1B | OLMo-2 7B | OLMo-2 13B | OLMo-2 32B | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Step | Tokens | Acc. | Step | Tokens | Acc. | Step | Tokens | Acc. | Step | Tokens | Acc. |
| 1,000 | 3B | 0.598 | 1,000 | 5B | 0.651 | 1,000 | 9B | 0.667 | 1,000 | 9B | 0.564 |
| 2,000 | 5B | 0.667 | 2,000 | 9B | 0.668 | 2,000 | 17B | 0.682 | 2,000 | 17B | 0.615 |
| 3,000 | 7B | 0.640 | 3,000 | 13B | 0.657 | 3,000 | 26B | 0.537 | 3,000 | 26B | 0.543 |
| 4,000 | 9B | 0.623 | 4,000 | 17B | 0.680 | 4,000 | 34B | 0.544 | 4,000 | 34B | 0.543 |
| 5,000 | 11B | 0.637 | 5,000 | 21B | 0.534 | 5,000 | 42B | 0.571 | 5,000 | 42B | 0.613 |
| Task | Step 200 | Step 400 | Step 600 | Step 800 | Step 1000 |
|---|---|---|---|---|---|
| Country area | Frequency (5.2) | Frequency (8.3) | Frequency (7.1) | Frequency (6.4) | Frequency (6.0) |
| Person birth year | Frequency (5.6) | Frequency (9.2) | Frequency (9.2) | Frequency (9.2) | Frequency (5.4) |
| Movie box office | Value (1.6) | Value (3.4) | Value (2.2) | Value (5.4) | Value (4.2) |
| Movie release year | Frequency (0.6) | Frequency (2.0) | Frequency (1.4) | Frequency (1.7) | Frequency (0.8) |
| State area | Frequency (21.3) | Frequency (11.9) | Frequency (12.4) | Frequency (10.9) | Frequency (8.3) |
| State population | Value (1.9) | Value (0.2) | Value (1.8) | Value (2.4) | Value (1.3) |
Language Model Pretraining
Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.
See all 473 papers