Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.
Figures & tables
Benchmark
TVS group
Example (question → gold answer)
PIQA
Low
“How do you take soap off of clean dishes?” → “rinse it in water.”
PIQA
High
“To tell when baked goods are done baking, you can” → “Wait for them to turn golden brown.”
ARC-Easy
Low
“What causes a change in the speed of a moving object?” → “Force.”
ARC-Easy
High
“A meteoroid becomes a meteorite when it” → “impacts the surface of a planet.”
Table 1: Sampled items from the low- and high- TVS groups.
Figure 1: Chance-normalized accuracy over pretraining for temporally variable (high TVS , red) and temporally stable (low TVS , green) items, by model (columns) and benchmark (rows). Groups are length-matched; training tokens are on a log scale. Pythia and OLMo show little early separation; high- TVS items generally pull ahead later.
Early
Final
Benchmarks
L > H
H > L
Gap
Gap ∣ NLL
All
13/24
34/36
+7.3
+3.7
HS/PIQA/WG
8/12
16/18
+8.5
+0.5
ARC/MMLU
5/12
18/18
+6.2
+6.9
Table 2: Low- (L) versus high-TVS (H) items at the earliest and final checkpoints. Early counts cover Pythia and OLMo, whose first checkpoints come early in training; final counts cover all six models. Gaps are high-minus-low chance-normalized accuracy in points. Gap ∣ NLL uses a split conditioned on mean NLL (§ 4.2 ).
Split (within length bins)
Final H > L
Gap
TVS (main)
34/36
+7.3
TVS∣ mean NLL
26/36
+3.7
TVS∣ mean NLL, PIT margin, PIT acc.
22/36
+1.4
Gold-vs.-wrong TVS margin
36/36
+12.3
Question-only TVS
26/36
+2.1
NLL standard deviation
8/36
− 2.6
Table 3: Final-checkpoint comparison of high- versus low-group items under alternative stratifications (chance-normalized points; 36 model–benchmark settings). Conditioned splits use the residual of TVS after regressing out the listed covariates within each benchmark and length bin. Full results are in Table 5 .
Benchmark
Final correct
Late margin slope
ARC-C
+0.024 (.04)
+0.070 ( 2×10−9 )
ARC-E
+0.008 (.2)
+0.059 ( 2×10−8 )
MMLU
+0.001 (.8)
+0.043 ( 2×10−7 )
PIQA
+0.001 (.8)
+0.011 ( 2×10−4 )
HellaSwag
+0.019 ( 10−5 )
0.000 (.8)
WinoGrande
+0.012 (.01)
+0.002 (.08)
Table 4: Difficulty-controlled effect of gold-specific TVS (per standard deviation) on final correctness and late gold-versus-distractor margin growth ( p -values in parentheses; Appendix D ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Stratification
Early L > H
Final H > L
Best H
Mean final H − L
Gold TVS tertiles (main)
15/36
34/36
28/36
+0.073
Gold TVS quintiles
15/36
32/36
25/36
+0.081
Mean gold NLL quintiles
35/36
0/36
0/36
− 0.311
Question-only TVS tertiles
13/36
26/36
20/36
+0.021
Gold-vs.-wrong TVS margin tertiles
20/36
36/36
35/36
+0.123
Gold NLL-SD tertiles
30/36
8/36
7/36
− 0.026
Appendix
Table 5: Alternative stratifications across 36 model–benchmark settings. Mean final H − L is the final-checkpoint high-minus-low gap in chance-normalized accuracy. For mean gold NLL, “high” is the higher-NLL group; for the gold-vs.-wrong margin, “high” has the larger TVS(q,agold)−TVS(q,awrong) . All splits are formed within ten length bins per benchmark; the question-only split bins by question length. NLL-SD is the unnormalized standard deviation of gold NLL across years. Rank disagreement is the standard deviation across years of an item’s NLL percentile within its benchmark and length bin. The conditioned rows split on the residual of TVS after regressing it (with quadratic terms) on the listed covariates within each benchmark and length bin (Appendix B.1 ).
Figure 2: Main result with five TVS quantiles. The most variable quintile ends ahead of the least variable one in 32 of 36 settings; all four exceptions are on WinoGrande.
Signal
Control
Pearson
Spearman
CVPPL
log mean PPL
0.486
0.426
CVPPL
mean NLL
0.443
0.423
CVPPL
length
− 0.169
− 0.331
CVPPL
log length
− 0.223
− 0.406
TVS
log mean PPL
0.166
0.011
TVS
mean NLL
0.134
0.007
Appendix
Table 6: Correlations (with benchmark fixed effects) between the two dispersion measures (TVS and perplexity-based CV) and simple item properties over all 16,239 items. Length is the PIT-4B token length of the gold string. CVPPL is strongly entangled with mean difficulty and length. TVS is much less related to mean difficulty but still correlates with length, which motivates length-stratified grouping; after length control, correlations with length and mean PPL are near zero.
Figure 3: Split by mean gold NLL. Higher mean NLL is associated with lower accuracy at all checkpoints. Difficulty-conditioned TVS results are reported separately in Appendix B.1 .
Figure 4: Split by question-only TVS . The effect is much weaker than for the question and gold answer together, so the signal is not reducible to the question stem.
Figure 5: Split by the gold-versus-wrong TVS margin. Items whose gold answer is more temporally variable than the best wrong answer show the largest final-checkpoint advantage.
Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.
An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings. Project page: https://horizon-lab.org/specialist/
Sheng Zhao, Yuhao Zhu
Department of Computer Science University of Rochester
Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.