What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Chang Su; Bin Wu; Emine Yilmaz; Yao Lu

cs.AIOct 27, 2026

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Authors: Chang Su, Bin Wu, Emine Yilmaz, Yao Lu

Organizations: University of Pennsylvania · Centre for Artificial Intelligence, University College London

Abstract

Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

Figures & tables

Benchmark	TVS group	Example (question $\rightarrow$ gold answer)
PIQA	Low	“How do you take soap off of clean dishes?” $\rightarrow$ “rinse it in water.”
PIQA	High	“To tell when baked goods are done baking, you can” $\rightarrow$ “Wait for them to turn golden brown.”
ARC-Easy	Low	“What causes a change in the speed of a moving object?” $\rightarrow$ “Force.”
ARC-Easy	High	“A meteoroid becomes a meteorite when it” $\rightarrow$ “impacts the surface of a planet.”

Table 1: Sampled items from the low- and high-

\mathrm{TVS}

groups.

Figure 1: Chance-normalized accuracy over pretraining for temporally variable (high $\mathrm{TVS}$ , red) and temporally stable (low $\mathrm{TVS}$ , green) items, by model (columns) and benchmark (rows). Groups are length-matched; training tokens are on a log scale. Pythia and OLMo show little early separation; high- $\mathrm{TVS}$ items generally pull ahead later.

	Early	Final
Benchmarks	L $>$ H	H $>$ L	Gap	Gap $\mid$ NLL
All	13/24	34/36	+7.3	+3.7
HS/PIQA/WG	8/12	16/18	+8.5	+0.5
ARC/MMLU	5/12	18/18	+6.2	+6.9

Table 2: Low- (L) versus high-TVS (H) items at the earliest and final checkpoints. Early counts cover Pythia and OLMo, whose first checkpoints come early in training; final counts cover all six models. Gaps are high-minus-low chance-normalized accuracy in points. Gap

\mid

NLL uses a split conditioned on mean NLL (§ 4.2 ).

Split (within length bins)	Final H $>$ L	Gap
$\mathrm{TVS}$ (main)	34/36	+7.3
$\mathrm{TVS}$ $\mid$ mean NLL	26/36	+3.7
$\mathrm{TVS}$ $\mid$ mean NLL, PIT margin, PIT acc.	22/36	+1.4
Gold-vs.-wrong $\mathrm{TVS}$ margin	36/36	+12.3
Question-only $\mathrm{TVS}$	26/36	+2.1
NLL standard deviation	8/36	$-$ 2.6

Table 3: Final-checkpoint comparison of high- versus low-group items under alternative stratifications (chance-normalized points; 36 model–benchmark settings). Conditioned splits use the residual of

\mathrm{TVS}

after regressing out the listed covariates within each benchmark and length bin. Full results are in Table 5 .

Benchmark	Final correct	Late margin slope
ARC-C	+0.024 (.04)	+0.070 ( $2\times 10^{-9}$ )
ARC-E	+0.008 (.2)	+0.059 ( $2\times 10^{-8}$ )
MMLU	+0.001 (.8)	+0.043 ( $2\times 10^{-7}$ )
PIQA	+0.001 (.8)	+0.011 ( $2\times 10^{-4}$ )
HellaSwag	+0.019 ( $10^{-5}$ )	0.000 (.8)
WinoGrande	+0.012 (.01)	+0.002 (.08)

Table 4: Difficulty-controlled effect of gold-specific

\mathrm{TVS}

(per standard deviation) on final correctness and late gold-versus-distractor margin growth (

p

-values in parentheses; Appendix D ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Stratification	Early L $>$ H	Final H $>$ L	Best H	Mean final H $-$ L
Gold TVS tertiles (main)	15/36	34/36	28/36	+0.073
Gold TVS quintiles	15/36	32/36	25/36	+0.081
Mean gold NLL quintiles	35/36	0/36	0/36	$-$ 0.311
Question-only TVS tertiles	13/36	26/36	20/36	+0.021
Gold-vs.-wrong TVS margin tertiles	20/36	36/36	35/36	+0.123
Gold NLL-SD tertiles	30/36	8/36	7/36	$-$ 0.026

Appendix

Table 5: Alternative stratifications across 36 model–benchmark settings. Mean final H

-

L is the final-checkpoint high-minus-low gap in chance-normalized accuracy. For mean gold NLL, “high” is the higher-NLL group; for the gold-vs.-wrong margin, “high” has the larger

\mathrm{TVS}(q,a^{\mathrm{gold}})-\mathrm{TVS}(q,a^{\mathrm{wrong}})

. All splits are formed within ten length bins per benchmark; the question-only split bins by question length. NLL-SD is the unnormalized standard deviation of gold NLL across years. Rank disagreement is the standard deviation across years of an item’s NLL percentile within its benchmark and length bin. The conditioned rows split on the residual of

\mathrm{TVS}

after regressing it (with quadratic terms) on the listed covariates within each benchmark and length bin (Appendix B.1 ).

Figure 2: Main result with five $\mathrm{TVS}$ quantiles. The most variable quintile ends ahead of the least variable one in 32 of 36 settings; all four exceptions are on WinoGrande.

Signal	Control	Pearson	Spearman
$\mathrm{CV}_{\mathrm{PPL}}$	log mean PPL	0.486	0.426
$\mathrm{CV}_{\mathrm{PPL}}$	mean NLL	0.443	0.423
$\mathrm{CV}_{\mathrm{PPL}}$	length	$-$ 0.169	$-$ 0.331
$\mathrm{CV}_{\mathrm{PPL}}$	log length	$-$ 0.223	$-$ 0.406
$\mathrm{TVS}$	log mean PPL	0.166	0.011
$\mathrm{TVS}$	mean NLL	0.134	0.007

Appendix

Table 6: Correlations (with benchmark fixed effects) between the two dispersion measures (TVS and perplexity-based CV) and simple item properties over all 16,239 items. Length is the PIT-4B token length of the gold string.

\mathrm{CV}_{\mathrm{PPL}}

is strongly entangled with mean difficulty and length.

\mathrm{TVS}

is much less related to mean difficulty but still correlates with length, which motivates length-stratified grouping; after length control, correlations with length and mean PPL are near zero.

Figure 3: Split by mean gold NLL. Higher mean NLL is associated with lower accuracy at all checkpoints. Difficulty-conditioned $\mathrm{TVS}$ results are reported separately in Appendix B.1 .

Figure 4: Split by question-only $\mathrm{TVS}$ . The effect is much weaker than for the question and gold answer together, so the signal is not reducible to the question stem.

Figure 5: Split by the gold-versus-wrong $\mathrm{TVS}$ margin. Items whose gold answer is more temporally variable than the best wrong answer show the largest final-checkpoint advantage.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 447 papers

Understanding Data Temporality Impact on Large Language Models Pre-training
May 21, 2026Hippolyte Pilchen, Romain Fabre, Franck Signe Talla +2Language Model Pretraining Temporal Reasoning in Language Models
Kyutai, Paris
Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency
Oct 9, 2026Sheng Zhao, Yuhao ZhuContinual Learning for LLMs Continued Pretraining
Department of Computer Science University of Rochester
When to Review: Spaced Repetition for Continual Pre-Training of Language Models
Aug 18, 2026Alankar Atreya, Devesh Batra, Yoages Kumar Mantri +3Continual Learning for LLMs Continued Pretraining
NatWest AI Research · University College London

May 21, 2026J/K move · Enter open · S save

Understanding Data Temporality Impact on Large Language Models Pre-training

Hippolyte Pilchen, Romain Fabre, Franck Signe Talla, Patrick Perez+1

Kyutai, Paris

Language Model Pretraining Temporal Reasoning in Language Models Factual Knowledge in Language Models Temporal QA

Open paper AskPDF arXiv

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.

Abstract

Language Model Pretraining

Explore similar work

Understanding Data Temporality Impact on Large Language Models Pre-training

Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency

When to Review: Spaced Repetition for Continual Pre-Training of Language Models