Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency

Sheng Zhao; Yuhao Zhu

cs.LGOct 9, 2026

Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency

Authors: Sheng Zhao, Yuhao Zhu

Organizations: Department of Computer Science University of Rochester

Abstract

An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and that closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings. Project page: https://horizon-lab.org/specialist/

Figures & tables

	Efficient updating & editing	Catastrophic forgetting
Pre-training	Ours Liu et al. (2025) , Kim et al. (2025)	Toneva et al. (2019) , Tirumala et al. (2022) , Ibrahim et al. (2024) Ke et al. (2023) , Dohare et al. (2024)
Post-training	Ovadia et al. (2024) , EntiGraph ( Yang et al., 2025 ) , SEAL ( Zweiger et al., 2025 ) ROME ( Meng et al., 2022 ) , MEMIT ( Meng et al., 2023 ) , MEND ( Mitchell et al., 2022a )	Scialom et al. (2022) , Luo et al. (2023) , Gekhman et al. (2024) EWC ( Kirkpatrick et al., 2017 ) , GRACE ( Hartvigsen et al., 2023 ) , WISE ( Wang et al., 2024 )

Table 1: Prior work by stage and task. Red : through data organization and training dynamics; Blue : through architecture or hyperparameters. Ours is the only data organization-level work on efficient editing at pre-training stage.

Figure 1: Example of an overwrite update. Bars represent the model’s predicted probabilities over an attribute’s values. Starting from initial parameters

\theta_{0}

, a LoRA adapter is trained for

N

steps on the loss

\mathcal{L}(\theta)=-\log P_{\mathrm{new}}(\theta)

, until the new value’s probability passes its competitor’s,

P_{\mathrm{old}}

(the strongest value model held before updating).

Figure 2: Query accuracy of the 110M CD and SD models over five training epochs; color represents accuracy. (a, b) Query by year (for CD and SD). Rows 2005–2020 query knowledge of a year (e.g., “In 2005, …”), and the “current” row queries knowledge from the most recent batch the model is trained on. (c) Query by batch (for SD). Each row queries knowledge from one of the ten batches in the training corpus, from the first batch (0) to the last (1). See Appendix A for details.

Figure 3: Intuition for generalists vs. specialists and for why specialists update knowledge faster. (a) Loss landscapes of knowledge in two years ( $C_{1}$ and $C_{2}$ ) and of all knowledge ( $S$ ) over the same model parameter space (bottom plane). Each valley in $C_{1}$ and $C_{2}$ is a chronological specialist that learns the knowledge of that year well. Every valley in $S$ is a shuffled-batch specialist for a batch during shuffled training. The specialists of $C_{1}$ (orange) and $C_{2}$ (green) overlap in a region, which is where generalists lie. Different shuffled-batch specialists (blue) lie close to each other and all fall inside the generalist region. (b) Top: the loss of the new value ( $-\log P_{\mathrm{new}}$ , solid) and the old value ( $-\log P_{\mathrm{old}}$ , dashed) across the parameter space $\theta$ . An update starts at $\theta_{0,C}$ and $\theta_{0,S}$ for a specialist and a generalist, respectively; $D_{0,C}$ and $D_{0,S}$ are the initial learning gap, respectively, and $G_{0,C}$ and $G_{0,S}$ are the corresponding initial gradients. $\theta_{N,C}$ and $\theta_{N,S}$ are the points where the two losses cross, i.e., where each update completes, in the two cases, respectively. Bottom: the actual change of loss over update steps, comparing a 110M specialist and a 110M generalist; an update ends where they cross ( $P_{\mathrm{new}}=P_{\mathrm{old}}$ ), and the specialist crosses first ( $N_{C}<N_{S}$ ).

Figure 4: Update cost under chronologically ordered and shuffled training. (a) Predicted updating steps against observed steps; Each of the 18 models (3 model sizes in Table 4

\times

6 training corpora) is shown in two variants: pre-trained from scratch (stars) and continued from a pre-trained model (triangles). (b) We plot these 18 pre-trained models using their initial distance

D_{0}

and the learning speed

\eta A\|g_{0}\|_{1}

; each point is a median over 160 updates, and the dashed lines mark a constant number of steps (Eq. 3 ). Marker sizes are proportional to the model sizes. (c) Contribution to the initial gradient

\|g_{0}\|_{1}

from each layer and each block within a layer, for a generalist (above; trained on SD) and a specialist CD (below; trained on CD); the area of a square is the contribution of that block, the histograms sum the contributions by layer and by block (see details in Appendix C.3 ).

		Overwrite				Inject
Scale	Pair	steps	saved	$D_{0}$ ratio	$\\|g_{0}\\|_{1}$ ratio	steps	saved	$D_{0}$ ratio	$\\|g_{0}\\|_{1}$ ratio
110M	C / S	51.0 / 88.0	$+$ 42%	1.27 $\times$	2.82 $\times$	37.0 / 59.0	$+$ 37%	1.18 $\times$	2.68 $\times$
	CD / SD	53.0 / 73.0	$+$ 27%	1.03 $\times$	2.02 $\times$	37.0 / 52.0	$+$ 29%	1.05 $\times$	1.90 $\times$
	CD 1/2 / SD 1/2	53.0 / 75.0	$+$ 29%	1.00 $\times$	1.61 $\times$	33.0 / 55.0	$+$ 40%	0.96 $\times$	2.02 $\times$
350M	C / S	50.0 / 116	$+$ 57%	1.00 $\times$	2.71 $\times$	31.5 / 65.0	$+$ 52%	0.77 $\times$	2.44 $\times$
	CD / SD	55.0 / 90.0	$+$ 39%	0.83 $\times$	1.63 $\times$	32.0 / 57.0	$+$ 44%	0.79 $\times$	1.45 $\times$

Table 2: Update cost comparison between different model sizes and training strategies. We use three model sizes, and each model is trained under the six training corpora. An overwrite update changes a known person’s latest value and an inject update adds a fact about an unseen person. The “saved” column is the percentage of update steps that a chronological model saves over the corresponding shuffled model (all results have a paired Wilcoxon

p<0.05

). We also show the

D_{0}

ratio and the

\|g_{0}\|_{1}

ratio between a corresponding specialist-generalist pair.

Figure 5: Query accuracy and update cost of the Mix corpora (Section 6.1 ). (a) Dated query accuracy (2005–2020 average) against undated query accuracy of different pre-trained models. (b) Dated query accuracy of Mix-C through five epochs of training, for historical years (2005–2020) whose knowledge is shuffled and for recent records (2021–2024) whose knowledge is ordered. (c) Median steps per update (log scale) against forgetting (percentage of other people’s facts that are incorrectly changed), for single updates (star), batch updates (cross) and sequential updates (triangle).

Figure 6: (a) Undated query accuracy of the continual pre-training stage vs. the median update steps on overwrite updates, for Gemma-2-9B-it (Base) and its two continually pre-trained variants. (b) Update cost vs. percentage of forgetting for the four knowledge editors applied to the 110M Mix-C and Mix-S models; each model’s four editors are joined by a dashed line. (c) Undated query accuracy against the median update steps in the versioned-software dataset; results are for a 110M model.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Targets of the three kinds of query in Figure 8 : a value in the queried year, the value in the batch trained at step

k

, and the final value.

Table 10

Figure 8: Query accuracy of the six 110M models over five epochs of training, the full version of Figure 2 . Rows of panels compare chronological and shuffled training, columns compare timestamp settings, and the right column queries SD and SD 1/2 by batch as in Figure 2 c. Rows 2005–2020 ask with a year (“In 2005, …”) and count any value the person held in that year; the current row asks about records of the batch the model has just trained on, in the form each was trained in. In the two panels on the left, the solid curve is this current score for the undated models and the dashed curve the probability of the final value, held in 2024.

		Overwrite						Inject
		Vocabulary top-1		Margin $\geq 3$		$P_{\mathrm{new}}\geq 0.5$		Vocabulary top-1		Margin $\geq 3$		$P_{\mathrm{new}}\geq 0.5$
Scale	Pair	steps	saved	steps	saved	steps	saved	steps	saved	steps	saved	steps	saved
110M	C / S	53.0 / 89.0	$+$ 40%	63.5 / 108	$+$ 41%	59.5 / 99.5	$+$ 40%	38.0 / 57.0	$+$ 33%	46.5 / 78.5	$+$ 41%	47.0 / 71.5	$+$ 34%
	CD / SD	51.5 / 75.0	$+$ 31%	63.0 / 88.5	$+$ 29%	57.5 / 80.5	$+$ 29%	40.0 / 52.0	$+$ 23%	53.0 / 66.5	$+$ 20%	50.0 / 62.5	$+$ 20%
	CD 1/2 / SD 1/2	54.5 / 88.0	$+$ 38%	67.0 / 103	$+$ 35%	61.0 / 96.5	$+$ 37%	31.5 / 61.5	$+$ 49%	40.5 / 84.5	$+$ 52%	39.5 / 80.0	$+$ 51%

Appendix

Table 5: Update cost of the 110M pairs under three stricter success criteria, which are the new value is the top-1 over the attribute’s whole vocabulary, its log-probability exceeds the competitor’s by 3 and its probability

P_{\mathrm{new}}

reaches 0.5. Cells represent median steps, chronological / shuffled, and the share of steps the chronological model saves (bold: paired Wilcoxon

p<0.05

). 60 overwrites and 60 injections updates per pair.

Figure 9: Mapping the example in Section 3 to the fact-level variables. The person and attribute jointly identify the fact

i

;

y=2010

is the record’s timestamp, and

v=\text{December 17, 1979}

is its value. Uppercase

I,Y,V

denote the variables and lowercase

i,y,v

their observed values. Here the attribute is time-invariant; for a changing attribute such as employer, the same

i

can have different values

v

in different years

y

Figure 10: Activation patching of the timestamp in the 110M SD (a) and CD (b). Rows are three positions of a dated question (the year, the person’s name and the answer position) and columns the twelve layers; each symbol is one activation: embedding, residual state, attention output or MLP output. A darker symbol recovers more of the model’s own year effect when that activation is copied from the same question asked about 2024; colors compare locations within one model. SD recovers the year at the answer position in the first attention block, whereas in CD the year effect builds up gradually through depth.

			Slow		Fast
	Scale	Pair	steps	saved	steps	saved
Overwrite	110M	C / S	49.0 / 91.5	$+$ 46%	39.0 / 86.0	$+$ 55%
		CD / SD	52.5 / 69.0	$+$ 24%	38.0 / 71.0	$+$ 46%
		CD 1/2 / SD 1/2	51.0 / 74.5	$+$ 32%	36.0 / 72.5	$+$ 50%
		Mix-C / Mix-S	46.5 / 70.5	$+$ 34%	45.0 / 64.5	$+$ 30%
	350M	C / S	47.0 / 118	$+$ 60%	36.0 / 118	$+$ 70%

Appendix

Table 6: Update cost by how often a fact changes: the time-varying facts of Table 2 split into slow and fast facts, with the Mix pair (Section 6.1 ) added. Each “steps” cell gives the median steps of the chronological and shuffled models. The “saved” column is the percentage of update steps that a chronological model saves over the corresponding shuffled model; bold values have a paired Wilcoxon

p<0.05

	Overwrite		Inject
Seed	steps	saved	steps	saved
0 (main)	44 / 89	$+$ 51%	29 / 57.5	$+$ 50%
1	46 / 89.5	$+$ 49%	29.5 / 61	$+$ 52%
2	48 / 89	$+$ 46%	32 / 57	$+$ 44%
3	44 / 89	$+$ 51%	28 / 55.5	$+$ 50%

Appendix

Table 7: Temporal order saves about half of the steps on time-varying facts in each of four independently trained 110M C / S pairs, each with its own population and initialization. Cells read as in Table 2 : median steps on time-varying facts, chronological / shuffled, and the share of steps temporal order saves.

		Overwrite		Inject
Varied	Setting	steps	saved	steps	saved
Optimizer	Adam	47.0 / 92.5	$+$ 49%	32.0 / 54.0	$+$ 41%
	Lion	51.0 / 108	$+$ 53%	32.5 / 58.5	$+$ 44%
	SGD	23.5 / 214	$+$ 89%	18.0 / 118	$+$ 85%
	SGD + momentum	7.5 / 33.5	$+$ 78%	6.0 / 21.5	$+$ 72%
Learning rate	Adam, $3\!\times\!10^{-5}$	18.0 / 32.5	$+$ 45%	10.0 / 20.0	$+$ 50%

Appendix

Table 8: The saving of temporal order does not depend on how the update is made. Each block changes one part of the update and keeps the rest as in the main experiments (LoRA of rank 16, Adam, learning rate

10^{-5}

). Cells read as in Table 2 : median steps, chronological / shuffled, and the share of steps temporal order saves on time-varying facts.

Figure 11: The step-law approximations along update trajectories, for time-varying overwrite updates in the 110M C (blue) and S (orange) under Adam. (a) Measured margin gain against its linear estimate along the first update; the lower plot shows their difference in log-probability, in units of $10^{-4}$ . Each model is represented by the update with the median first-step error. (b) Gradient norm over update progress $k/N$ . (c) Update speed over the same progress: dots mark the first step and dashed lines the median of per-update average speeds. In (b, c), solid curves are medians and shaded bands interquartile ranges. (d) First-step speed against its gradient-based estimate $\eta\|g_{0}\|_{1}/q_{\mathrm{old}}$ ; the diagonal is exact agreement, and faint points show later steps.

Model	ARC-Easy	PIQA	HellaSwag	CounterFact
	( Clark et al., 2018 )	( Bisk et al., 2020 )	( Zellers et al., 2019 )	( Meng et al., 2022 )
Base	85.8	81.1	52.8	89.0
C-C	84.7	81.3	52.8	88.9
C-S	84.4	80.9	52.8	89.1

Appendix

Table 9: General ability of Gemma-2-9B-it before (Base) and after the continual pre-training stage (C-C, C-S). Accuracy (%) on each benchmark and on CounterFact facts, higher is better.

			Forgetting (%)
Editor	Updating cost	saved	Other people	reduced	Own history	reduced
ROME ( Meng et al., 2022 )	5.36 / 5.65	$+$ 5%	2.1 / 4.2	$+$ 50%	81.8 / 80.2	$-$ 2%
MEMIT ( Meng et al., 2023 )	0.78 / 1.44	$+$ 46%	0.9 / 2.2	$+$ 59%	14.8 / 22.1	$+$ 33%
AlphaEdit ( Fang et al., 2025 )	0.77 / 1.31	$+$ 41%	1.2 / 3.2	$+$ 63%	16.0 / 24.7	$+$ 35%
ENCORE ( Gupta et al., 2025 )	0.75 / 1.10	$+$ 32%	0.8 / 2.1	$+$ 60%	14.5 / 18.8	$+$ 23%

Appendix

Table 10: The four editors on Mix-C and Mix-S, over the same 60 overwrite updates. Each cell is Mix-C / Mix-S, followed by the share Mix-C saves; lower is better. Updating cost:

10^{12}

FLOPs per fact written. Forgetting, other people: percentage of other people’s facts whose answer the update changes. Forgetting, own history: percentage of the updated person’s dated history that the update changes. Bold:

p<0.05

Pre-training from scratch: Mix-C / Mix-S
	No version in prompt				Version in prompt
	Overwrite		Inject		Overwrite		Inject
Fact	steps	saved	steps	saved	steps	saved	steps	saved
slow	53.0 / 102	$+$ 48%	38.5 / 64.0	$+$ 40%	55.0 / 77.0	$+$ 29%	40.5 / 57.0	$+$ 29%
fast	47.0 / 84.0	$+$ 44%	33.5 / 47.5	$+$ 29%	48.5 / 68.5	$+$ 29%	32.0 / 42.0	$+$ 24%
Continual pre-training: C-C / C-S

Appendix

Table 11: Median steps to update a field in the tool-call world, for slow and fast fields. Rows: pre-training from scratch (Mix-C / Mix-S) and continual pre-training (C-C / C-S). saved is the share of steps the chronological model saves; bold:

p<0.05

Figure 12: Reshaping the 110M SD 1/2 model. (a) Undated query accuracy against the median update steps for Base, C-S and C-C. (b) Steps per update against forgetting for single, batch and sequential updates of Base (grey), C-S (orange) and C-C (blue); lower left is better.

Explore similar work

May 21, 2026cs.CL

Understanding Data Temporality Impact on Large Language Models Pre-training

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.

Hippolyte Pilchen, Romain Fabre, Franck Signe Talla +2

Kyutai, Paris

Language Model Pretraining Temporal Reasoning in Language Models Factual Knowledge in Language Models Temporal QA

Ask PDF

Aug 18, 2026cs.AI

When to Review: Spaced Repetition for Continual Pre-Training of Language Models

Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.

Alankar Atreya, Devesh Batra, Yoages Kumar Mantri +3

NatWest AI Research · University College London

Continual Learning for LLMs Continued Pretraining Replay-Based Continual Learning Catastrophic Forgetting

Ask PDF

Apr 25, 2026cs.LG

Evolve: A Persistent Knowledge Lifecycle for Small Language Models

Evolve pairs a small local language model with a persistent, teacher-compiled knowledge store -- refined through sleep consolidation and usage-driven refresh -- to deliver substantial accuracy gains over the model's parametric baseline while amortizing teacher costs through cross-query knowledge reuse. Rather than retrieving document fragments at query time, Evolve constructs a store of semantically coherent sections compiled by teacher models at natural conceptual boundaries; new sections are staged on acquisition, consolidated offline through teacher-mediated merging, and refreshed inline when expired. A 2B-parameter local model handles classification and generation; large teacher models are invoked only for knowledge operations. Across 750 benchmark queries spanning custom specialist questions, NaturalQuestions, and TriviaQA, the 2B model augmented by Evolve improves from 20-33% baseline accuracy to 60-84% (+40-52pp) while reducing teacher invocations by over 50% through reuse. Post-consolidation compresses the knowledge store by 31-33.5% across three independent benchmarks while preserving accuracy; section-based retrieval outperforms chunk-based retrieval by 5-9pp across every lifecycle condition. The architecture supports two generation modes over the same lifecycle -- suppress (strict section-only grounding, auditable) and augment (section-supplemented responses).

Dikran Hovagimian

Independent Researcher

Retrieval-Augmented Generation Memory-Augmented Language Models Knowledge Augmentation for Language Models Persistent Memory for Language Models Small Language Models Knowledge-Intensive QA

Ask PDF