Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.
Figures & tables
Figure 1: Parameter-agnostic CPT erodes high-Fisher directions. Diagonal Fisher entries after CPT. Left: Parameter-agnostic CPT produces substantial negative drift on high-Fisher coordinates (orange regions), indicating damage to directions where the pretrained model had strong commitments. Right: Our Fisher-guided selector mitigates this drift while concentrating acquisition (blue regions) in low-Fisher coordinates where the pretrained model is less committed.
Figure 2: Practical data selection pipeline. Before the stream is processed, a short warmup pass on target-domain documents produces the LoRA operating point θ⋆ , and a diagonal Fisher Λ is estimated from a small proxy set Dproxy . For each streamed candidate, we compute its LoRA gradient g∈RD at θ⋆ , form the Fisher-weighted anchor and frontier gradients ganc=Λ⊙g and gfr=g⊘Λε , project both to dimension d≪D , and evaluate the marginal gain Δ=Δanc+αΔfr under eq. equation 3 . Sieve-Streaming tests this gain against a geometric grid of thresholds {τi} to accept or discard the candidate, and returns the highest-utility filled sieve as S⋆ .
Variant
A↑
Φ↓
A−Φ↑
Med-PPL ↓
PubMedQA ↑
Ours (full, α=1 )
+5.5
0.4
+5.1
6.2
70.0
No ganc (drop anchor term)
+4.2
5.0
− 0.8
6.5
68.0
No gfr (drop frontier term)
+0.8
0.5
+0.3
8.5
65.5
Identity Fisher ( Λ=I )
+1.5
2.5
− 1.0
7.3
65.5
Set-level on raw g (no Fisher reweighting)
+2.0
2.0
+ 0.0
7.2
66.0
Table 2: Ablation on the Fisher decomposition. All variants share the streaming submodular pipeline, the random projection, the Adam preconditioner, the Fisher floor, and the perplexity gate.
Selection rule
A↑
Φ↓
cos↓
\logdetG~↑
wall-clock (ms)
Top- B by ∥ganc∥
+1.8
0.6
0.45
0 8.5
00 0.3
Top- B by ∥gfr∥
+3.5
4.0
0.22
11.5
00 0.3
Top- B by δ0(x)
+3.0
2.2
0.30
10.0
00 0.4
Sieve-Streaming f(S) (Ours)
+5.5
0.4
0.12
18.5
00 1.2
Table 3: Subset-level \logdet vs. per-sample top- k . Diversity columns measure non-redundancy of the chosen subset on ganc : lower mean pairwise cosine and higher Gram \logdet are better.
Token budget
1B
B=2B tok
B=4B tok
B=10B tok
Method
A↑
Φ↓
A↑
Φ↓
A↑
Φ↓
A↑
Φ↓
Replay
− 3.5
1.2
− 2.9
1.0
− 2.6
0.9
− 2.0
0.8
Ours
+3.0
0.6
+4.5
0.5
+5.5
0.4
-
-
Table 4: Token efficiency comparison. We compare the performance between our method and Replay on TinyLlama-1.1B at four selection budgets. Ours at 1B tokens already exceeds Replay at 10B tokens on both A and Φ .
Figure 3: Replay trade-off and training dynamics.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Filter
Rule
Docs removed
% of pool
1. Source-ID
Drop every PMC/PubMed record whose PMID or DOI is in the source lists of PubMedQA (PQA-L/U/A, 273.5 k PMIDs) or BioASQ
298,142
0.99
2. n -gram overlap
Drop any candidate sharing a 13 -gram with any evaluation instance (question + context + answer)
412,905
1.37
3. Near-duplicates
MinHash LSH; drop pairs with Jaccard similarity ≥0.8 on 5 -gram shingles
96,318
0.32
Total
807,365
2.68
Appendix
Table 5: Decontamination filters applied to the PMC/PubMed candidate pool. The three counts are disjoint and add up to the total; percentages are relative to the documents of the unfiltered pool.
Method
Stage
Med-PPL ↓
PubMedQA ↑
BioASQ ↑
A↑
Φ↓
DSIR
Pretraining
7.2
66.9
75.4
+2.4
2.9
DoReMi ‡
Pretraining
7.5
65.8
74.1
+1.2
1.6
FisherSFT †
SFT
6.9
67.4
76.1
+3.0
3.3
SPICE †
SFT
6.7
67.9
76.7
+3.6
2.6
Ours
CPT
6.2
70.0
78.5
+5.5
0.4
Appendix
Table 6: Comparison with data-selection methods (TinyLlama-1.1B, 4 B tokens). All methods share the candidate pool, LoRA configuration, CPT recipe, and evaluation; only the selection rule differs. † SFT-stage methods ported to CPT. ‡ Domain-level method whose domains are the four corpus sources (App. C ). Bold / underline : best/second-best within this table.
Stage
EFLOP
GPU-h
Share
(i) Warmup adapter θ⋆
01.30
5.9
3.2%
(ii) Diagonal Fisher Λ
01.27
24.7
13.4%
(iii) Perplexity gate
03.06
13.7
7.4%
(iv) Per-sample LoRA gradients
03.36
26.7
14.5%
(v) TRAK projection
01.88
4.4
2.4%
(vi) Sieve-Streaming selection
<0.01
1.1
0.6%
Appendix
Table 7: End-to-end compute of our pipeline on TinyLlama-1.1B at the default 4 B-token budget, measured on 4 nodes with 4× A100-40G GPUs each. GPU-hours are summed over all GPUs; the share is the fraction of total GPU-hours.
Method
CPT tokens
EFLOP
GPU-h
FLOP ratio
GPU-h ratio
A↑
Φ↓
Replay
10 B
62.1
271.4
1.0×
1.0×
− 2.0
0.8
Ours
4 B
35.5
184.0
1.7×
1.5×
+5.5
0.4
Ours
1 B
10.9
69.3
5.7×
3.9×
+3.0
0.6
Appendix
Table 8: End-to-end cost against Replay (TinyLlama-1.1B). Totals include every selection stage of Table 7 ; Replay performs no selection. The ratio columns give Replay- 10 B’s cost divided by the method’s cost. A and Φ are from Tables 1 and 4 .
εΛ
Coords clipped
Max ∥gfr∥ inflation
A↑
Φ↓
10−8 (no effective floor)
0.0%
3.1×104×
+3.0
4.5
p5(Λ)
5%
210×
+4.1
1.9
p25(Λ)
25%
21×
+5.3
0.5
p50(Λ) = median ⋆
50%
6.4×
+5.5
0.4
p75(Λ)
75%
2.3×
+5.2
0.4
p100 (constant Λ ; frontier on g^ )
100%
1.0×
+3.4
1.2
Appendix
Table 9: Fisher floor εΛ (TinyLlama-1.1B, 4 B tokens). pq(Λ) is the q -th percentile of the Fisher entries; “coords clipped” is the fraction of coordinates raised to the floor; the inflation column reports the largest inflation of ∥gfr∥ relative to the constant- Λ row. ⋆ marks the default.
Sweep
Value
A↑
Φ↓
Frontier weight α
0.5
+4.8
0.7
1.0 ⋆
+5.5
0.4
2.0
+5.4
0.5
4.0
+4.6
0.4
Per-term scale β=γ
5
+3.8
1.0
10
+4.9
0.6
Appendix
Table 10: Hyperparameter sensitivity on TinyLlama-1.1B under the 4B-token default setting. Each block sweeps one hyperparameter holding the rest at their defaults (marked ⋆ ). Last column reports the best TinyLlama baseline from Table 1 under the same A−Φ metric.
ε
∣T∣
f(Sours)/f(Sgreedy)
per-candidate wall-clock (ms)
0.05
94
99.1%
7.8
0.1
48
98.5%
4.0
0.2
25
96.8%
2.1
0.4
14
94.6%
1.2
Offline greedy (diagnostic reference)
100%
720
Appendix
Table 11: Small-stream Sieve-Streaming diagnostic on cached projected gradients ( ∣Dstream∣=104 , B=1,000 , TinyLlama-1.1B). Offline greedy is used only to normalize the projected objective in this tractable regime; no downstream CPT is run for this diagnostic. ∣T∣ is the number of sieve thresholds.
Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as continual Factual Knowledge Acquisition (cFKA), remain unclear. In this work, we present a theoretical framework that characterizes the training dynamics of cFKA using a single-layer Transformer, offering a unified explanation for the behavior of representative CPT methods. Our analysis reveals that regularization-based methods merely adjust the convergence rate of parameters without altering the inherent forgetting tendency, whereas data replay methods succeed in shifting convergence dynamics and stabilizing pretrained knowledge. Building on these insights, we propose a novel generative data replay approach, called Selecting Tokens via attentiOn Contribution~(STOC), which identifies influential factual snippets to guide replay data generation. Extensive experiments on both synthetic and real-world datasets validate our findings and demonstrate that STOC effectively enhances cFKA by mitigating catastrophic forgetting.
Haoyu Wang, Yifan Shang, Zhongxiang Sun +3
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China
Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15-35 times lower search cost. Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.
Haiyue Song, Masao Utiyama
National Institute of Information and Communications Technology, Kyoto, Japan
Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible original data. In this data-free regime, we analyze where forgetting occurs and why. Systematic parameter freezing across five settings up to 1.4B reveals that forgetting concentrates selectively in the output embeddings of tokens rarely seen in the new corpus, whereas the same sqrt(v-hat) band of the body is inert and new learning resides elsewhere. This localization is governed by the vocabulary deficiency of the corpus rather than the training mode, allowing pre-retraining risk ranking from token counts alone within a fixed base model. Mechanistically, absent tokens receive persistent one-sided softmax gradients that Adam's second-moment (sqrt(v-hat)) normalization amplifies into full-sized updates. We therefore propose an intervention: raising Adam's epsilon exclusively for the output projection during training. Across eight settings spanning 160M to 12B parameters and four model families, this removes 39.4% to 67.9% of forgetting across all seven stable configurations without degrading target learning or requiring per-model tuning. The defense combines additively or better with replay (79.8% on Qwen/Korean) and rescues released-head LoRA from a 23-fold forgetting surge. Because post-hoc editing of the drifted rows recovers under 5% of forgetting, the intervention must operate during training. Our findings indicate that a single-line optimizer adjustment may serve as the primary defense against catastrophic forgetting where the corpus starves the vocabulary.
Jonghyun Han, Younghoon Song, Jongyoul Park
Seoul National University of Science and Technology · Korea Institute of Land & Infrastructure Safety Technology