Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Zhenghao Zhao; Gaowen Liu; Zhiling Lan; Yan Yan

cs.LGOct 1, 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Authors: Zhenghao Zhao, Gaowen Liu, Zhiling Lan, Yan Yan

Organizations: University of Illinois Chicago · Cisco Research

Abstract

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.

Figures & tables

Figure 1: Parameter-agnostic CPT erodes high-Fisher directions. Diagonal Fisher entries after CPT. Left: Parameter-agnostic CPT produces substantial negative drift on high-Fisher coordinates (orange regions), indicating damage to directions where the pretrained model had strong commitments. Right: Our Fisher-guided selector mitigates this drift while concentrating acquisition (blue regions) in low-Fisher coordinates where the pretrained model is less committed.

Figure 2: Practical data selection pipeline. Before the stream is processed, a short warmup pass on target-domain documents produces the LoRA operating point $\theta^{\star}$ , and a diagonal Fisher $\Lambda$ is estimated from a small proxy set $\mathcal{D}_{\mathrm{proxy}}$ . For each streamed candidate, we compute its LoRA gradient $g\in\mathbb{R}^{D}$ at $\theta^{\star}$ , form the Fisher-weighted anchor and frontier gradients $g_{\mathrm{anc}}=\sqrt{\Lambda}\odot g$ and $g_{\mathrm{fr}}=g\oslash\sqrt{\Lambda_{\varepsilon}}$ , project both to dimension $d\ll D$ , and evaluate the marginal gain $\Delta=\Delta_{\mathrm{anc}}+\alpha\,\Delta_{\mathrm{fr}}$ under eq. equation 3 . Sieve-Streaming tests this gain against a geometric grid of thresholds $\{\tau_{i}\}$ to accept or discard the candidate, and returns the highest-utility filled sieve as $S^{\star}$ .

Variant	$A\uparrow$	$\Phi\downarrow$	$A-\Phi\uparrow$	Med-PPL $\downarrow$	PubMedQA $\uparrow$
Ours (full, $\alpha=1$ )	+5.5	0.4	+5.1	6.2	70.0
No $g_{\mathrm{anc}}$ (drop anchor term)	+4.2	5.0	$-$ 0.8	6.5	68.0
No $g_{\mathrm{fr}}$ (drop frontier term)	+0.8	0.5	+0.3	8.5	65.5
Identity Fisher ( $\Lambda=I$ )	+1.5	2.5	$-$ 1.0	7.3	65.5
Set-level on raw $g$ (no Fisher reweighting)	+2.0	2.0	$+$ 0.0	7.2	66.0

Table 2: Ablation on the Fisher decomposition. All variants share the streaming submodular pipeline, the random projection, the Adam preconditioner, the Fisher floor, and the perplexity gate.

Selection rule	$A\uparrow$	$\Phi\downarrow$	$\overline{\cos}\downarrow$	$\logdet\tilde{G}\uparrow$	wall-clock (ms)
Top- $B$ by $\\|g_{\mathrm{anc}}\\|$	+1.8	0.6	0.45	0 8.5	00 0.3
Top- $B$ by $\\|g_{\mathrm{fr}}\\|$	+3.5	4.0	0.22	11.5	00 0.3
Top- $B$ by $\delta_{0}(x)$	+3.0	2.2	0.30	10.0	00 0.4
Sieve-Streaming $f(S)$ (Ours)	+5.5	0.4	0.12	18.5	00 1.2

Table 3: Subset-level

\logdet

vs. per-sample top-

k

. Diversity columns measure non-redundancy of the chosen subset on

g_{\mathrm{anc}}

: lower mean pairwise cosine and higher Gram

\logdet

are better.

Token budget	$1\mathrm{B}$		$B{=}2\mathrm{B}$ tok		$B{=}4\mathrm{B}$ tok		$B{=}10\mathrm{B}$ tok
Method	$A\uparrow$	$\Phi\downarrow$	$A\uparrow$	$\Phi\downarrow$	$A\uparrow$	$\Phi\downarrow$	$A\uparrow$	$\Phi\downarrow$
Replay	$-$ 3.5	1.2	$-$ 2.9	1.0	$-$ 2.6	0.9	$-$ 2.0	0.8
Ours	+3.0	0.6	+4.5	0.5	+5.5	0.4	-	-

Table 4: Token efficiency comparison. We compare the performance between our method and Replay on TinyLlama-1.1B at four selection budgets. Ours at 1B tokens already exceeds Replay at 10B tokens on both

A

and

\Phi

Panel of Figure 3 — Figure 3: Replay trade-off and training dynamics.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Filter	Rule	Docs removed	% of pool
1. Source-ID	Drop every PMC/PubMed record whose PMID or DOI is in the source lists of PubMedQA (PQA-L/U/A, $273.5$ k PMIDs) or BioASQ	$298{,}142$	$0.99$
2. $n$ -gram overlap	Drop any candidate sharing a $13$ -gram with any evaluation instance (question + context + answer)	$412{,}905$	$1.37$
3. Near-duplicates	MinHash LSH; drop pairs with Jaccard similarity $\geq 0.8$ on $5$ -gram shingles	$96{,}318$	$0.32$
Total		$\mathbf{807{,}365}$	$\mathbf{2.68}$

Appendix

Table 5: Decontamination filters applied to the PMC/PubMed candidate pool. The three counts are disjoint and add up to the total; percentages are relative to the documents of the unfiltered pool.

Method	Stage	Med-PPL $\downarrow$	PubMedQA $\uparrow$	BioASQ $\uparrow$	$A\uparrow$	$\Phi\downarrow$
DSIR	Pretraining	7.2	66.9	75.4	+2.4	2.9
DoReMi ‡	Pretraining	7.5	65.8	74.1	+1.2	1.6
FisherSFT †	SFT	6.9	67.4	76.1	+3.0	3.3
SPICE †	SFT	6.7	67.9	76.7	+3.6	2.6
Ours	CPT	6.2	70.0	78.5	+5.5	0.4

Appendix

Table 6: Comparison with data-selection methods (TinyLlama-1.1B,

4

B tokens). All methods share the candidate pool, LoRA configuration, CPT recipe, and evaluation; only the selection rule differs. † SFT-stage methods ported to CPT. ‡ Domain-level method whose domains are the four corpus sources (App. C ). Bold / underline : best/second-best within this table.

Stage	EFLOP	GPU-h	Share
(i) Warmup adapter $\theta^{\star}$	$\phantom{0}1.30$	$5.9$	$3.2\%$
(ii) Diagonal Fisher $\Lambda$	$\phantom{0}1.27$	$24.7$	$13.4\%$
(iii) Perplexity gate	$\phantom{0}3.06$	$13.7$	$7.4\%$
(iv) Per-sample LoRA gradients	$\phantom{0}3.36$	$26.7$	$14.5\%$
(v) TRAK projection	$\phantom{0}1.88$	$4.4$	$2.4\%$
(vi) Sieve-Streaming selection	$<0.01$	$1.1$	$0.6\%$

Appendix

Table 7: End-to-end compute of our pipeline on TinyLlama-1.1B at the default

4

B-token budget, measured on

4

nodes with

4\times

A100-40G GPUs each. GPU-hours are summed over all GPUs; the share is the fraction of total GPU-hours.

Method	CPT tokens	EFLOP	GPU-h	FLOP ratio	GPU-h ratio	$A\uparrow$	$\Phi\downarrow$
Replay	$10$ B	$62.1$	$271.4$	$1.0\times$	$1.0\times$	$-$ 2.0	0.8
Ours	$4$ B	$35.5$	$184.0$	$1.7\times$	$1.5\times$	+5.5	0.4
Ours	$1$ B	$10.9$	$69.3$	$5.7\times$	$3.9\times$	+3.0	0.6

Appendix

Table 8: End-to-end cost against Replay (TinyLlama-1.1B). Totals include every selection stage of Table 7 ; Replay performs no selection. The ratio columns give Replay-

10

B’s cost divided by the method’s cost.

A

and

\Phi

are from Tables 1 and 4 .

$\varepsilon_{\Lambda}$	Coords clipped	Max $\\|g_{\mathrm{fr}}\\|$ inflation	$A\uparrow$	$\Phi\downarrow$
$10^{-8}$ (no effective floor)	$0.0\%$	$3.1\times 10^{4}\times$	+3.0	4.5
$p_{5}(\Lambda)$	$5\%$	$210\times$	+4.1	1.9
$p_{25}(\Lambda)$	$25\%$	$21\times$	+5.3	0.5
$p_{50}(\Lambda)$ = median ⋆	$50\%$	$6.4\times$	+5.5	0.4
$p_{75}(\Lambda)$	$75\%$	$2.3\times$	+5.2	0.4
$p_{100}$ (constant $\Lambda$ ; frontier on $\hat{g}$ )	$100\%$	$1.0\times$	+3.4	1.2

Appendix

Table 9: Fisher floor

\varepsilon_{\Lambda}

(TinyLlama-1.1B,

4

B tokens).

p_{q}(\Lambda)

is the

q

-th percentile of the Fisher entries; “coords clipped” is the fraction of coordinates raised to the floor; the inflation column reports the largest inflation of

\|g_{\mathrm{fr}}\|

relative to the constant-

\Lambda

row. ⋆ marks the default.

Sweep	Value	$A\uparrow$	$\Phi\downarrow$
Frontier weight $\alpha$	0.5	+4.8	0.7
	1.0 ⋆	+5.5	0.4
	2.0	+5.4	0.5
	4.0	+4.6	0.4
Per-term scale $\beta=\gamma$	5	+3.8	1.0
	10	+4.9	0.6

Appendix

Table 10: Hyperparameter sensitivity on TinyLlama-1.1B under the 4B-token default setting. Each block sweeps one hyperparameter holding the rest at their defaults (marked ⋆ ). Last column reports the best TinyLlama baseline from Table 1 under the same

A-\Phi

metric.

$\varepsilon$	$\|\mathcal{T}\|$	$f(S_{\text{ours}})/f(S_{\text{greedy}})$	per-candidate wall-clock (ms)
0.05	94	99.1%	7.8
0.1	48	98.5%	4.0
0.2	25	96.8%	2.1
0.4	14	94.6%	1.2
Offline greedy (diagnostic reference)		100%	720

Appendix

Table 11: Small-stream Sieve-Streaming diagnostic on cached projected gradients (

|\mathcal{D}_{\mathrm{stream}}|=10^{4}

B=1{,}000

, TinyLlama-1.1B). Offline greedy is used only to normalize the projected objective in this tractable regime; no downstream CPT is run for this diagnostic.

|\mathcal{T}|

is the number of sieve thresholds.

Continued Pretraining

Continued pretraining adapts pretrained models to new domains, languages, modalities, or contexts by training further on targeted data.

See all 56 papers

Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
May 11, 2026Haoyu Wang, Yifan Shang, Zhongxiang Sun +3Continual Learning for LLMs Generative Replay
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China
OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
Mar 30, 2026Haiyue Song, Masao UtiyamaContinued Pretraining Data Mixture Optimization
National Institute of Information and Communications Technology, Kyoto, Japan
A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
Oct 7, 2026Jonghyun Han, Younghoon Song, Jongyoul ParkCatastrophic Forgetting Continual Learning for LLMs
Seoul National University of Science and Technology · Korea Institute of Land & Infrastructure Safety Technology

May 11, 2026J/K move · Enter open · S save

Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

Haoyu Wang, Yifan Shang, Zhongxiang Sun, Weijie Yu+2

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China

Continual Learning for LLMs Generative Replay Continual Learning Replay-Based Continual Learning Continued Pretraining Catastrophic Forgetting

Open paper AskPDF arXiv

Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as continual Factual Knowledge Acquisition (cFKA), remain unclear. In this work, we present a theoretical framework that characterizes the training dynamics of cFKA using a single-layer Transformer, offering a unified explanation for the behavior of representative CPT methods. Our analysis reveals that regularization-based methods merely adjust the convergence rate of parameters without altering the inherent forgetting tendency, whereas data replay methods succeed in shifting convergence dynamics and stabilizing pretrained knowledge. Building on these insights, we propose a novel generative data replay approach, called Selecting Tokens via attentiOn Contribution~(STOC), which identifies influential factual snippets to guide replay data generation. Extensive experiments on both synthetic and real-world datasets validate our findings and demonstrate that STOC effectively enhances cFKA by mitigating catastrophic forgetting.

Abstract

Continued Pretraining

Explore similar work

Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks