NADS: Navigator-Guided Data Selection for Mitigating Catastrophic Forgetting in Fine-Tuning

Jianhao Zhang; Ou Wu; Yi Du

cs.LGOct 19, 2026

NADS: Navigator-Guided Data Selection for Mitigating Catastrophic Forgetting in Fine-Tuning

Authors: Jianhao Zhang, Ou Wu, Yi Du

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China

Abstract

Catastrophic forgetting is a central obstacle in single-stage fine-tuning of large language models (LLMs) on a new task, especially when the original pretraining data are unavailable. Synthetic data offers a practical mitigation path, but candidate pools are often noisy and redundant, making effective selection essential. Existing methods often rely on indirect proxy signals whose alignment with actual training benefit can be limited, while utility--diversity selection over large candidate pools remains computationally demanding. To address these challenges, we propose NAvigator-guided Data Selection (NADS), a fine-tuning framework for LLMs. NADS first follows the target fine-tuning recipe without preservation regularization to obtain a navigator model, whose deviation from the pretrained model exposes recipe-induced capability drift. It then uses the predictive divergence between the navigator and the pretrained model to score forgetting-aware utility for each synthetic candidate, and constructs a constraint set through an efficient utility--diversity selection strategy. Finally, NADS fine-tunes on the new-task data while distilling pretrained behavior on the selected constraints to preserve general capabilities. Experiments show that NADS delivers a stronger balance between new-task performance and general capability preservation, while reducing utility--diversity selection overhead. Code at: https://github.com/zjh-vinky/NADS.

Figures & tables

Figure 1: The overall framework of NADS. First, we train a navigator $M_{\text{nav}}$ from $M_{\text{pretrain}}$ on $\mathcal{D}_{\text{new}}$ without regularization. Next, we score sample utility from the predictive divergence between $M_{\text{nav}}$ and $M_{\text{pretrain}}$ , and construct $\mathcal{D}_{\text{constraint}}$ via utility–diversity selection with DPPs. Finally, we train $M_{\text{final}}$ on $\mathcal{D}_{\text{new}}$ with distillation on $\mathcal{D}_{\text{constraint}}$ to preserve general capabilities during new-task adaptation.

Figure 2: (a–b) Utility versus response-only NLL degradation and full-sequence loss increase, with Pearson and Spearman measuring sample correlation and ranking consistency, respectively. (c–d) Mean response-only NLL degradation and full-sequence loss increase across utility deciles. Higher-utility samples exhibit larger increases, indicating strong correlation with samples’ forgetting risk.

Model	Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Model	Method	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
	Pre-trained	$\text{55.83}_{\pm\text{0.00}}$	$\text{31.20}_{\pm\text{0.00}}$	$\text{56.92}_{\pm\text{0.00}}$	$\text{27.90}_{\pm\text{0.00}}$	$\text{42.96}_{\pm\text{0.00}}$	$\text{28.16}_{\pm\text{0.00}}$	$\text{35.56}_{\pm\text{0.00}}$
	Standard FT	$\text{52.12}_{\pm\text{0.52}}$	$\text{28.40}_{\pm\text{0.69}}$	$\text{48.61}_{\pm\text{0.40}}$	$\text{20.75}_{\pm\text{0.74}}$	$\text{37.47}_{\pm\text{0.30}}$	$\text{45.13}_{\pm\text{0.32}}$	$\text{41.30}_{\pm\text{0.22}}$
	LoRA	$\text{53.38}_{\pm\text{0.55}}$	$\text{29.47}_{\pm\text{0.64}}$	$\text{51.98}_{\pm\text{0.42}}$	$\text{21.18}_{\pm\text{0.79}}$	$\text{39.00}_{\pm\text{0.31}}$	$\text{41.92}_{\pm\text{0.30}}$	$\text{40.46}_{\pm\text{0.21}}$
	Random	$\text{52.03}_{\pm\text{0.54}}$	$\text{28.13}_{\pm\text{0.64}}$	$\text{49.21}_{\pm\text{0.43}}$	$\text{21.83}_{\pm\text{0.73}}$	$\text{37.80}_{\pm\text{0.30}}$	$\text{41.77}_{\pm\text{0.46}}$	$\text{39.79}_{\pm\text{0.27}}$
	InsCL	$\text{53.95}_{\pm\text{0.54}}$	$\text{29.27}_{\pm\text{0.70}}$	${\color[rgb]{0,0,1}\text{54.86}_{\pm\text{0.43}}}$	$\text{25.40}_{\pm\text{0.76}}$	$\text{40.87}_{\pm\text{0.31}}$	$\text{42.02}_{\pm\text{0.82}}$	$\text{41.45}_{\pm\text{0.44}}$

Table 1: Performance comparison with methods selecting

\mathcal{D}_{\text{constraint}}

for replay on Magicoder (%). The best results are highlighted in red , and the second-best results are highlighted in blue .

Model	Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Model	Method	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
	LESS	${\color[rgb]{0,0,1}\text{54.12}_{\pm\text{0.49}}}$	${\color[rgb]{0,0,1}\text{30.67}_{\pm\text{0.61}}}$	${\color[rgb]{0,0,1}\text{54.38}_{\pm\text{0.40}}}$	$\text{23.73}_{\pm\text{0.62}}$	${\color[rgb]{0,0,1}\text{40.73}_{\pm\text{0.27}}}$	${\color[rgb]{0,0,1}\text{43.01}_{\pm\text{0.84}}}$	${\color[rgb]{0,0,1}\text{41.87}_{\pm\text{0.44}}}$
	FisherSFT	$\text{54.03}_{\pm\text{0.49}}$	$\text{29.80}_{\pm\text{0.60}}$	$\text{52.89}_{\pm\text{0.41}}$	$\text{23.15}_{\pm\text{0.62}}$	$\text{39.97}_{\pm\text{0.27}}$	$\text{41.85}_{\pm\text{0.74}}$	$\text{40.91}_{\pm\text{0.39}}$
	SPICE	$\text{53.58}_{\pm\text{0.50}}$	$\text{29.60}_{\pm\text{0.60}}$	$\text{53.39}_{\pm\text{0.40}}$	${\color[rgb]{0,0,1}\text{24.92}_{\pm\text{0.73}}}$	$\text{40.37}_{\pm\text{0.29}}$	$\text{42.61}_{\pm\text{0.66}}$	$\text{41.49}_{\pm\text{0.36}}$
LLaMA3.2	NADS (Ours)	${\color[rgb]{1,0,0}\text{55.17}_{\pm\text{0.49}}}$	${\color[rgb]{1,0,0}\text{31.07}_{\pm\text{0.58}}}$	${\color[rgb]{1,0,0}\text{56.45}_{\pm\text{0.41}}}$	${\color[rgb]{1,0,0}\text{27.37}_{\pm\text{0.67}}}$	${\color[rgb]{1,0,0}\text{42.52}_{\pm\text{0.27}}}$	${\color[rgb]{1,0,0}\text{44.39}_{\pm\text{0.40}}}$	${\color[rgb]{1,0,0}\text{43.46}_{\pm\text{0.24}}}$
	LESS	$\text{55.56}_{\pm\text{0.52}}$	$\text{31.80}_{\pm\text{0.69}}$	$\text{57.26}_{\pm\text{0.39}}$	$\text{35.08}_{\pm\text{0.83}}$	$\text{44.93}_{\pm\text{0.31}}$	${\color[rgb]{0,0,1}\text{57.07}_{\pm\text{0.91}}}$	${\color[rgb]{0,0,1}\text{51.00}_{\pm\text{0.48}}}$

Table 2: Cross-paradigm reference comparison with methods selecting

\mathcal{D}_{\text{new}}

on Magicoder (%).

Figure 3: (a) Training dynamics of the navigator model on the Magicoder dataset. (b) Dynamics of individual general-capability benchmarks during navigator training. (c) NADS performance with navigators selected at different training stages on Magicoder with LLaMA3.2-3B.

Figure 4: Controlled comparison with a fixed KD objective on Magicoder with LLaMA3.2.

Model	HASW	OBQA	MMLU	GSM8K	HE
$M_{\text{pretrain}}$	55.83	31.20	56.92	27.90	28.16
$M_{\text{nav}}$	52.80	29.20	50.60	22.09	45.50

Table 3: Performance comparison between

M_{\text{pretrain}}

and

M_{\text{nav}}

Method	General	Target	Overall
Random	37.96	41.91	39.94
PPL-based	38.76	42.02	40.39
DPP-based	38.04	41.27	39.66
NADS	42.52	44.39	43.46

Table 4: Controlled constraint-selection comparison on Magicoder.

Figure 5: (a) Quantitative diversity diagnostics of selected data. (b) t-SNE visualizations of selected data. (c) Data-selection cost in terms of selection time and peak GPU memory usage.

Method	General	Target	Overall
NADS	42.52	44.39	43.46
- w/o KL drift	37.64	42.31	39.98
- w/o Diversity	40.21	42.83	41.52
- w/o Entropy Gate	41.26	43.17	42.22

Table 5: Ablation of NADS components.

Figure 6: (a) Performance comparison under different utility signals. (b) Hyperparameter sensitivity of NADS. (c) Comparison with catastrophic forgetting mitigation methods on the Pareto frontier.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Trajectory mismatch and ordering fidelity under different KD strengths on Magicoder with LLaMA3.2-3B. The dashed line marks the default setting

\lambda=5

used in the main experiments. Increasing

\lambda

enlarges the predictive-behavior mismatch between the unconstrained navigator and the constrained final model, while the navigator preserves high ordering fidelity in the practical regime.

Checkpoint pair	Spearman	Top- $K$ overlap
$M_{\text{nav}}-1$ vs. $M_{\text{nav}}-2$	0.67	0.56
$M_{\text{nav}}-2$ vs. $M_{\text{nav}}-3$	0.72	0.61
$M_{\text{nav}}-1$ vs. $M_{\text{nav}}-3$	0.47	0.39

Appendix

Table 6: Utility ranking stability across navigator checkpoints.

Constraint source	General Avg.	Target	Overall
Low-utility	37.82	41.78	39.80
Middle-utility	39.38	42.52	40.95
High-utility	41.78	43.81	42.80
Full NADS	42.52	44.39	43.46

Appendix

Table 7: Utility-range analysis under the same final KD objective on Magicoder with LLaMA3.2-3B.

Figure 8: Scalable selection surrogate consistency on Magicoder with LLaMA3.2. (a) reports the off-diagonal ratio and relative diagonal approximation gap. (b) reports Spearman rank correlation between the full log-det diversity term and its diagonal surrogate over same-size candidate subsets.

Figure 9: Split-spectrum certificate for the diagonal log-determinant bound on the final selected constraint set. The stacked bar gives

B_{\text{split}}=B_{+}+B_{-}=1206.65

. The dashed red line marks the observed gap

\Delta_{\text{diag}}=64.45

. The certificate ratio is

\Delta_{\text{diag}}/B_{\text{split}}=0.053

, and the observed relative approximation gap is

2.40\%

Figure 10: Approximation consistency under the original discrete objective. Each point corresponds to one random trial under one greedy comparison. The x-axis denotes the objective value of the subset selected by Fast Greedy or Lazy-Fast Greedy, and the y-axis denotes that of NADS. Points near the diagonal indicate that the scalable NADS procedure remains close to greedy optimization when all subsets are evaluated under the original utility–diversity objective.

Datasets	Aspect	Nums
Magicoder [ 80 ]	Code Generation	111K
MetaMathQA [ 81 ]	Math Reasoning	395K
Magpie-Qwen2.5-Pro-300K-Filtered [ 82 ]	\	300K

Appendix

Table 8: Summary of training datasets used in our experiments.

Datasets	Aspects	Metrics	Nums	Evaluate
HellaSwag [ 87 ]	Commonsense Reasoning	Accuracy	10,042	0-shot
OpenBookQA [ 89 ]	Commonsense Reasoning	Accuracy	500	0-shot
MMLU [ 86 ]	Multi Task Understanding	Accuracy	14,042	5-shot
GSM8K [ 88 ]	Math Reasoning	Exact Match	1,319	5-shot
HumanEval [ 85 ]	Code Generation	PASS@1	164	0-shot

Appendix

Table 9: Evaluation benchmarks and detailed settings.

Model	Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Model	Method	HASW	OBQA	MMLU	HE	Avg.	GSM8K	Avg.
	Pre-trained	$\text{55.83}_{\pm\text{0.00}}$	$\text{31.20}_{\pm\text{0.00}}$	$\text{56.92}_{\pm\text{0.00}}$	$\text{28.16}_{\pm\text{0.00}}$	$\text{43.03}_{\pm\text{0.00}}$	$\text{27.90}_{\pm\text{0.00}}$	$\text{35.47}_{\pm\text{0.00}}$
	Standard FT	$\text{50.02}_{\pm\text{0.50}}$	$\text{28.40}_{\pm\text{0.53}}$	$\text{48.49}_{\pm\text{0.41}}$	$\text{15.27}_{\pm\text{0.49}}$	$\text{35.55}_{\pm\text{0.24}}$	$\text{65.93}_{\pm\text{0.81}}$	$\text{50.74}_{\pm\text{0.42}}$
	LoRA	$\text{52.06}_{\pm\text{0.50}}$	$\text{28.60}_{\pm\text{0.53}}$	$\text{52.34}_{\pm\text{0.40}}$	$\text{25.31}_{\pm\text{0.52}}$	$\text{39.58}_{\pm\text{0.25}}$	$\text{61.03}_{\pm\text{0.83}}$	$\text{50.31}_{\pm\text{0.43}}$
	Random	$\text{50.38}_{\pm\text{0.50}}$	$\text{28.47}_{\pm\text{0.58}}$	$\text{49.54}_{\pm\text{0.40}}$	$\text{17.35}_{\pm\text{0.46}}$	$\text{36.44}_{\pm\text{0.24}}$	$\text{62.70}_{\pm\text{0.79}}$	$\text{49.57}_{\pm\text{0.41}}$
	InsCL	$\text{52.85}_{\pm\text{0.50}}$	$\text{28.93}_{\pm\text{0.58}}$	$\text{55.17}_{\pm\text{0.40}}$	$\text{26.34}_{\pm\text{0.21}}$	$\text{40.82}_{\pm\text{0.22}}$	$\text{62.77}_{\pm\text{0.87}}$	$\text{51.80}_{\pm\text{0.45}}$

Appendix

Table 10: Performance comparison with methods selecting

\mathcal{D}_{\text{constraint}}

for replay on MetaMathQA (%). The best results are highlighted in red , and the second-best results are highlighted in blue .

Model	Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Model	Method	HASW	OBQA	MMLU	HE	Avg.	GSM8K	Avg.
	LESS	${\color[rgb]{0,0,1}\text{53.09}_{\pm\text{0.49}}}$	${\color[rgb]{0,0,1}\text{30.47}_{\pm\text{0.58}}}$	$\text{53.62}_{\pm\text{0.40}}$	$\text{26.27}_{\pm\text{0.52}}$	$\text{40.86}_{\pm\text{0.25}}$	${\color[rgb]{0,0,1}\text{63.63}_{\pm\text{0.87}}}$	${\color[rgb]{0,0,1}\text{52.25}_{\pm\text{0.45}}}$
	FisherSFT	$\text{53.01}_{\pm\text{0.49}}$	$\text{29.60}_{\pm\text{0.60}}$	$\text{55.07}_{\pm\text{0.40}}$	$\text{26.94}_{\pm\text{0.61}}$	$\text{41.16}_{\pm\text{0.26}}$	$\text{61.46}_{\pm\text{0.87}}$	$\text{51.31}_{\pm\text{0.45}}$
	SPICE	$\text{52.50}_{\pm\text{0.50}}$	$\text{29.13}_{\pm\text{0.58}}$	${\color[rgb]{0,0,1}\text{55.57}_{\pm\text{0.40}}}$	${\color[rgb]{0,0,1}\text{27.48}_{\pm\text{0.59}}}$	${\color[rgb]{0,0,1}\text{41.17}_{\pm\text{0.26}}}$	$\text{62.55}_{\pm\text{0.85}}$	$\text{51.86}_{\pm\text{0.44}}$
LLaMA3.2	NADS (Ours)	${\color[rgb]{1,0,0}\text{54.86}_{\pm\text{0.50}}}$	${\color[rgb]{1,0,0}\text{30.80}_{\pm\text{0.60}}}$	${\color[rgb]{1,0,0}\text{56.23}_{\pm\text{0.41}}}$	${\color[rgb]{1,0,0}\text{28.21}_{\pm\text{0.42}}}$	${\color[rgb]{1,0,0}\text{42.53}_{\pm\text{0.24}}}$	${\color[rgb]{1,0,0}\text{65.18}_{\pm\text{0.83}}}$	${\color[rgb]{1,0,0}\text{53.86}_{\pm\text{0.43}}}$
	LESS	$\text{54.93}_{\pm\text{0.49}}$	$\text{32.00}_{\pm\text{0.60}}$	$\text{56.24}_{\pm\text{0.40}}$	$\text{30.97}_{\pm\text{0.89}}$	$\text{43.54}_{\pm\text{0.32}}$	${\color[rgb]{0,0,1}\text{65.23}_{\pm\text{0.77}}}$	$\text{54.39}_{\pm\text{0.42}}$

Appendix

Table 11: Cross-paradigm comparison with methods selecting

\mathcal{D}_{\text{new}}

on MetaMathQA (%).

Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Method	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
LESS	$\text{54.12}_{\pm\text{0.49}}$	$\text{30.67}_{\pm\text{0.61}}$	$\text{54.38}_{\pm\text{0.40}}$	$\text{23.73}_{\pm\text{0.62}}$	$\text{40.73}_{\pm\text{0.27}}$	$\textbf{43.01}_{\pm\textbf{0.84}}$	$\text{41.87}_{\pm\text{0.44}}$
LESS + Random + KD	$\text{53.21}_{\pm\text{0.49}}$	$\text{29.73}_{\pm\text{0.61}}$	$\text{53.52}_{\pm\text{0.39}}$	$\text{22.31}_{\pm\text{0.64}}$	$\text{39.69}_{\pm\text{0.27}}$	$\text{42.15}_{\pm\text{0.81}}$	$\text{40.92}_{\pm\text{0.43}}$
LESS + Navigator + KD	$\textbf{55.39}_{\pm\textbf{0.49}}$	$\textbf{31.13}_{\pm\textbf{0.61}}$	$\textbf{55.74}_{\pm\textbf{0.41}}$	$\textbf{25.35}_{\pm\textbf{0.64}}$	$\textbf{41.90}_{\pm\textbf{0.27}}$	$\text{42.76}_{\pm\text{0.86}}$	$\textbf{42.33}_{\pm\textbf{0.45}}$
FisherSFT	$\text{54.03}_{\pm\text{0.49}}$	$\text{29.80}_{\pm\text{0.60}}$	$\text{52.89}_{\pm\text{0.41}}$	$\text{23.15}_{\pm\text{0.62}}$	$\text{39.97}_{\pm\text{0.27}}$	$\text{41.85}_{\pm\text{0.74}}$	$\text{40.91}_{\pm\text{0.39}}$
FisherSFT + Random + KD	$\text{52.44}_{\pm\text{0.49}}$	$\text{29.33}_{\pm\text{0.61}}$	$\text{51.71}_{\pm\text{0.40}}$	$\text{21.86}_{\pm\text{0.64}}$	$\text{38.84}_{\pm\text{0.27}}$	$\text{41.26}_{\pm\text{0.82}}$	$\text{40.05}_{\pm\text{0.43}}$

Appendix

Table 12: Effect of constraint-set quality under the same KD objective. Random + KD uses randomly selected constraint data, while Navigator + KD uses navigator-selected constraint data.

Model	Target Dataset	Navigator	Target	General	Overall
LLaMA3.2-3B	MetaMathQA	Nav-1	64.75	41.21	52.98
		Nav-2	65.35	42.71	54.03
		Nav-3	65.05	41.73	53.39
Gemma3-4B	Magicoder	Nav-1	57.32	45.05	51.19
		Nav-2	58.54	46.53	52.54
		Nav-3	57.93	45.51	51.72

Appendix

Table 13: Comparison of three Navigator checkpoints across five additional model–dataset combinations. Results (%) are obtained using a single seed.

Figure 11: t-SNE visualization of the selected data under different selection methods.

Validation	Utility Spearman	Top- $K$ Overlap	Target	General	Overall
Original	$1.00$	$1.00$	$\text{44.39}_{\pm\text{0.40}}$	$\text{42.52}_{\pm\text{0.27}}$	$\text{43.46}_{\pm\text{0.24}}$
20% label noise	$0.83$	$0.72$	$\text{44.11}_{\pm\text{0.35}}$	$\text{41.72}_{\pm\text{0.31}}$	$\text{42.92}_{\pm\text{0.26}}$
40% label noise	$0.69$	$0.58$	$\text{43.29}_{\pm\text{0.61}}$	$\text{40.94}_{\pm\text{0.35}}$	$\text{42.12}_{\pm\text{0.30}}$
Template shifted	$0.76$	$0.64$	$\text{43.70}_{\pm\text{0.35}}$	$\text{41.20}_{\pm\text{0.33}}$	$\text{42.45}_{\pm\text{0.28}}$

Appendix

Table 14: Navigator selection sensitivity to validation perturbations on Magicoder with LLaMA3.2.

Figure 12: Quantitative diversity diagnostics of the selected data under different selection methods. Higher values indicate stronger diversity under the corresponding metric.

Figure 13: Performance and cost visualization of different selection paradigms on Magicoder with LLaMA3.2-3B. Target-data selection methods select from $\mathcal{D}_{\text{new}}$ , whereas replay-based methods and NADS select constraints from the synthetic pool $\mathcal{C}$ .

Navigator	Spearman	Overlap@K	Rel. Time	Rel. Mem.
LoRA navigator	0.89	0.82	0.74 $\times$	0.16 $\times$
Full navigator	1.00	1.00	1.00 $\times$	1.00 $\times$

Appendix

Table 15: Comparison between full-parameter and LoRA-based navigators.

Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Method	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
NADS + LoRA navigator	55.03	30.93	56.24	27.22	42.36	44.28	43.32
NADS + Full navigator	55.17	31.07	56.45	27.37	42.52	44.39	43.46

Appendix

Table 16: Comparison between LoRA-based and full-parameter navigators under the same final training objective. Both use the same synthetic candidate pool and KD objective.

Model	Method	Navigator Training (LoRA)	Utility Computation	Subset Optimization	Final Training	Total Cost (LoRA)	Overall (LoRA)
LLaMA3.2-3B	InsCL	–	–	12.69	22.85	35.54	41.45
	LER	–	–	7.96	22.85	30.81	41.81
	Mix-cd	–	–	18.23	22.85	41.08	42.01
	NADS	15.83 (11.72)	5.19	0.02	23.78	44.82 (40.71)	43.46 (43.32)
Gemma3-4B	InsCL	–	–	12.69	27.61	40.30	50.34
	LER	–	–	9.67	27.61	37.28	50.76

Appendix

Table 17: Stage-wise computational costs (GPU-hours) and overall performance (%) on Magicoder. Parenthesized values correspond to LoRA-based Navigator training.

Figure 14: NADS performance under different synthetic datasets across seven evaluation metrics on the Magicoder dataset with LLaMA3.2-3B. Each group of bars compares the results obtained with different synthetic data pools under the same setting.

$K$	General Capability					Target	Overall
$K$	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
1%	52.84	28.73	52.54	22.87	39.25	44.93	42.09
3%	53.97	30.13	54.62	25.40	41.03	44.50	42.77
5%	55.17	31.07	56.45	27.37	42.52	44.39	43.46
10%	55.34	31.13	56.73	27.42	42.66	44.38	43.52
15%	55.15	30.87	56.24	27.14	42.35	43.86	43.11

Appendix

Table 18: Target performance and general capability of NADS under different constraint-set sizes

K

Tasks	Pre-trained	Standard FT	InsCL	LER	Mix-cd	NADS
General Capability $\uparrow$
ARC-Challenge	$42.92$	$44.37$	$44.71$	$45.05$	$45.65$	$46.67$
ARC-Easy	$74.58$	$73.48$	$75.34$	$75.13$	$75.25$	$75.93$
PIQA	$76.28$	$75.41$	$76.50$	$76.71$	$77.58$	$77.37$
HellaSwag	$55.83$	$57.42$	$57.05$	$57.55$	$58.10$	$59.00$
OpenBookQA	$31.20$	$33.20$	$33.00$	$33.80$	$33.60$	$34.60$

Appendix

Table 19: Performance comparison on general capability and IFEval benchmarks (%). Results are obtained using a single random seed. Pre-trained and Standard FT are excluded from the ranking. The best results are highlighted in red , and the second-best results are highlighted in blue .

Method	General Capability $\uparrow$					Target $\uparrow$	Overall $\uparrow$
Method	HASW	OBQA	MMLU	GSM8K	Avg.	HE	Avg.
LLaMA3.2-3B
Pre-trained	$\text{55.83}_{\pm\text{0.00}}$	$\text{31.20}_{\pm\text{0.00}}$	$\text{56.92}_{\pm\text{0.00}}$	$\text{27.90}_{\pm\text{0.00}}$	$\text{42.96}_{\pm\text{0.00}}$	$\text{28.16}_{\pm\text{0.00}}$	$\text{35.56}_{\pm\text{0.00}}$
Standard FT	$\text{52.12}_{\pm\text{0.52}}$	$\text{28.40}_{\pm\text{0.69}}$	$\text{48.61}_{\pm\text{0.40}}$	$\text{20.75}_{\pm\text{0.74}}$	$\text{37.47}_{\pm\text{0.30}}$	$\text{45.13}_{\pm\text{0.32}}$	$\text{41.30}_{\pm\text{0.22}}$
LoRA \venuebox ICLR 22	$\text{53.38}_{\pm\text{0.55}}$	$\text{29.47}_{\pm\text{0.64}}$	$\text{51.98}_{\pm\text{0.42}}$	$\text{21.18}_{\pm\text{0.79}}$	$\text{39.00}_{\pm\text{0.31}}$	$\text{41.92}_{\pm\text{0.30}}$	$\text{40.46}_{\pm\text{0.21}}$
STM \venuebox NIPS 25	$\text{53.02}_{\pm\text{0.49}}$	$\text{29.67}_{\pm\text{0.61}}$	$\text{53.24}_{\pm\text{0.41}}$	$\text{23.81}_{\pm\text{0.75}}$	$\text{39.94}_{\pm\text{0.29}}$	$\text{42.04}_{\pm\text{0.82}}$	$\text{40.99}_{\pm\text{0.44}}$

Appendix

Table 20: Comparison with catastrophic forgetting mitigation methods on Magicoder dataset (%). The best results are highlighted in red , and the second-best results are highlighted in blue .

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

May 28, 2026

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Data selection is increasingly used to reduce the cost of large language model (LLM) fine-tuning, with recent methods prioritizing samples by current utility, diversity, quality, or influence. This paper studies a different question: when fine-tuning occurs over multiple stages, can selection strategies that look optimal now make the model less adaptable later? We introduce a long-horizon view of LLM data selection in which a selector is evaluated not only by immediate task performance, but also by future adaptation speed, forgetting, capability imbalance, and out-of-distribution robustness. We compare representative random, loss-based, gradient-based, diversity-based, quality-based, and utility-diversity selection families under a unified multi-stage protocol. Through controlled experiments designed to instantiate this protocol, we show how short-term selectors can exhibit rank reversal: they improve the current stage while slowing subsequent learning and increasing forgetting. We formalize this behavior as myopic selection, provide a simple local analysis of why it can occur, and propose a diagnostic Long-Horizon Aware Selection (LHAS) objective that augments immediate utility with coverage, future-proxy transfer, and anti-concentration terms. The study argues that data selection should be evaluated as a training intervention that shapes the model's learning trajectory, rather than only as a local data-efficiency mechanism.

Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Shanghai University · East China Normal University

Continual Learning for LLMs LLM Fine-Tuning Data Selection Training Data Selection Selective Fine-Tuning Data Selection for Language Model Fine-Tuning

Ask PDF

Mar 8, 2026

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

Gradient-based data selection offers a principled framework for estimating sample utility in large language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the current optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.

Fangxin Wang, Peyman Baghershahi, Langzhou He +3

Department of Computer Science, University of Illinois Chicago

Data Selection for Language Model Fine-Tuning Training Data Selection LLM Fine-Tuning

Ask PDF

Oct 1, 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.

Zhenghao Zhao, Gaowen Liu, Zhiling Lan +1

University of Illinois Chicago · Cisco Research

Continued Pretraining Catastrophic Forgetting Fisher Information

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models