Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

Fangxin Wang; Peyman Baghershahi; Langzhou He; Henry Peng Zou; Sourav Medya; Philip S. Yu

cs.LGMar 8, 2026

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

Authors: Fangxin Wang, Peyman Baghershahi, Langzhou He, Henry Peng Zou, Sourav Medya, Philip S. Yu

Organizations: Department of Computer Science, University of Illinois Chicago

Abstract

Gradient-based data selection offers a principled framework for estimating sample utility in large language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the current optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.

Figures & tables

	Naive	Ghost	Ours
Time	$\mathcal{O}(LT^{2}B_{tr}B_{val}d_{1}d_{2})$	$\mathcal{O}(LT^{2}B_{tr}B_{val}(d_{1}+d_{2}))$	$\mathcal{O}(LTB_{tr}d_{2}(B_{val}d_{1}+T))$
Space	$\mathcal{O}(LB_{tr}B_{val}d_{1}d_{2})$	$\mathcal{O}(LT(B_{tr}+B_{val})d_{1}+T^{2}B_{tr}B_{val})$	$\mathcal{O}(LT(B_{tr}+B_{val})d_{1}+B_{tr}d_{2}\max(d_{1},T))$

Table 1: Time and space complexity comparison for sequential data in a batch.

		References		Selection only			Coupled	Ours
Dataset	Backbone	Origin	Full	TracIn	LESS	GREATS	GRAD-MATCH	Filter-then-Weight
MMLU (Acc)	Llama-3.2-1B	28.13 $\pm$ 0.00	29.09 $\pm$ 1.01	29.24 $\pm$ 0.24	29.74 $\pm$ 0.07	30.07 $\pm$ 0.37	30.14 $\pm$ 0.41	30.93 $\pm$ 0.03
MMLU (Acc)	Qwen3-0.6B	45.86 $\pm$ 0.00	46.64 $\pm$ 0.17	46.45 $\pm$ 0.08	46.56 $\pm$ 0.28	46.06 $\pm$ 0.42	45.72 $\pm$ 0.62	46.63 $\pm$ 0.40
TyDiQA (F1)	Llama-3.2-1B	12.69 $\pm$ 0.54	42.47 $\pm$ 0.30	47.01 $\pm$ 1.43	47.21 $\pm$ 2.58	47.50 $\pm$ 0.56	45.35 $\pm$ 1.35	48.86 $\pm$ 1.67
TyDiQA (F1)	Qwen3-0.6B	20.94 $\pm$ 0.68	43.65 $\pm$ 0.61	45.17 $\pm$ 0.97	45.07 $\pm$ 0.95	45.25 $\pm$ 0.43	44.23 $\pm$ 1.33	46.29 $\pm$ 0.62

Table 2: Best-of-run performance. For each seed we select the best checkpoint per method and report mean

\pm

std over seeds. Bold marks the best result, underline the second best.

		References		Selection only			Coupled	Ours
Dataset	Backbone	Origin	Full	TracIn	LESS	GREATS	GRAD-MATCH	Filter-then-Weight
MMLU (Acc)	Llama-3.2-1B	28.13 $\pm$ 0.00	28.67 $\pm$ 0.63	29.04 $\pm$ 0.04	29.67 $\pm$ 1.31	29.59 $\pm$ 0.40	28.77 $\pm$ 0.56	30.93 $\pm$ 0.03
MMLU (Acc)	Qwen3-0.6B	45.86 $\pm$ 0.00	46.59 $\pm$ 0.10	46.16 $\pm$ 0.07	45.98 $\pm$ 0.37	46.06 $\pm$ 0.42	44.98 $\pm$ 0.72	46.23 $\pm$ 0.67
TyDiQA (F1)	Llama-3.2-1B	12.69 $\pm$ 0.54	38.68 $\pm$ 1.36	47.01 $\pm$ 1.34	46.88 $\pm$ 0.91	47.16 $\pm$ 1.08	44.15 $\pm$ 0.85	48.67 $\pm$ 2.68
TyDiQA (F1)	Qwen3-0.6B	20.94 $\pm$ 0.68	41.14 $\pm$ 0.64	43.07 $\pm$ 0.97	43.57 $\pm$ 0.28	42.70 $\pm$ 0.92	43.14 $\pm$ 1.13	44.82 $\pm$ 1.00

Table 3: Fixed-budget performance. All methods evaluated at the final checkpoint of the

5\%

data budget, formatting as in Table 2 .

Table 5

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Time (s / step)	Relative to SFT
Standard Training
Full Data (SFT, 4 steps)	5.75	1.0 $\times$
Selection Baselines (Online Adapted)
TracIn [ 22 ]	4.12	0.72 $\times$
LESS [ 32 ]	4.14	0.72 $\times$
GREATS [ 29 ]	6.23	1.08 $\times$

Appendix

Table 6: Running Time Comparison. We report the average running time (seconds per step) on NVIDIA H200 GPUs.

$B_{tr}$	Method	$T{=}512$	$T{=}1024$	$T{=}2048$	$T{=}4096$	$T{=}8192$	$T{=}16384$	$T{=}65536$
8	Ghost	136	439	1,651	6,489	25,829	OOM	OOM
8	Ours	36	39	44	55	76	118	371
32	Ghost	441	1,654	6,495	25,842	OOM	OOM	OOM
32	Ours	43	52	69	105	177	321	1,183
64	Ghost	848	3,273	12,955	51,646	OOM	OOM	OOM
64	Ours	51	68	103	173	312	591	2,265

Appendix

Table 7: Empirical peak GPU memory (MB) of the score tensor under the Llama-3.2-1B LoRA setting. Bold = OOM on H100 80 GB.

$B_{tr}$	Method	$T{=}512$	$T{=}1,024$	$T{=}2,048$	$T{=}4,096$	$T{=}8,192$	$T{=}16,384$	$T{=}65,536$
8	Ghost	0.18	0.46	1.55	5.59	22.30	OOM	OOM
8	Ours	0.20	0.13	0.12	0.15	0.08	0.12	0.35
64	Ghost	0.75	2.87	11.29	48.73	OOM	OOM	OOM
64	Ours	0.08	0.09	0.12	0.20	0.32	0.60	2.05
128	Ghost	1.48	5.68	22.46	OOM	OOM	OOM	OOM
128	Ours	0.08	0.11	0.17	0.30	0.57	1.03	3.97

Appendix

Table 8: Empirical wallclock (ms, mean over 3 calls) of one score-tensor invocation. Same setting as Table 7 .

$T$	Stock PyTorch	+ Liger
$2{,}048$	✓	✓
$4{,}096$	OOM ( $15.6$ GB CE)	✓
$8{,}192$	—	✓
$16{,}384$	—	✓
$32{,}768$	—	OOM

Appendix

Table 9: End-to-end OOM frontier with and without Liger Kernel (Llama-3.2-1B LoRA,

B_{tr}{=}8

B_{val}{=}4

, single H100 80 GB).

Data Selection for Language Model Fine-Tuning

See all 25 papers

The Long-Term Effects of Data Selection in LLM Fine-Tuning
May 28, 2026Yuxin Yang, Aoxiong Zeng, Xiangquan YangContinual Learning for LLMs LLM Fine-Tuning
Shanghai University · East China Normal University
Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning
Sep 30, 2026Hongyu Chen, Xinyi Luo, Ming Zhao +6LLM Fine-Tuning Data Selection for Language Model Fine-Tuning
Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China
BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
Jun 17, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Training Data Selection Bilevel Optimization
University of Oxford

May 28, 2026J/K move · Enter open · S save

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Shanghai University · East China Normal University

Continual Learning for LLMs LLM Fine-Tuning Data Selection Training Data Selection Selective Fine-Tuning Data Selection for Language Model Fine-Tuning

Open paper AskPDF arXiv

Data selection is increasingly used to reduce the cost of large language model (LLM) fine-tuning, with recent methods prioritizing samples by current utility, diversity, quality, or influence. This paper studies a different question: when fine-tuning occurs over multiple stages, can selection strategies that look optimal now make the model less adaptable later? We introduce a long-horizon view of LLM data selection in which a selector is evaluated not only by immediate task performance, but also by future adaptation speed, forgetting, capability imbalance, and out-of-distribution robustness. We compare representative random, loss-based, gradient-based, diversity-based, quality-based, and utility-diversity selection families under a unified multi-stage protocol. Through controlled experiments designed to instantiate this protocol, we show how short-term selectors can exhibit rank reversal: they improve the current stage while slowing subsequent learning and increasing forgetting. We formalize this behavior as myopic selection, provide a simple local analysis of why it can occur, and propose a diagnostic Long-Horizon Aware Selection (LHAS) objective that augments immediate utility with coverage, future-proxy transfer, and anti-concentration terms. The study argues that data selection should be evaluated as a training intervention that shapes the model's learning trajectory, rather than only as a local data-efficiency mechanism.

Abstract

Data Selection for Language Model Fine-Tuning

Explore similar work

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training