Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning

Hongyu Chen; Xinyi Luo; Ming Zhao; Lin Tang; Zihan Xu; Jing Li; Yuxuan Wang; Haoran Deng

cs.LGSep 30, 2026

Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuning

Authors: Hongyu Chen, Xinyi Luo, Ming Zhao, Lin Tang, Zihan Xu, Jing Li, Yuxuan Wang, Haoran Deng, +1 more

Organizations: Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Online batch selection fine-tunes a language model on the most useful part of each candidate batch. Selectors that match the gradient of the candidate batch are attractive because they need no held-out data, yet they rarely beat training on the whole batch. We show why. In-sample gradient matching uses every example as part of its own target, so its objective credits each example with its own gradient noise. This is the covariance penalty that makes training error optimistic, now sitting on the diagonal of the gradient Gram matrix: it steers selection toward the noisiest examples and makes the full batch the best solution the objective can reach. The fix costs nothing. For each example, the other candidates form an independent sample of the data distribution, so removing the diagonal turns the matching objective into an unbiased estimate of the update's error with respect to the population gradient. The minimizer of this leave-one-out objective weights examples by their gradient signal-to-noise ratio (SNR), and whenever per-example SNR is heterogeneous enough, half of a batch yields a lower-error update than the whole batch; we give the exact condition. We build \method{} on this principle. It computes the Gram matrix in the metric of the Adam preconditioner during the ordinary backward pass, selects a weighted subset greedily with a $(1-e^{-γ})$ guarantee, and uses no held-out data. Across four fine-tuning tasks and seven backbones from 1.5B to 8B parameters, LOOM improves on full-batch training by 2.3 and 2.4 points on Llama-3.1-8B and Qwen2.5-7B, exceeds every in-sample gradient matcher by 2.4 points and the validation-guided GREATS and OPUS by 1.6--2.0, and selects injected label noise at under a fifth of its base rate.

Figures & tables

Figure 1: The diagonal is the bias (exact computation on the signal-plus-noise model of § 3 ; $B{=}64$ , log-normal noise). (a) In-sample targets over-estimate an example’s alignment with the population gradient by its noise energy divided by $B$ (Prop. 1 ); leave-one-out targets are unbiased. (b) At $b{=}B/2$ , in-sample matchers and MaxGrad select the noisiest examples, while Loom selects the cleanest. (c) Error of the selected update relative to the full batch: in-sample matching never beats the full batch, whereas leave-one-out matching does at every $b/B\geq 1/4$ . (d) At $b{=}B/2$ the best achievable risk ratio is $2/H_{1/2}$ (Thm. 1 ), so half of a batch beats the whole once the SNR heterogeneity $H_{1/2}$ exceeds 2.

Figure 2: Loom in one training step. One backward pass yields per-example LoRA gradients, scaled by the Adam preconditioner, and their Gram matrix $K$ . In-sample targets count its diagonal as signal; Loom validates each candidate against the other $B{-}1$ , selects a weighted subset greedily, and takes an unchanged AdamW step.

		Llama-3.1-8B					Qwen2.5-7B
Method	Ext.	MMLU	SciQA	GSM8K	HumanE.	Avg.	MMLU	SciQA	GSM8K	HumanE.	Avg.
Regular	–	38.3	93.2	56.0	29.3	54.2	55.3	94.6	78.2	45.8	68.5
Random	–	35.6	92.9	54.9	26.8	52.6	54.6	93.5	77.8	41.3	66.8
MaxLoss	–	35.7	92.8	55.4	27.2	52.8	54.8	93.2	77.9	42.1	67.0
MaxGrad	–	35.9	92.8	55.1	26.9	52.7	54.7	93.9	77.7	41.6	67.0
UDS	–	40.1	94.3	58.9	30.8	56.0	59.6	95.2	79.6	46.2	70.2

Table 1: Main results (accuracy or pass@1, %, mean of four seeds; standard deviations in Table 6 ). Every selector keeps

b{=}64

B{=}128

candidates per step; Regular updates on all 128. Ext. : external resource (ref.: reference model; val.: validation set; proxy: held-out proxy set). Groups: full batch; validation-free scoring; validation-free in-sample gradient matching; external resources; ours. Bold / underline : best/second best.

Figure 3: Mechanism on real batches (Llama-3.1-8B unless noted). (a) Error of the selected update against a reference gradient from 4,096 held-out training examples (MMLU, step 400), relative to Regular. (b) SNR heterogeneity

H_{1/2}

of candidate batches during training. (c) Gain of Loom over Regular against the task’s median

H_{1/2}

for seven backbones and four tasks (colors as in b). (d) Share of corrupted examples among the selected ones when 20% of responses are corrupted.

Variant	Avg.	$\bm{\Delta}$	Err.
Loom (full)	56.53	—	0.61
Target
in-sample target (keeps diagonal)	53.74	$-2.79$	1.38
momentum target $m_{t-1}$	55.34	$-1.19$	0.86
2-fold cross-fitted target	56.21	$-0.32$	0.68
Metric

Table 2: Ablations on Llama-3.1-8B (four-task average). Err. : error of the selected update against a 4,096-example reference gradient, relative to Regular (lower is better). Further variants are in Table 13 .

Figure 4: Training dynamics and budgets (Llama-3.1-8B, four-task average). (a) Accuracy over training; dashed: Loom reaches Regular’s final accuracy (dotted) after 57% of the steps. (b) The same runs over wall-clock time. (c) Selected fraction

\rho

(

\rho{=}1

is Regular for all but Loom , which reweights the batch). (d) Candidate batch size

B

\rho{=}1/2

: gain over Regular trained on the same batches.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Result
Prop. 1 (ii): in-sample target error regressed on $\tfrac{1}{B}\langle g_{j},g_{j}-\mu\rangle$	slope $0.998$ (theory $1$ )
Prop. 1 (i): mean leave-one-out target error ( $\\|\mu\\|^{2}{=}1$ )	$-0.0030\pm 0.0019$ (theory $0$ )
Prop. 1 (iii): risk estimate of a fixed separable rule, relative error	LOO $-0.0\%$ ; in-sample $-11.5\%$ (penalty $-11.5\%$ )
Prop. 2 : closed-form SNR weights vs. numerical optimum	max. relative difference $<10^{-14}$
Prop. 2 : optimal risk vs. $\\|\mu\\|^{2}/(1+\mathrm{SNR}_{S})$	$0.16435$ vs. $0.16435$
Thm. 1 : oracle risk ratio at $b{=}B/2$ , MC vs. $B\,\mathrm{HM}/(b\,\mathrm{AM}_{b})$	$1.14$ vs. $1.14$ ; $0.45$ vs. $0.46$ ; $0.13$ vs. $0.14$

Appendix

Table 3: Exact numerical checks of the analytic results on synthetic gradient batches ( code/theory_checks.py ). MC: Monte Carlo over 120–4,000 batches.

Task	Train (size)	Test (size)
MMLU	auxiliary train (99,842)	test (14,042)
ScienceQA	train (12,726)	test (4,241)
GSM8K	train (7,473)	test (1,319)
Code	CodeAlpaca-20k (20,022)	HumanEval (164)

Appendix

Table 4: Training and evaluation data. The metric is accuracy, except pass@1 on HumanEval.

Hyperparameter	Value
LoRA rank / $\alpha$ / dropout	8 / 16 / 0
Target modules	q, k, v, o, gate, up, down
Optimizer	AdamW ( $\beta{=}(0.9,0.999)$ , wd 0)
Learning rate	$1.5{\times}10^{-4}$ (MMLU), $3{\times}10^{-4}$ (others)
Schedule	warm-up ratio 0.01, cosine decay
Epochs	1 (MMLU, GSM8K), 20 (ScienceQA), 2 (Code)

Appendix

Table 5: Training and Loom hyperparameters.

	Llama-3.1-8B				Qwen2.5-7B
Method	MMLU	SciQA	GSM8K	HumanE.	MMLU	SciQA	GSM8K	HumanE.
Regular	$38.3_{\pm 0.35}$	$93.2_{\pm 0.21}$	$56.0_{\pm 0.47}$	$29.3_{\pm 0.48}$	$55.3_{\pm 0.79}$	$94.6_{\pm 0.17}$	$78.2_{\pm 0.08}$	$45.8_{\pm 0.41}$
Random	$35.6_{\pm 1.12}$	$92.9_{\pm 0.27}$	$54.9_{\pm 0.73}$	$26.8_{\pm 0.61}$	$54.6_{\pm 1.61}$	$93.5_{\pm 0.33}$	$77.8_{\pm 0.31}$	$41.3_{\pm 1.06}$
MaxLoss	$35.7_{\pm 0.79}$	$92.8_{\pm 0.24}$	$55.4_{\pm 0.41}$	$27.2_{\pm 0.33}$	$54.8_{\pm 1.29}$	$93.2_{\pm 0.38}$	$77.9_{\pm 0.35}$	$42.1_{\pm 0.79}$
MaxGrad	$35.9_{\pm 0.94}$	$92.8_{\pm 0.26}$	$55.1_{\pm 0.55}$	$26.9_{\pm 0.64}$	$54.7_{\pm 0.72}$	$93.9_{\pm 0.47}$	$77.7_{\pm 0.29}$	$41.6_{\pm 0.52}$
UDS	$40.1_{\pm 0.52}$	$94.3_{\pm 0.25}$	$58.9_{\pm 0.28}$	$30.8_{\pm 0.63}$	$59.6_{\pm 0.39}$	$95.2_{\pm 0.21}$	$79.6_{\pm 0.24}$	$46.2_{\pm 0.37}$

Appendix

Table 6: Main results with standard deviations over four seeds.

Method	Q2.5-1.5B	L3.2-3B	Q2.5-3B	Mis-7B	Q3-8B
Regular	57.6	45.9	63.3	51.8	75.9
Random	56.0	44.4	61.7	50.2	74.5
GradMatch	57.0	45.3	62.8	51.3	75.4
UDS	58.7	47.1	64.4	53.0	76.8
GREATS	58.1	46.4	63.8	52.3	76.3
Loom	59.5	47.8	65.2	53.7	77.6

Appendix

Table 7: Four-task averages on five further backbones (Q: Qwen, L: Llama, Mis: Mistral-v0.3; Q3-8B is the base model).

Backbone	Method	MMLU	SciQA	GSM8K	HumanE.	Avg.
Qwen2.5-1.5B	Regular	44.8	88.1	61.9	35.4	57.6
	Random	43.1	87.2	60.8	32.7	56.0
	GradMatch	44.3	87.8	61.5	34.4	57.0
	UDS	46.2	88.9	63.4	36.2	58.7
	GREATS	45.6	88.6	62.7	35.3	58.1
	Loom	47.1	89.4	63.9	37.4	59.5

Appendix

Table 8: Per-task results on the further backbones.

Figure 5: Per-task accuracy over training on Llama-3.1-8B.

Method	0%	20%	40%	Corr. sel.
Regular	54.20	51.10	47.35	(20.0)
Random	52.55	49.60	45.70	(20.0)
MaxLoss	52.78	47.20	41.90	43.6
UDS	56.03	53.05	49.40	15.2
CRAIG	53.40	49.80	45.60	29.8
GradMatch	53.68	50.20	46.30	27.5

Appendix

Table 9: Label noise (Llama-3.1-8B, four-task average) at 0/20/40% corrupted responses. Corr. sel. : corrupted share of the selected examples at 20% (%, lower is better; parentheses: no selection).

Method	MMLU	SciQA	GSM8K	HumanE.	Avg.
Regular	35.6	91.8	52.4	24.6	51.10
Random	33.4	91.1	51.1	22.8	49.60
MaxLoss	31.2	90.2	48.6	18.8	47.20
UDS	37.0	93.1	55.2	26.9	53.05
CRAIG	34.1	91.3	51.3	22.5	49.80
GradMatch	34.5	91.6	51.7	23.0	50.20

Appendix

Table 10: Per-task results at 20% corrupted responses (Llama-3.1-8B).

Method	Thr.	Mem.	Ovh.	Steps	Time
Regular	1.00	37.9	–	100%	100%
Random	1.74	31.6	–	–	–
UDS	1.21	32.8	2.9%	62%	51%
GradMatch	0.95	41.2	4.8%	–	–
GREATS	0.92	41.9	8.6%	88%	96%
Loom -fast	1.17	33.4	3.6%	64%	55%

Appendix

Table 11: Cost on Llama-3.1-8B (8

\times

A100). Thr. : throughput relative to Regular; Mem. : peak GB per GPU; Ovh. : selection overhead; Steps / Time : steps and wall-clock needed to reach Regular’s final average, relative to Regular.

Stage	ms / step	Share
forward	3,120	32.0%
backward	6,030	61.9%
per-example gradients	318	3.3%
Gram matrix	142	1.5%
greedy selection + NNLS	39	0.4%
optimizer update	93	1.0%

Appendix

Table 12: Time per optimizer step of Loom (Llama-3.1-8B, 128 candidates, 8

\times

A100). Share: fraction of Loom ’s step.

Variant	Avg.	$\bm{\Delta}$	Err.
Loom (full)	56.53	—	0.61
Target
in-sample target (keeps diagonal)	53.74	$-2.79$	1.38
momentum target $m_{t-1}$	55.34	$-1.19$	0.86
2-fold cross-fitted target	56.21	$-0.32$	0.68
Metric

Appendix

Table 13: Extended ablations on Llama-3.1-8B (four-task average; Err. as in Table 2 ).

Figure 6: Diagnostics on Llama-3.1-8B. (a) Estimated per-example SNR per task. (b) Loom ’s weights against estimated SNR for the selected examples of one MMLU batch (dotted: log–log fit). (c) Fraction of the batch with nonzero weight when

b{=}B

, and estimated shrinkage

\hat{\kappa}

, over training. (d) Share of

\mathrm{tr}(K)

by layer block in the Euclidean and the Adam metric.

Figure 7: Sensitivity on Llama-3.1-8B. (a) Ridge

\bar{\lambda}

(four-task average). (b) Accuracy against the rate of corrupted responses. (c) Mean weight by candidate-loss decile (uniform selection

=1

Category	%
answer inconsistent with the question or options	31
truncated or malformed response	19
duplicate / near-duplicate of a batch neighbour	14
off-format (explanation without an answer letter)	17
correct and well-formed	19

Appendix

Table 14: The 500 lowest-weight examples of one MMLU run of Loom (Llama-3.1-8B), by manual inspection.

	Llama-3.1-8B				Mistral-7B-v0.3
Method	MMLU	TydiQA	BBH	Avg.	MMLU	TydiQA	BBH	Avg.
Regular	64.1	56.3	63.2	61.2	61.2	55.7	57.9	58.3
Random	63.4	55.1	62.0	60.2	60.3	54.6	56.8	57.2
GradMatch	63.8	55.8	62.7	60.8	60.8	55.2	57.4	57.8
UDS	64.9	57.2	63.9	62.0	61.9	56.5	58.6	59.0
GREATS	65.3	60.4	64.1	63.3	62.2	60.2	58.9	60.4

Appendix

Table 15: GREATS protocol: training on a mixed instruction pool, evaluating MMLU (5-shot accuracy), TydiQA (1-shot F1), and BBH (3-shot exact match).

Method	MMLU	SciQA	GSM8K	HumanE.	Avg.
ratio $\rho$	12.5%	50%	25%	50%
Regular	55.3	94.6	78.2	45.8	68.5
GradMatch	53.9	94.3	77.4	44.5	67.5
GREATS	58.1	94.2	78.6	45.1	69.0
UDS	63.2	95.2	79.9	46.2	71.1
Loom	63.9	95.1	80.6	47.6	71.8

Appendix

Table 16: Qwen2.5-7B at the per-task selection ratios

\rho

of Zou et al. (2025a) .

Method	MMLU	SciQA	GSM8K	HumanE.	Avg.
Regular	38.9	92.4	55.1	28.2	53.7
GradMatch	38.1	92.2	53.9	26.4	52.7
UDS	40.3	93.3	56.8	28.9	54.8
Loom	41.4	93.8	54.3	27.9	54.4
Loom (stratified)	41.2	93.9	57.9	29.6	55.7

Appendix

Table 17: Training on the union of the four training sets (Llama-3.1-8B).

Method	MMLU	SciQA	GSM8K	HumanE.	Avg.
Regular	46.1	88.7	63.0	36.1	58.5
GradMatch	45.5	88.4	62.6	35.2	57.9
UDS	47.4	89.5	64.4	36.9	59.6
Loom	48.5	89.9	65.1	38.0	60.4

Appendix

Table 18: Full fine-tuning of Qwen2.5-1.5B.

Method	GSM8K	MATH500	SVAMP	Avg.
Regular	56.0	17.8	71.2	48.3
UDS	58.9	18.9	72.6	50.1
GREATS	57.0	18.1	71.8	49.0
GradMatch	55.7	17.5	70.9	48.0
Loom	58.6	19.6	73.4	50.5

Appendix

Table 19: Transfer of GSM8K-trained Llama-3.1-8B to MATH500 and SVAMP (accuracy).

Explore similar work

Jun 17, 2026cs.LG

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories. Beyond static heuristic filtering, advanced data selection methods for LLM training largely follow two paradigms, each with fundamental limitations. Influence-based methods provide principled bi-level objectives but require intractable inverse-Hessian computations, while excess-loss methods are computationally efficient but rely on a static reference model that becomes misaligned with the evolving proxy model during training. We propose BLADE (Bi-Level Adaptive Data sElection), a Hessian-free framework for data selection. BLADE reformulates the bi-level optimization problem underlying influence-based methods as a penalized single-level objective via Lagrange multipliers, avoiding inverse-Hessian computation while revealing a principled connection to excess-loss based data selection. The resulting objective recovers an excess-loss form but replaces the static reference model with a dynamic one that stays synchronized with training. Theoretically, we prove that this penalized formulation guarantees first-order convergence. For efficient online batch selection, we instantiate BLADE as a memoryless randomized block-coordinate Frank-Wolfe algorithm. Extensive experiments show that BLADE consistently outperforms state-of-the-art data selection baselines, providing a practical recipe for LLM training.

Jiaxing Wang, Deping Xiang, Jin Xu +9

Alignment-Aware Selection Large Language Model Training Hessian

Ask PDF

Jun 6, 2026cs.LG

Minibatch Selection for Language Models via Partition Matroid Constrained Gradient Matching

Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains. Existing methods either select samples independently within each domain or rely on computationally expensive proxy models to learn continuous domain weights. We propose PartitionSel, a cross-domain minibatch selection approach that maximizes a validation-guided gradient-matching utility under per-domain budgets encoded as a partition-matroid constraint. By coupling the per-domain budgets through a single utility, PartitionSel is designed to reduce redundancy in selections across domains. The proposed objective is weakly submodular and admits an orthogonal matching pursuit algorithm with provable approximation guarantees. Empirically, we evaluate PartitionSel for minibatch selection during the fine-tuning of Qwen2.5 and Llama-3 on MetaMathQA and Mol-Instructions. PartitionSel achieves robust gains over per-domain and domain-agnostic baselines on both benchmarks. It also reduces the number of conflicting gradient pairs within each batch, indicating that the cross-domain coupling translates into more compatible training updates.

Prayas Agrawal, Prateek Chanda, Ishita Khatri +3

Department of Computer Science and Engineering, Indian Institute of Technology Bombay · Microsoft Research India · Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay +1

Topic pending

Ask PDF

May 28, 2026cs.LG

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Data selection is increasingly used to reduce the cost of large language model (LLM) fine-tuning, with recent methods prioritizing samples by current utility, diversity, quality, or influence. This paper studies a different question: when fine-tuning occurs over multiple stages, can selection strategies that look optimal now make the model less adaptable later? We introduce a long-horizon view of LLM data selection in which a selector is evaluated not only by immediate task performance, but also by future adaptation speed, forgetting, capability imbalance, and out-of-distribution robustness. We compare representative random, loss-based, gradient-based, diversity-based, quality-based, and utility-diversity selection families under a unified multi-stage protocol. Through controlled experiments designed to instantiate this protocol, we show how short-term selectors can exhibit rank reversal: they improve the current stage while slowing subsequent learning and increasing forgetting. We formalize this behavior as \emph{myopic selection}, provide a simple local analysis of why it can occur, and propose a diagnostic Long-Horizon Aware Selection (LHAS) objective that augments immediate utility with coverage, future-proxy transfer, and anti-concentration terms. The study argues that data selection should be evaluated as a training intervention that shapes the model's learning trajectory, rather than only as a local data-efficiency mechanism.

Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Shanghai University · East China Normal University

Large Language Model Fine-Tuning Model Fine-Tuning Long-Horizon Planning

Ask PDF