Refreshing Less, Selecting Better: Reusing Stale Gradient Features for Efficient Influence-Based Data Selection

Jianchang Su; Yifan Zhang; Wei Zhang

cs.LGSep 26, 2026

Refreshing Less, Selecting Better: Reusing Stale Gradient Features for Efficient Influence-Based Data Selection

Authors: Jianchang Su, Yifan Zhang, Wei Zhang

Organizations: University of Connecticut

Abstract

Gradient-based data selection methods such as LESS score each candidate by the alignment between its gradient and a target validation gradient, and recomputing per-example gradient features at every new checkpoint dominates their cost. Across three selection seeds, two model families, two candidate pools, and two target tasks, features cached at a post-warmup checkpoint and paired with fresh validation gradients preserve the ranking 40 optimizer steps later with Spearman correlation from 0.952 to 0.991, while the top-10% subset they induce misses 10 to 22% of the examples that full recomputation selects. We therefore propose Cached Diverse Influence Selection (CDIS), which recomputes gradient features for the top-ranked fraction $p$ of candidates under the stale scores, fits an affine calibration on the recomputed examples, and selects the final subset under source and length quotas. A refresh fraction at or above the selection fraction recovers the exact top- $k$ subset whenever the calibrated stale scores have bounded error, and the budget curves follow this rule: at $p=0.3$ the recovered top- $k$ subset coincides with full recomputation in every setting, allocating the same budget per stratum recovers the stratified subset at 0.92 to 1.00, and gradient-stage wall-clock drops 3.5 to 3.6 times. Iterating the cache over four checkpoints keeps top- $k$ overlap at 0.98 or higher at 1.9 gradient features per example against 4 for full recomputation, and stale-to-recomputed agreement on the refreshed examples provides a free check for unsafe reuse. Downstream, unconstrained top- $k$ selection collapses to a single data source and scores 13 points below random selection on GSM8K. CDIS scores 12 points above random selection with paired confidence intervals that exclude zero and trails full recomputation by 4.3 points, one training-run standard deviation, at 3.4 times lower selection cost.

Figures & tables

Figure 1: Headline results. (a) Overlap between the top-1,000 subset selected from partially refreshed scores and the subset from full recomputation against the refresh fraction

p

, for Qwen3-4B-Base on three GSM8K +Dolly pools (mean and min–max band), Llama-3.2-1B with its own gradient features, and Qwen3-4B-Base on a GSM8K +FLAN-v2 pool; the dotted line marks

k/N=0.1

. (b) Full GSM8K exact match against the fraction of train-gradient features recomputed per selection cycle, three-seed means with min–max whiskers.

Panel of Figure 2 — Figure 2: From prior balanced influence selection to CDIS. The selection rule on the right of each panel is shared; the refresh layer in (b) reduces the number of train-gradient features recomputed per checkpoint, and the stale and recomputed scores on $\mathcal{R}_{p}$ provide the online check of Section 4.3 .

		Top-1,000 overlap			Stratified, global allocation			Stratified, per-stratum
Setting	Spearman	stale	$p{=}0.2$	$p{=}0.3$	stale	$p{=}0.2$	$p{=}0.3$	$p{=}0.2$	$p{=}0.3$
Qwen3-4B, seed 0	0.987	0.856	0.999	1.000	0.824	0.920	0.927	0.986	0.999
Qwen3-4B, seed 1	0.976	0.782	0.971	1.000	0.720	0.837	0.857	0.955	0.996
Qwen3-4B, seed 2	0.988	0.817	0.993	1.000	0.830	0.935	0.940	0.986	0.998
Llama-3.2-1B, seed 0	0.991	0.889	1.000	1.000	0.829	0.885	0.904	0.978	0.992
Qwen3-4B, GSM8K +FLAN-v2 pool	0.989	0.855	0.999	1.000	0.833	0.929	0.936	0.989	1.000

Table 1: Selection fidelity of stale reuse and targeted refresh (checkpoint 80 to 120; GSM8K target on GSM8K +Dolly pools unless noted). Overlap is on the top-10% subset under unconstrained top-

k

and under the stratified rule; global allocation refreshes the top-

pN

candidates of the pool and per-stratum allocation the top-

p

fraction of every stratum, at the same budget.

Figure 3: (a) Stratified overlap with full recomputation for stale reuse, global allocation, and per-stratum allocation at

p=0.3

on the seven settings of Table 1 . (b) Repeated selection with an iterated cache on the three-epoch run (Qwen3-4B seed 0): a full pass at checkpoint 80, then

0.3N

features refreshed at each later checkpoint, at 1.9 features per example against 4.0 (Appendix L ).

Figure 4: (a) Fidelity against checkpoint distance on Qwen3-4B seed 0, single refresh from the origin; filled markers start at checkpoint 80 and hollow markers at checkpoint 40. (b) The online check (Spearman correlation between stale and recomputed scores on the

p=0.3

refresh set) against the top-1,000 overlap of stale reuse and of targeted refresh on all fourteen single-refresh intervals (Appendix K ).

Panel of Figure 5 — Figure 5: Unconstrained top- $k$ selection on influence scores collapses to a single source and a narrow length window; CDIS keeps source balance and a wide length range (per-seed profiles in Appendix F ).

		Full GSM8K		Full ASDiv (transfer)
Method	Recompute	EM	CDIS $-$ method [95% CI]	EM	CDIS $-$ method [95% CI]
Unconstrained top- $k$ , full recompute	$N$	0.263	$+$ 24.7 [ $+$ 23.1, $+$ 26.3]	0.163	$+$ 15.6 [ $+$ 14.4, $+$ 16.7]
Plain random	0	0.389	$+$ 12.2 [ $+$ 10.6, $+$ 13.7]	0.248	$+$ 7.1 [ $+$ 6.0, $+$ 8.1]
Length-matched random	0	0.323	$+$ 18.7 [ $+$ 17.0, $+$ 20.3]	0.170	$+$ 14.9 [ $+$ 13.7, $+$ 16.0]
Stratified, full recompute	$N$	0.553	$-$ 4.3 [ $-$ 5.5, $-$ 3.0]	0.339	$-$ 2.0 [ $-$ 2.9, $-$ 1.1]
CDIS, $p=0.3$ , global allocation	$0.3N$	0.510	—	0.319	—

Table 2: Downstream quality on Qwen3-4B-Base (top-10% of the 10k GSM8K +Dolly pool, five-epoch LoRA, same adapters on full GSM8K and full ASDiv). EM is the three-seed mean (per-seed values in Appendix E ); differences are percentage points with paired, seed-stratified bootstrap 95% CIs over per-item correctness (10,000 resamples).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Main model; second family	Qwen3-4B-Base; Llama-3.2-1B
Candidate pool	5k GSM8K train + 5k Dolly (or 5k FLAN-v2), sampled per seed
Selection size	top 10% = 1,000 examples
Projection dimension; gradient type	8,192; Adam-aware projected gradients (LESS)
Cached and target checkpoints	80 and 120 (main); 40 $\to$ 80, 40 $\to$ 120 (early); 80 $\to$ 160/240/320 (three-epoch run)
Refresh fractions	0, 0.05, 0.1, 0.2, 0.3, 0.5, 1

Appendix

Table 3: Settings for the headline experiments.

	Full GSM8K				Full ASDiv
Method	Seed 0	Seed 1	Seed 2	Mean	Seed 0	Seed 1	Seed 2	Mean
Unconstrained top- $k$ , full recompute	0.196	0.237	0.356	0.263	0.126	0.164	0.199	0.163
Plain random	0.529	0.282	0.355	0.389	0.365	0.174	0.206	0.248
Length-matched random	0.356	0.434	0.180	0.323	0.182	0.247	0.081	0.170
Stratified, full recompute	0.661	0.453	0.544	0.553	0.467	0.250	0.299	0.339
CDIS, $p=0.3$	0.616	0.477	0.438	0.510	0.436	0.289	0.232	0.319

Appendix

Table 4: Per-seed exact match behind Table 2 . Both stratified selectors exceed both random controls on every seed and both test sets; CDIS exceeds full recomputation on seed 1 and trails it on seeds 0 and 2, and the two random controls swap places across seeds.

Method	Seed	GSM8K	Dolly	Source entropy	Length mean	Length p10–p90
Unconstrained top- $k$	0	1000	0	0.000	44.8	30–64
Unconstrained top- $k$	1	1000	0	0.000	46.9	30–69
Plain random	0	525	475	0.998	57.9	9–99
Plain random	1	490	510	1.000	61.0	10–121
Length-matched random	0	772	228	0.775	43.4	26–63
Length-matched random	1	844	156	0.625	46.0	29–69

Appendix

Table 5: Source counts, normalized source entropy, and prompt-length statistics of the selected subsets on seeds 0 and 1. Unconstrained selection collapses to one source and a narrow length window; the stratified rules keep source balance and a wide length range.

Fixed selection	Seed a	Seed b	Seed c	Mean	SD	Range
Seed-0 pool, stratified full recompute	0.661	0.666	0.577	0.635	0.050	0.089
Seed-0 pool, CDIS $p=0.3$	0.616	0.583	0.557	0.586	0.030	0.059
Seed-2 pool, stratified full recompute	0.544	0.567	0.479	0.530	0.046	0.088
Seed-2 pool, CDIS $p=0.3$	0.438	0.503	0.425	0.455	0.042	0.077

Appendix

Table 6: Training-seed decomposition on full GSM8K. Each row retrains the same selected subset with three training seeds (the original seed and two others) under identical hyperparameters. The pooled within-selection standard deviation is 0.0425.

		Top-1,000 overlap at step 120
Strategy	Gradient cost	Qwen3 s0	Qwen3 s1	Llama
Checkpoint skipping (reuse step-40 features)	half the scoring passes	0.603	0.568	0.765
Random refresh, 30%	$0.3N$ per checkpoint	0.890	—	0.920
Targeted refresh, 30%	$0.3N$ per checkpoint	1.000	1.000	1.000

Appendix

Table 7: Alternatives at equal or larger savings. Checkpoint skipping reuses features from twice the distance without refresh. Random and targeted refresh recompute the same number of features and differ in which candidates receive them. Random refresh on seed 1 was measured under the stratified rule (Table 8 ) and is omitted at the score level.

Seed	Refresh policy at 30%	Stratified overlap	Top-500 prefix overlap	GSM200	Full GSM8K
0	Targeted	0.927	1.000	0.600	0.616
0	Random	0.839	0.826	0.550	0.545
1	Targeted	0.857	1.000	0.535	0.477
1	Random	0.739	0.734	0.310	0.322

Appendix

Table 8: Targeted against random refresh at the same 30% budget under the stratified rule, with downstream exact match of the resulting adapters.

		stale	$p=0.05$		$p=0.1$		$p=0.2$	$p=0.3$	$p=0.5$
Setting	Interval		cal.	raw	cal.	raw
Qwen3-4B s0	40 $\to$ 80	0.631	0.648	0.631	0.689	0.631	0.826	0.931	0.999
Qwen3-4B s0	40 $\to$ 120	0.603	0.625	0.603	0.661	0.603	0.795	0.904	0.995
Qwen3-4B s0	80 $\to$ 120	0.856	0.859	0.859	0.894	0.894	0.999	1.000	1.000
Qwen3-4B s1	40 $\to$ 80	0.629	0.654	0.629	0.693	0.629	0.819	0.925	0.999
Qwen3-4B s1	40 $\to$ 120	0.568	0.599	0.568	0.643	0.569	0.785	0.913	0.998

Appendix

Table 9: Top-1,000 overlap with full recomputation for targeted refresh. At

p=0.05

and

p=0.1

the columns give the value with (cal.) and without (raw) affine calibration; at

p\geq 0.2

the two agree to within 0.001 on every interval and one value is shown.

Setting	Interval	Check	Stale	Targeted
Qwen3-4B s0	40 $\to$ 80	0.641	0.631	0.931
Qwen3-4B s1	40 $\to$ 80	0.647	0.629	0.925
Qwen3-4B s0	40 $\to$ 120	0.589	0.603	0.904
Qwen3-4B s1	40 $\to$ 120	0.533	0.568	0.913
Qwen3-4B s0	80 $\to$ 120	0.912	0.856	1.000
Qwen3-4B s1	80 $\to$ 120	0.810	0.782	1.000

Appendix

Table 10: The online check on all fourteen single-refresh intervals. Check is the Spearman correlation between stale and recomputed scores on the

p=0.3

refresh set, observable at refresh time; Stale and Targeted are the top-1,000 overlaps of stale reuse and of targeted refresh at

p=0.3

with full recomputation.

Setting	Interval	Global $\rho$	Stale top-10%	Check $p{=}0.1$	Check $p{=}0.2$	Check $p{=}0.3$	Band $R^{2}$
Qwen3-4B s0	40 $\to$ 80	0.911	0.631	0.526	0.621	0.641	0.433
Qwen3-4B s0	80 $\to$ 120	0.987	0.856	0.811	0.864	0.912	0.860
Qwen3-4B s1	40 $\to$ 80	0.916	0.629	0.509	0.612	0.647	0.439
Qwen3-4B s1	80 $\to$ 120	0.976	0.782	0.745	0.781	0.810	0.717

Appendix

Table 11: The global rank correlation stays above 0.91 on the early interval although the top-10% overlap drops to 0.63, whereas the check on the refresh set drops to 0.64.

		Global allocation, top-1,000 / stratified			Per-stratum allocation, top-1,000 / stratified
Setting	stale	$p{=}0.1$	$p{=}0.2$	$p{=}0.3$	$p{=}0.1$	$p{=}0.2$	$p{=}0.3$
Qwen3-4B s0, GSM8K	0.856 / 0.824	0.894 / 0.879	0.999 / 0.920	1.000 / 0.927	0.871 / 0.841	0.910 / 0.986	0.935 / 0.999
Qwen3-4B s1, GSM8K	0.782 / 0.720	0.829 / 0.792	0.971 / 0.837	1.000 / 0.857	0.817 / 0.772	0.884 / 0.955	0.920 / 0.996
Qwen3-4B s2, GSM8K	0.817 / 0.830	0.856 / 0.889	0.993 / 0.935	1.000 / 0.940	0.847 / 0.852	0.894 / 0.986	0.919 / 0.998
Llama-3.2-1B, GSM8K	0.889 / 0.829	0.920 / 0.871	1.000 / 0.885	1.000 / 0.904	0.906 / 0.866	0.915 / 0.978	0.937 / 0.992
Qwen3-4B, FLAN pool, GSM8K	0.855 / 0.833	0.893 / 0.888	0.999 / 0.929	1.000 / 0.936	0.871 / 0.874	0.908 / 0.989	0.935 / 1.000

Appendix

Table 12: Global against per-stratum refresh allocation at equal budget, checkpoint 80 to 120. Under the GSM8K target every example that global allocation misses at

p=0.3

is a Dolly example (73, 143, 60, 96, and 64 of 1,000 in the five rows); under the MMLU target every missed example is a GSM8K example (181 and 235).

Policy ( $p=0.3$ )	Cost	Step 160	Step 240	Step 320	Aggregate
Full recomputation	4.0	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000	1.000 / 1.000
Stale features	1.0	0.736 / 0.573	0.702 / 0.625	0.688 / 0.586	0.820 / 0.713
Iterated, global allocation	1.9	0.992 / 0.749	0.984 / 0.816	0.979 / 0.786	0.999 / 0.844
Iterated, per-stratum allocation	1.9	0.846 / 0.942	0.790 / 0.944	0.777 / 0.939	0.846 / 0.990

Appendix

Table 13: Repeated selection with an iterated cache on the three-epoch run (Qwen3-4B seed 0): a full pass at checkpoint 80, then

0.3N

features refreshed and written back at each later checkpoint. Cost is the cumulative number of train-gradient features per example. Cells are top-1,000 / stratified overlap with full recomputation at that checkpoint; Aggregate compares the learning-rate-weighted sum of the four checkpoint scores, the LESS selection rule.

Setting	Policy ( $p=0.3$ )	Step 1	Check	Step 2	Check	Step 3	Aggregate
Qwen3-4B s0, 40 $\to$ 80 $\to$ 120	stale features	0.631 / 0.456	—	0.603 / 0.477	—	—	0.782 / 0.664
	global, naive	0.931 / 0.688	0.641	0.898 / 0.702	0.950	—	0.992 / 0.792
	global, cohort	0.908 / 0.676	0.722	0.927 / 0.727	0.875	—	0.988 / 0.785
	per-stratum, naive	0.675 / 0.818	0.894	0.613 / 0.838	0.973	—	0.811 / 0.935
	per-stratum, cohort	0.676 / 0.791	0.904	0.751 / 0.875	0.944	—	0.844 / 0.912
Qwen3-4B s1, 40 $\to$ 80 $\to$ 120	stale features	0.629 / 0.473	—	0.568 / 0.394	—	—	0.775 / 0.649

Appendix

Table 14: Iterated chains under the GSM8K target: early-origin chains with both calibrations, and the post-warmup chain with the cohort calibration (its naive rows are in Table 13 above). Cells are top-1,000 / stratified overlap with full recomputation at that checkpoint; Aggregate compares the learning-rate-weighted checkpoint sums. Under the MMLU target the early-origin global chains reach 0.891 and 0.929 top-1,000 overlap at checkpoint 120 and 0.960 and 0.973 for the aggregate on seeds 0 and 1.

Rule (GSM200, seed 0)	Full	$p=0.3$
Unconstrained top- $k$	0.220	—
Length quotas	0.100	0.160
Source quotas	0.410	0.440
Source $\times$ length quotas (50/50)	0.655	0.600
Source $\times$ length quotas (70/30)	0.400	0.440
Source $\times$ length quotas (80/20)	0.350	0.345

Appendix

Table 15: Selection rules on seed 0. Left: single-run exact match on the 200-item GSM200 probe for quota, ratio, and normalization variants. Right: single-run full GSM8K exact match for the label-free cluster rule (

k

-means with

k=8

on cached features) against the labeled rule and controls; Table 6 gives the training-run range for single runs.

Stage	Full recompute	CDIS $p=0.3$	Changed
Train-gradient features at checkpoint 120	3,935.75 s	1,087.33 s	yes
Validation-gradient extraction	106.70 s	106.70 s	no
Scoring, calibration, selection (CPU)	6.83 s	6.83 s	no
Selection cycle	4,049.28 s	1,200.86 s	3.37 $\times$
LoRA training on 1,000 selected examples	1,433 s	1,442 s	no

Appendix

Table 16: Wall-clock accounting of one selection cycle on Qwen3-4B-Base with the 10k seed-0 pool on one NVIDIA A6000. The train-gradient stage covers 10,000 examples for full recomputation and 3,000 for CDIS; LoRA times are two-seed means. The cache footprint is

10{,}000\times 8{,}192

float32 values, about 313 MB.

Cycles $R$	Selection stage	With training
1	1.00 $\times$	1.00 $\times$
2	1.54 $\times$	1.35 $\times$
3	1.88 $\times$	1.53 $\times$
5	2.29 $\times$	1.71 $\times$
10	2.73 $\times$	1.87 $\times$
$\infty$	3.37 $\times$	2.07 $\times$

Appendix

Table 17: Left: speedup over

R

selection cycles when CDIS pays one full pass to build the cache. Right: steady-state per-cycle speedup as the pool grows.

Transferred subset (selected by Qwen3-4B, seed 0)	Mistral train loss	Mistral full GSM8K EM
Plain random	0.740	0.427
Stratified, full recompute	0.619	0.397
CDIS, $p=0.3$	0.649	0.396

Appendix

Table 18: Subset transfer to Mistral-7B-v0.3.

Data Selection

Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.

See all 82 papers

Oct 2, 2026

LESSER: Post-Training Data Selection with Output-Layer Gradients

The choice of post-training data for large language models substantially affects downstream performance. Gradient-based data selection is a popular approach that ranks training data by how well their gradients align with those of a small validation set. However, ranking with full-parameter gradients requires an expensive backward pass on every sample, making computation intractable for large candidate pools. This raises a natural question: can we approximate full-gradient features at a fraction of the cost? Conveniently, we find that output-layer gradients suffice for effective data selection, yet require only the cheaper forward pass. We implement this as LESSER, a drop-in wrapper for selection methods that reduces the feature-extraction FLOP cost by

9.7\times

for SFT and

3.0\times

for RL benchmarks, while tracking full-gradient performance on downstream tasks. Empirically, we find that even when output-layer and full gradients rank individual samples differently, they select batches with aligned gradients.

Lyuxin David Zhang, Eric Wong, Surbhi Goel +1

University of Pennsylvania · University of Texas at Austin

Training Data Selection Data Selection for Language Model Fine-Tuning LLM Post-Training Language Model Post-Training Instruction-Tuning Data Selection

Ask PDF

Mar 5, 2026

Rethinking Representativeness and Diversity in Dynamic Data Selection

Dynamic data selection accelerates training by sampling a changing subset of the dataset while preserving accuracy. We rethink two core notions underlying sample evaluation: representativeness and diversity. Instead of local geometric centrality, we define representativeness as coverage of dataset-level common or high-frequency feature factors. Instead of within-subset dispersion, we define diversity at the process level, requiring the selection trajectory to gradually include complementary rare factors over training. Based on this view, we propose a dynamic selection framework with three components. First, we score representativeness in a plug-in feature space to prioritize samples covering frequent factors. We instantiate this with a sparse autoencoder trained on the target dataset, using sparse unit activations to summarize both individual samples and dataset-wide factor statistics. Second, we realize process-level diversity by combining rare-factor sampling with a Usage-Frequency Penalty that promotes sample rotation, provably discourages monopoly, and reduces gradient bias. Third, we couple the two-dimensional scoring with a smooth scheduler that transitions selection from core-pattern consolidation to rare-factor exploration, without extra gradients, influence estimates, or second-order computations on the training model. Extensive experiments on five benchmarks across vision and text tasks demonstrate improved accuracy-efficiency trade-offs across models. Our method matches or exceeds full-data accuracy with over 2x training acceleration. Code will be released.

Yuzhe Zhou, Zhenglin Hua, Haiyun Guo +1

Southeast University, Nanjing, China · Chinese Academy of Sciences, Beijing, China

Diversity-Aware Sampling Data Selection Training Data Selection Data-Efficient Learning

Ask PDF

May 10, 2026

Let the Target Select for Itself: Data Selection via Target-Aligned Paths

Targeted data selection seeks training examples from a candidate pool that improve downstream task performance. While trajectory-based selectors effectively guide this process, existing approaches typically construct reference states by warming up on the candidate pool itself, thereby inheriting pool-dependent distributional biases. To address this coupling, we introduce Target-Aligned Candidate Selection (TACS), which constructs a short, capacity-constrained reference path using a compact target-validation proxy. Candidates are ranked by their normalized loss reduction along this trajectory, requiring only forward passes and allowing the path to be reused across distinct pools for a fixed model-target pair. Empirically, TACS achieves competitive downstream performance across controlled logistic, vision, and NLP benchmarks, while slashing selection compute by up to 69% and candidate storage to less than 10 MB in large-scale instruction tuning.

Huitao Yang, Hengzhi He, Tung Sum Thomas Kwok +1

University of California, Los Angeles

Data Selection Training Data Selection

Ask PDF

Abstract

Data Selection

Explore similar work

LESSER: Post-Training Data Selection with Output-Layer Gradients

Rethinking Representativeness and Diversity in Dynamic Data Selection

Let the Target Select for Itself: Data Selection via Target-Aligned Paths