Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits a model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce Iprox, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's influence. Experimental results across diverse LLM families and evaluation tasks show that Iprox consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with Iprox achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, Iprox achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that Iprox provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.
Figures & tables
Figure 1: For Qwen3-4B, a 1.5B IProX outperforms the Qwen3-1.7B off-the-shelf proxy, demonstrating that a smaller influence-pre-serving proxy can achieve better data selection performance.
Figure 2: Overview of IProX . In the first stage (left), IPSVD leverages hidden states and gradients to build second-moment matrices that reweight the model weights for proxy initialization. In the second stage (right), the proxy is further aligned with the target LLM through internal gradient alignment in the low-rank space and external logits anchoring for stability.
Figure 3: Loss and influence (TracIn) retention of SVD and our IPSVD under different compression sparsity.
Dataset
Task
Dtest
Dval
# Shots
Metric
TyDiQA
Multilingual QA
1,713
9
1
Exact Match
MMLU
Multiple choice
18,721
285
5
Accuracy
BBH
Reasoning
920
81
3
Accuracy
Table 1: Statistics of the evaluation datasets for fine-tuning.
Target Model
Proxy Model
#Params
MMLU
BBH
TyDiQA
Avg.
Llama3.2-3B
Llama3.2-3B
3B
56.28
47.78
43.10
49.05
Llama3.2-1B
1B
55.89
47.31
38.84
47.35
IProX , ρ=0.3
2.5B
56.77
49.16
40.98
48.97
IProX , ρ=0.5
1.8B
56.35
47.69
39.77
47.94
IProX , ρ=0.7
1.3B
56.28
47.31
39.04
47.54
Gemma3-4B
Gemma3-4B
4B
59.67
47.68
28.14
45.16
Table 2: IProX compared with off-the-shelf proxies across four target model families. For each target model, we report results using the full model (shown in gray, provided only as a reference), an off-the-shelf proxy from the same family, and IProX with different sparsity levels ρ . Bold and underline indicate the best and second-best proxy results, respectively.
Llama3.2-3B
Gemma3-4B
Task
Layer Extraction
Influence Scorer
IProX
Δ
Layer Extraction
Influence Scorer
IProX
Δ
MMLU
56.44
56.42
56.77
0.33
59.30
59.49
59.36
-0.13
BBH
46.85
46.57
49.16
2.31
48.79
47.87
49.63
0.84
TyDiQA
35.18
34.11
40.98
5.80
26.91
26.91
32.19
5.28
Avg.
46.16
45.70
48.97
2.81
45.00
44.76
47.06
1.99
Table 3: Comparison of IProX with two baselines: Layer Extraction and Influence Scorer . For IProX and Layer Extraction, we report the results based on ρ=0.3 . Δ denotes the performance gain of IProX over the strongest baseline. Bold indicates the best results.
Llama3.2-3B
Gemma3-4B
Task
IProX
IProX
Llama3.2-3B
Llama3.2-1B
ρ=0.3
ρ=0.5
ρ=0.7
Gemma3-4B
Gemma3-1B
ρ=0.3
ρ=0.5
ρ=0.7
MMLU
56.31
56.10
56.09
55.96
56.52
59.50
59.18
59.37
59.57
59.34
BBH
48.43
46.20
48.24
47.96
47.31
49.54
45.09
48.98
48.52
48.15
TyDiQA
41.88
38.13
44.35
41.57
39.05
32.48
30.01
34.18
33.94
28.44
Avg.
48.87
46.81
49.56
48.50
47.63
47.17
44.76
47.51
47.34
45.31
Table 4: Evaluation results of IProX on Influence Function. Bold and underline indicate the best and second-best results within each target group, respectively.
Figure 4: TFLOPs breakdown on Llama3.2-3B across different sparsity levels.
Model
Stage 1
Stage 2
Infl. Calc.
Llama3.2-3B
–
–
∼ 90 mins
Llama3.2-1B
–
–
∼ 40 mins
IProX
∼ 2 mins
∼ 3–5 mins
∼ 38–44 mins
Table 5: Computation breakdown on Llama3.2-3B measured in single GH200 GPU hours. Infl. Calc. denotes the time for influence calculation.
Proxy Model
MMLU
BBH
TyDiQA
SA ↑
1-NND ↑
SA ↑
1-NND ↑
SA ↑
1-NND ↑
Llama3.2-1B
29.01
13.91
20.94
13.29
18.61
13.13
IProX , ρ=0.3
33.39
14.04
21.78
15.67
24.59
15.26
IProX , ρ=0.5
33.14
14.31
21.32
15.45
20.59
16.17
IProX , ρ=0.7
32.19
16.07
21.32
15.63
19.72
15.82
Table 6: Similarity and diversity of selected subsets with the target model Llama3.2-3B. SA measures subspace alignment with the target task (higher is better), and 1-NND measures average nearest-neighbor distance within the selected dataset for diversity (higher is better). Bold and underline indicate the best and second-best proxy results.
Probe Size
MMLU
BBH
TyDiQA
Avg.
0.5×
56.12
46.85
37.71
46.89
Default
56.28
47.31
39.04
47.54
3×
56.26
47.41
39.89
47.85
5×
56.41
47.50
38.76
47.55
Table 7: Impact of Probe Set Size. Increasing probe size yields diminishing returns while significantly increasing computational overhead.
Diversity Setting
MMLU
BBH
TyDiQA
Avg.
Default (Random)
56.28
47.31
39.04
47.54
10% redundancy
56.28
47.22
38.76
47.42
20% redundancy
56.15
46.76
38.40
47.10
30% redundancy
56.12
45.65
37.67
46.48
Table 8: Impact of Probe Set Diversity. Reducing diversity (via redundancy injection) while keeping size fixed leads to performance degradation.
Model
MMLU
BBH
TyDiQA
IProX , ρ=0.3
56.77
49.16
40.98
w/o KL anchoring
56.52
48.88
40.85
w/o alignment
56.41
48.51
39.33
IProX , ρ=0.5
56.35
47.69
39.77
w/o KL anchoring
56.19
47.59
39.04
w/o alignment
56.08
47.03
36.43
Table 9: Ablation study on Llama3.2-3B. Removing KL anchoring or the entire aligning stage leads to consistent drops in performance across all tasks.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen3-4B
Qwen2-7B
Task
Layer Extraction
Influence Scorer
IProX
Δ
Layer Extraction
Influence Scorer
IProX
Δ
MMLU
69.86
69.60
70.15
0.29
70.31
70.28
70.41
0.10
BBH
74.25
74.90
75.18
0.28
59.63
59.17
60.93
1.30
TyDiQA
46.78
46.52
50.63
3.85
44.15
45.72
53.56
7.84
Avg.
63.63
63.67
65.32
1.47
58.03
58.39
61.63
3.08
Appendix
Table 10: Additional evaluation results of IProX on Influence Function. Bold and underline indicate the best and second-best results within each target group.
1% Data
20% Data
Task
IProX
IProX
Llama3.2-3B
Llama3.2-1B
ρ=0.3
ρ=0.5
ρ=0.7
Llama3.2-3B
Llama3.2-1B
ρ=0.3
ρ=0.5
ρ=0.7
MMLU
56.43
56.13
56.52
56.22
56.23
55.77
55.15
56.36
55.18
55.66
BBH
46.85
46.67
48.33
47.31
47.41
47.13
45.83
47.47
46.20
46.11
TyDiQA
34.37
32.39
36.79
35.27
33.32
40.73
36.55
38.20
37.49
36.63
Avg.
45.88
45.06
47.21
46.27
45.65
47.88
45.84
47.34
46.29
46.13
Appendix
Table 11: Evaluation results of IProX on different data budgets. Bold and underline indicate the best and second-best results within each target group.
Target Model
Proxy Model
MMLU
BBH
TyDiQA
Avg.
Llama3.2-3B
IProX , ρ=0.3
56.77
49.16
40.98
48.97
w/o IPSVD
56.42 (-0.35)
46.94 (-2.22)
36.53 (-4.45)
46.63 (-2.34)
IProX , ρ=0.5
56.35
47.69
39.77
47.94
w/o IPSVD
56.11 (-0.24)
46.30 (-1.39)
34.50 (-5.27)
45.64 (-2.30)
IProX , ρ=0.7
56.28
47.31
39.04
47.54
w/o IPSVD
55.97 (-0.31)
46.11 (-1.20)
32.73 (-6.31)
44.94 (-2.60)
Appendix
Table 12: Ablation of IPSVD vs. standard SVD on Llama3.2-3B. “w/o IPSVD” replaces IPSVD with standard SVD; numbers in parentheses denote the drop relative to IPSVD.
Candidate Training Data
Proxy Model
MMLU
BBH
TyDiQA
Avg.
CoT
Llama3.2-3B
56.53
48.61
47.90
51.01
Llama3.2-1B
56.17
47.31
42.67
48.72
IProX, ρ=0.3
56.96
48.80
48.72
51.49
IProX, ρ=0.5
56.48
48.06
46.73
50.42
IProX, ρ=0.7
56.26
47.60
43.18
49.01
BioInstruct
Llama3.2-3B
56.61
47.22
38.96
47.60
Appendix
Table 13: Performance on diverse candidate training data. IProX achieves competitive performance with the full model and outperforms the 1B proxy, with optimal results at ρ=0.3 .
Target Model
Method
MMLU
BBH
TyDiQA
Avg.
Llama3.2-3B
IProX
56.17 (+0.20)
46.57 (+0.64)
37.26 (+5.25)
46.67 (+2.03)
Layer Extraction
55.97
45.93
32.01
44.64
Qwen2-7B
IProX
70.25 (+0.21)
60.00 (+0.56)
48.67 (+6.02)
59.64 (+2.26)
Layer Extraction
70.04
59.44
42.65
57.38
Appendix
Table 14: Performance under Extreme Compression ( ρ=0.9 ). Even at 90% sparsity, IProX consistently outperforms the Layer Extraction baseline. Gains shown in parentheses.
Target Model
Configuration
MMLU
BBH
TyDiQA
Avg.
Llama3.2-3B
IProX, γKL=0.5
56.02
46.85
37.83
46.90
IProX, γKL=0.1
56.28
47.31
39.04
47.54
IProX, γKL=0.01
56.12
47.04
36.61
46.59
IProX, γKL=0.001
56.05
46.48
36.54
46.36
Appendix
Table 15: Sensitivity of KL Coefficient ( γKL ). A moderate coefficient ( γKL=0.1 ) strikes the best balance between influence alignment and output stability. Experiments performed on Llama3.2-3B with ρ=0.7 .
Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.
Lisong He, Yunxuan Jiang, Haishan Ye +1
School of Management, Xi’an Jiaotong University, Xi’an, China
Gradient-based data selection offers a principled framework for estimating sample utility in large language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the current optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
Fangxin Wang, Peyman Baghershahi, Langzhou He +3
Department of Computer Science, University of Illinois Chicago
As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories. Beyond static heuristic filtering, advanced data selection methods for LLM training largely follow two paradigms, each with fundamental limitations. Influence-based methods provide principled bi-level objectives but require intractable inverse-Hessian computations, while excess-loss methods are computationally efficient but rely on a static reference model that becomes misaligned with the evolving proxy model during training. We propose BLADE (Bi-Level Adaptive Data sElection), a Hessian-free framework for data selection. BLADE reformulates the bi-level optimization problem underlying influence-based methods as a penalized single-level objective via Lagrange multipliers, avoiding inverse-Hessian computation while revealing a principled connection to excess-loss based data selection. The resulting objective recovers an excess-loss form but replaces the static reference model with a dynamic one that stays synchronized with training. Theoretically, we prove that this penalized formulation guarantees first-order convergence. For efficient online batch selection, we instantiate BLADE as a memoryless randomized block-coordinate Frank-Wolfe algorithm. Extensive experiments show that BLADE consistently outperforms state-of-the-art data selection baselines, providing a practical recipe for LLM training.