Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.
Figures & tables
Figure 1: ZOIF framework and information flow across its three forward-only steps.
Peak GPU memory (GB)
LLM
FOIF
ZOIF
Mode
S1
dia
hvp
opt
S3
S1
dia
hvp
opt
S3
pythia-70m
0.68
0.94
0.86
2.70
0.92
0.63
0.90
0.63
1.72
0.51
Full
Llama-3.2-1B
3.91
3.94
3.96
10.67
3.91
2.96
2.97
2.97
3.00
2.95
LoRA
Llama-2-7B
15.74
15.56
15.62
34.39
15.46
13.08
13.15
13.15
13.27
13.02
LoRA
Qwen3-0.6B
3.09
3.11
3.13
15.77
3.10
1.92
1.94
1.94
1.98
1.91
LoRA
Table 1: Step-wise peak GPU memory and runtime for FOIF and ZOIF . S1 and S3 denote Steps 1 and 3; the dia , hvp , and opt columns report the corresponding Step 2 variants.
Method
MMLU
BBH
SQuAD
Avg.
ZOIF_dia
44.09
34.67
77.17
51.97 ± 0.32
ZOIF_hvp
43.89
34.38
74.70
50.99 ± 0.96
ZOIF_ide
44.50
34.32
76.30
51.71 ± 0.44
ZOIF_opt
44.41
34.81
73.42
50.88 ± 0.76
FOIF_dia
43.95
34.19
72.38
50.17 ± 1.02
FOIF_hvp
43.89
33.77
75.21
50.96 ± 0.65
Table 2: Held-out Qwen3-0.6B performance (Strategy 1). Avg. is the macro-average of MMLU/BBH accuracy and SQuAD F1. The Avg. entry is reported as mean ± standard deviation over three independent runs.
Gradient ascent or continual updating on the test set
Loss change before and after the test-side update
Candidate scoring only; proxy/test-side updates still use gradients
Z0-Inf ( Kokhlikyan et al., 2025 )
Loss values from intermediate training checkpoints
Candidate loss evolution along the checkpoint trajectory
Loss-only scoring along an existing training trajectory
For-Value ( Deng et al., 2026 )
Final-layer representations and token-level prediction errors
Closed-form alignment of representations and prediction errors
Forward features and errors used directly for valuation
ZOIF (ours)
Averaged proxy ZO gradient followed by one shared direction
Directional loss difference in Eq. ( 11 )
All three stages: ZO proxy gradient, ZO Step 2, and forward finite difference
Appendix
Table 4: Comparison of forward-oriented influence and data-valuation methods. The fourth column identifies the stage at which ZO or loss-only computation is used.
Dataset
∣Dproxy∣
∣Dtest∣
MMLU
1,531
14,042
BBH
649
5,862
SQuAD
1,174
10,570
Appendix
Table 5: Sample sizes for the proxy and test splits of each dataset.
LLM
FOIF
ZOIF
Notes
Step 1
Step 2
Step 3
Step 1
Step 2
Step 3
FOIF_dia
FOIF_hvp
FOIF_opt
ZOIF_dia
ZOIF_hvp
ZOIF_opt
pythia-70m
0.68
0.94
0.86
2.70
0.92
0.63
0.90
0.63
1.72
0.51
FULL
Llama-3.2-1B
3.91
3.94
3.96
10.67
3.91
2.96
2.97
2.97
3.00
2.95
LORA
Llama-2-7B
15.74
15.56
15.62
34.39
15.46
13.08
13.15
13.15
13.27
13.02
LORA
Qwen3-0.6B
3.09
3.11
3.13
15.77
3.10
1.92
1.94
1.94
1.98
1.91
LORA
Appendix
Table 6: Peak GPU memory usage (in GB) for FOIF and ZOIF across all steps.
LLM
FOIF
ZOIF
Notes
Step 1
Step 2
Step 3
Step 1
Step 2
Step 3
FOIF_dia
FOIF_hvp
FOIF_opt
ZOIF_dia
ZOIF_hvp
ZOIF_opt
pythia-70m
0.017
0.021
0.040
0.033
37.60
0.27
0.33
0.58
0.60
22.50
FULL
Llama-3.2-1B
0.058
0.083
0.175
0.050
34.00
0.80
0.92
2.28
0.80
30.00
LORA
Llama-2-7B
0.100
0.150
0.320
0.050
43.00
2.00
2.50
5.00
2.00
40.00
LORA
Qwen3-0.6B
0.100
0.150
0.310
0.050
40.00
1.50
1.70
3.50
1.00
31.75
LORA
Appendix
Table 7: Computation time (in hours) for FOIF and ZOIF across all steps.
Method
Memory (GB)
Time (h)
FOIF
3.91
0.058
ZOIF
2.96
0.800
FOIF+GC
3.21
0.125
FOIF+Offload
3.19
0.750
FOIF+Both
3.18
0.167
Appendix
Table 8: Step 1 memory and runtime on Llama-3.2-1B ( K=30 ).
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
23.45
7.24
0.64
10.44 ± 0.01
ZOIF_hvp
23.17
9.25
0.65
11.03 ± 0.00
ZOIF_ide
23.38
7.38
0.66
10.47 ± 0.01
ZOIF_opt
23.30
6.31
0.67
10.09 ± 0.01
FOIF_dia
23.29
7.03
0.59
10.30 ± 0.01
FOIF_hvp
23.60
7.09
0.54
10.41 ± 0.02
Appendix
Table 9: Held-out test performance on pythia-70m (Strategy 1). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
23.29
7.09
0.70
10.36 ± 0.02
ZOIF_hvp
23.29
7.45
0.64
10.46 ± 0.01
ZOIF_ide
23.35
8.59
0.55
10.83 ± 0.01
ZOIF_opt
23.55
6.51
0.57
10.21 ± 0.02
FOIF_dia
23.48
6.28
0.61
10.12 ± 0.01
FOIF_hvp
23.19
7.14
0.62
10.32 ± 0.03
Appendix
Table 10: Held-out test performance on pythia-70m (Strategy 2). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
34.36
30.81
63.21
42.79 ± 0.92
ZOIF_hvp
34.23
29.63
68.09
43.98 ± 0.65
ZOIF_ide
33.85
30.57
64.27
42.90 ± 0.71
ZOIF_opt
31.37
30.77
65.80
42.64 ± 1.59
FOIF_dia
30.59
29.92
63.31
41.27 ± 1.23
FOIF_hvp
31.11
30.93
59.50
40.51 ± 1.38
Appendix
Table 11: Held-out test performance on Llama-3.2-1B (Strategy 1). The average is the unweighted macro-average of MMLU accuracy, BBH accuracy, and SQuAD F1. The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
33.05
30.31
66.40
43.25 ± 1.04
ZOIF_hvp
33.17
31.15
58.45
40.92 ± 2.08
ZOIF_ide
33.05
30.34
63.17
42.19 ± 1.13
ZOIF_opt
31.33
31.26
65.08
42.55 ± 0.89
FOIF_dia
33.69
30.86
65.61
43.39 ± 1.73
FOIF_hvp
32.14
30.44
64.11
42.23 ± 1.19
Appendix
Table 12: Held-out test performance on Llama-3.2-1B (Strategy 2). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
25.29
35.15
79.70
46.71 ± 0.50
ZOIF_hvp
25.45
36.16
82.86
48.16 ± 0.77
ZOIF_ide
26.22
35.57
83.58
48.46 ± 1.00
ZOIF_opt
25.93
35.24
83.33
48.17 ± 0.61
FOIF_dia
25.53
36.31
82.84
48.23 ± 0.35
FOIF_hvp
25.36
36.84
82.50
48.24 ± 0.98
Appendix
Table 13: Held-out test performance on Llama-2-7B (Strategy 1). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
25.77
35.85
80.32
47.31 ± 0.32
ZOIF_hvp
25.51
34.89
83.18
47.86 ± 0.36
ZOIF_ide
25.06
35.02
69.44
43.17 ± 0.81
ZOIF_opt
25.60
35.16
82.49
47.75 ± 1.32
FOIF_dia
25.73
33.81
83.81
47.78 ± 0.46
FOIF_hvp
25.39
34.37
82.49
47.42 ± 0.11
Appendix
Table 14: Held-out test performance on Llama-2-7B (Strategy 2). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
43.32
35.06
75.55
51.31 ± 0.45
ZOIF_hvp
44.33
33.13
76.05
51.17 ± 0.81
ZOIF_ide
44.21
34.64
72.78
50.54 ± 0.94
ZOIF_opt
45.12
34.53
75.52
51.72 ± 0.65
FOIF_dia
43.04
34.26
73.91
50.41 ± 0.60
FOIF_hvp
44.35
33.96
70.88
49.73 ± 1.15
Appendix
Table 15: Held-out test performance on Qwen3-0.6B (Strategy 2). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
57.55
41.46
82.49
60.50 ± 0.37
ZOIF_hvp
57.38
41.85
80.16
59.80 ± 0.28
ZOIF_ide
57.68
42.75
81.08
60.50 ± 0.61
ZOIF_opt
57.36
41.78
83.20
60.78 ± 0.16
FOIF_dia
58.25
42.37
83.99
61.53 ± 0.59
FOIF_hvp
58.01
42.25
82.58
60.95 ± 1.06
Appendix
Table 16: Held-out test performance on Qwen3-1.7B (Strategy 1). The macro-average entry is reported as mean ± standard deviation over three independent runs.
Method
MMLU Acc.
BBH Acc.
SQuAD F1
Macro Avg.
ZOIF_dia
57.81
41.51
82.25
60.52 ± 0.40
ZOIF_hvp
57.60
41.08
83.60
60.76 ± 0.45
ZOIF_ide
57.15
41.04
81.81
60.00 ± 1.17
ZOIF_opt
57.84
41.34
82.52
60.57 ± 0.47
FOIF_dia
58.52
42.03
83.88
61.47 ± 0.80
FOIF_hvp
56.81
40.36
83.39
60.19 ± 0.61
Appendix
Table 17: Held-out test performance on Qwen3-1.7B (Strategy 2). The macro-average entry is reported as mean ± standard deviation over three independent runs.
α Sensitivity (fixed K=30 )
ZOIF Variant
α=0.001
α=0.01
α=0.1
ZOIF_dia
62.40
69.36
65.19
ZOIF_hvp
59.28
69.43
64.08
ZOIF_ide
61.95
67.06
55.95
ZOIF_opt
65.22
70.58
61.02
K Sensitivity (fixed α=0.001 )
Appendix
Table 18: Performance sensitivity to perturbation radius α (fixed K=30 ) and random directions K (fixed α=0.001 ) on Llama-3.2-1B (SQuAD).
Method
2.5%
5.0%
10.0%
15.0%
ZOIF_dia
60.66
62.40
73.18
69.69
ZOIF_hvp
61.06
59.28
58.61
60.66
ZOIF_ide
64.80
61.95
64.96
62.91
ZOIF_opt
62.50
65.22
69.08
65.39
Random
58.80
62.41
65.23
64.33
Appendix
Table 19: Sensitivity analysis of selection budget on Llama-3.2-1B (SQuAD, Strategy 2).
Method
Top- K Overlap
Spearman ρ∩
ZOIF_dia
16.44%
0.3277
ZOIF_hvp
19.74%
0.2725
ZOIF_ide
27.41%
0.2871
ZOIF_opt
19.72%
0.4096
Appendix
Table 20: Consistency analysis: ZOIF vs. FOIF ( K=10,000 ).
Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits a model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce Iprox, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's influence. Experimental results across diverse LLM families and evaluation tasks show that Iprox consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with Iprox achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, Iprox achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that Iprox provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.
The choice of post-training data for large language models substantially affects downstream performance. Gradient-based data selection is a popular approach that ranks training data by how well their gradients align with those of a small validation set. However, ranking with full-parameter gradients requires an expensive backward pass on every sample, making computation intractable for large candidate pools. This raises a natural question: can we approximate full-gradient features at a fraction of the cost? Conveniently, we find that output-layer gradients suffice for effective data selection, yet require only the cheaper forward pass. We implement this as LESSER, a drop-in wrapper for selection methods that reduces the feature-extraction FLOP cost by 9.7× for SFT and 3.0× for RL benchmarks, while tracking full-gradient performance on downstream tasks. Empirically, we find that even when output-layer and full gradients rank individual samples differently, they select batches with aligned gradients.
Lyuxin David Zhang, Eric Wong, Surbhi Goel +1
University of Pennsylvania · University of Texas at Austin
Influence functions approximate how removing a training example changes a quantity of interest, called the target function, such as a held-out loss. To estimate the influence of a group of examples, the standard practice is to sum the individual influences of its members. However, this sum does not capture how examples jointly affect the target: a pair of examples may be redundant or complementary, but the sum cannot distinguish these cases. We propose an interaction-aware influence function that characterizes how interactions between examples influence the target. By expanding the target to second order around the trained parameters, we obtain an estimator that augments the standard sum with a pairwise interaction term that captures the alignment between two examples' effects on the target. We empirically evaluate our estimator in two settings. First, on six dataset-model pairs spanning logistic regression, MLPs, and ResNet-9, our estimator tracks leave-group-out retraining substantially better than first-order influence across all settings. Second, when used as a greedy selection rule for instruction-tuning data on Llama-3.1-8B, it beats prior influence-based and representation-similarity baselines on five of seven downstream tasks, in a regime where standard influence-based selection underperforms random selection.