ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

Lisong He; Yunxuan Jiang; Haishan Ye; Xiangyu Chang

cs.LGOct 11, 2026

ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

Authors: Lisong He, Yunxuan Jiang, Haishan Ye, Xiangyu Chang

Organizations: School of Management, Xi’an Jiaotong University, Xi’an, China

Abstract

Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.

Figures & tables

Figure 1: ZOIF framework and information flow across its three forward-only steps.

Peak GPU memory (GB)
LLM	FOIF					ZOIF					Mode
LLM	S1	dia	hvp	opt	S3	S1	dia	hvp	opt	S3	Mode
pythia-70m	0.68	0.94	0.86	2.70	0.92	0.63	0.90	0.63	1.72	0.51	Full
Llama-3.2-1B	3.91	3.94	3.96	10.67	3.91	2.96	2.97	2.97	3.00	2.95	LoRA
Llama-2-7B	15.74	15.56	15.62	34.39	15.46	13.08	13.15	13.15	13.27	13.02	LoRA
Qwen3-0.6B	3.09	3.11	3.13	15.77	3.10	1.92	1.94	1.94	1.98	1.91	LoRA

Table 1: Step-wise peak GPU memory and runtime for FOIF and ZOIF . S1 and S3 denote Steps 1 and 3; the dia , hvp , and opt columns report the corresponding Step 2 variants.

Method	MMLU	BBH	SQuAD	Avg.
ZOIF_dia	44.09	34.67	77.17	51.97 $\pm$ 0.32
ZOIF_hvp	43.89	34.38	74.70	50.99 $\pm$ 0.96
ZOIF_ide	44.50	34.32	76.30	51.71 $\pm$ 0.44
ZOIF_opt	44.41	34.81	73.42	50.88 $\pm$ 0.76
FOIF_dia	43.95	34.19	72.38	50.17 $\pm$ 1.02
FOIF_hvp	43.89	33.77	75.21	50.96 $\pm$ 0.65

Table 2: Held-out Qwen3-0.6B performance (Strategy 1). Avg. is the macro-average of MMLU/BBH accuracy and SQuAD F1. The Avg. entry is reported as mean

\pm

standard deviation over three independent runs.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Step	FOIF Cost	ZOIF Cost	Ratio (ZO/FO)
Step 1	$N_{\text{prox}}\cdot 3C_{\text{fwd}}$	$N_{\text{prox}}\cdot(K+1)C_{\text{fwd}}$	$\approx 10.3\times$
Step 2	$TB\cdot 3C_{\text{fwd}}$	$TB\cdot(K+1)C_{\text{fwd}}$	$\approx 10.3\times$
Step 3	$N_{\text{pool}}\cdot 3C_{\text{fwd}}$	$N_{\text{pool}}\cdot 2C_{\text{fwd}}$	$\approx 0.67\times$

Appendix

Table 3: Computational Complexity (FLOPs) Comparison (

K=30

)

Method	Proxy/test-side construction	Candidate-side scoring signal	Placement of ZO/loss-only computation
Forward-INF ( Ko et al., 2024 )	Gradient ascent or continual updating on the test set	Loss change before and after the test-side update	Candidate scoring only; proxy/test-side updates still use gradients
Z0-Inf ( Kokhlikyan et al., 2025 )	Loss values from intermediate training checkpoints	Candidate loss evolution along the checkpoint trajectory	Loss-only scoring along an existing training trajectory
For-Value ( Deng et al., 2026 )	Final-layer representations and token-level prediction errors	Closed-form alignment of representations and prediction errors	Forward features and errors used directly for valuation
ZOIF (ours)	Averaged proxy ZO gradient followed by one shared direction	Directional loss difference in Eq. ( 11 )	All three stages: ZO proxy gradient, ZO Step 2, and forward finite difference

Appendix

Table 4: Comparison of forward-oriented influence and data-valuation methods. The fourth column identifies the stage at which ZO or loss-only computation is used.

Dataset	$\|\mathcal{D}_{\text{proxy}}\|$	$\|\mathcal{D}_{\text{test}}\|$
MMLU	1,531	14,042
BBH	649	5,862
SQuAD	1,174	10,570

Appendix

Table 5: Sample sizes for the proxy and test splits of each dataset.

LLM	FOIF					ZOIF					Notes
	Step 1	Step 2			Step 3	Step 1	Step 2			Step 3
		FOIF_dia	FOIF_hvp	FOIF_opt			ZOIF_dia	ZOIF_hvp	ZOIF_opt
pythia-70m	0.68	0.94	0.86	2.70	0.92	0.63	0.90	0.63	1.72	0.51	FULL
Llama-3.2-1B	3.91	3.94	3.96	10.67	3.91	2.96	2.97	2.97	3.00	2.95	LORA
Llama-2-7B	15.74	15.56	15.62	34.39	15.46	13.08	13.15	13.15	13.27	13.02	LORA
Qwen3-0.6B	3.09	3.11	3.13	15.77	3.10	1.92	1.94	1.94	1.98	1.91	LORA

Appendix

Table 6: Peak GPU memory usage (in GB) for FOIF and ZOIF across all steps.

LLM	FOIF					ZOIF					Notes
	Step 1	Step 2			Step 3	Step 1	Step 2			Step 3
		FOIF_dia	FOIF_hvp	FOIF_opt			ZOIF_dia	ZOIF_hvp	ZOIF_opt
pythia-70m	0.017	0.021	0.040	0.033	37.60	0.27	0.33	0.58	0.60	22.50	FULL
Llama-3.2-1B	0.058	0.083	0.175	0.050	34.00	0.80	0.92	2.28	0.80	30.00	LORA
Llama-2-7B	0.100	0.150	0.320	0.050	43.00	2.00	2.50	5.00	2.00	40.00	LORA
Qwen3-0.6B	0.100	0.150	0.310	0.050	40.00	1.50	1.70	3.50	1.00	31.75	LORA

Appendix

Table 7: Computation time (in hours) for FOIF and ZOIF across all steps.

Method	Memory (GB)	Time (h)
FOIF	3.91	0.058
ZOIF	2.96	0.800
FOIF+GC	3.21	0.125
FOIF+Offload	3.19	0.750
FOIF+Both	3.18	0.167

Appendix

Table 8: Step 1 memory and runtime on Llama-3.2-1B (

K=30

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	23.45	7.24	0.64	10.44 $\pm$ 0.01
ZOIF_hvp	23.17	9.25	0.65	11.03 $\pm$ 0.00
ZOIF_ide	23.38	7.38	0.66	10.47 $\pm$ 0.01
ZOIF_opt	23.30	6.31	0.67	10.09 $\pm$ 0.01
FOIF_dia	23.29	7.03	0.59	10.30 $\pm$ 0.01
FOIF_hvp	23.60	7.09	0.54	10.41 $\pm$ 0.02

Appendix

Table 9: Held-out test performance on pythia-70m (Strategy 1). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	23.29	7.09	0.70	10.36 $\pm$ 0.02
ZOIF_hvp	23.29	7.45	0.64	10.46 $\pm$ 0.01
ZOIF_ide	23.35	8.59	0.55	10.83 $\pm$ 0.01
ZOIF_opt	23.55	6.51	0.57	10.21 $\pm$ 0.02
FOIF_dia	23.48	6.28	0.61	10.12 $\pm$ 0.01
FOIF_hvp	23.19	7.14	0.62	10.32 $\pm$ 0.03

Appendix

Table 10: Held-out test performance on pythia-70m (Strategy 2). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	34.36	30.81	63.21	42.79 $\pm$ 0.92
ZOIF_hvp	34.23	29.63	68.09	43.98 $\pm$ 0.65
ZOIF_ide	33.85	30.57	64.27	42.90 $\pm$ 0.71
ZOIF_opt	31.37	30.77	65.80	42.64 $\pm$ 1.59
FOIF_dia	30.59	29.92	63.31	41.27 $\pm$ 1.23
FOIF_hvp	31.11	30.93	59.50	40.51 $\pm$ 1.38

Appendix

Table 11: Held-out test performance on Llama-3.2-1B (Strategy 1). The average is the unweighted macro-average of MMLU accuracy, BBH accuracy, and SQuAD F1. The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	33.05	30.31	66.40	43.25 $\pm$ 1.04
ZOIF_hvp	33.17	31.15	58.45	40.92 $\pm$ 2.08
ZOIF_ide	33.05	30.34	63.17	42.19 $\pm$ 1.13
ZOIF_opt	31.33	31.26	65.08	42.55 $\pm$ 0.89
FOIF_dia	33.69	30.86	65.61	43.39 $\pm$ 1.73
FOIF_hvp	32.14	30.44	64.11	42.23 $\pm$ 1.19

Appendix

Table 12: Held-out test performance on Llama-3.2-1B (Strategy 2). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	25.29	35.15	79.70	46.71 $\pm$ 0.50
ZOIF_hvp	25.45	36.16	82.86	48.16 $\pm$ 0.77
ZOIF_ide	26.22	35.57	83.58	48.46 $\pm$ 1.00
ZOIF_opt	25.93	35.24	83.33	48.17 $\pm$ 0.61
FOIF_dia	25.53	36.31	82.84	48.23 $\pm$ 0.35
FOIF_hvp	25.36	36.84	82.50	48.24 $\pm$ 0.98

Appendix

Table 13: Held-out test performance on Llama-2-7B (Strategy 1). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	25.77	35.85	80.32	47.31 $\pm$ 0.32
ZOIF_hvp	25.51	34.89	83.18	47.86 $\pm$ 0.36
ZOIF_ide	25.06	35.02	69.44	43.17 $\pm$ 0.81
ZOIF_opt	25.60	35.16	82.49	47.75 $\pm$ 1.32
FOIF_dia	25.73	33.81	83.81	47.78 $\pm$ 0.46
FOIF_hvp	25.39	34.37	82.49	47.42 $\pm$ 0.11

Appendix

Table 14: Held-out test performance on Llama-2-7B (Strategy 2). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	43.32	35.06	75.55	51.31 $\pm$ 0.45
ZOIF_hvp	44.33	33.13	76.05	51.17 $\pm$ 0.81
ZOIF_ide	44.21	34.64	72.78	50.54 $\pm$ 0.94
ZOIF_opt	45.12	34.53	75.52	51.72 $\pm$ 0.65
FOIF_dia	43.04	34.26	73.91	50.41 $\pm$ 0.60
FOIF_hvp	44.35	33.96	70.88	49.73 $\pm$ 1.15

Appendix

Table 15: Held-out test performance on Qwen3-0.6B (Strategy 2). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	57.55	41.46	82.49	60.50 $\pm$ 0.37
ZOIF_hvp	57.38	41.85	80.16	59.80 $\pm$ 0.28
ZOIF_ide	57.68	42.75	81.08	60.50 $\pm$ 0.61
ZOIF_opt	57.36	41.78	83.20	60.78 $\pm$ 0.16
FOIF_dia	58.25	42.37	83.99	61.53 $\pm$ 0.59
FOIF_hvp	58.01	42.25	82.58	60.95 $\pm$ 1.06

Appendix

Table 16: Held-out test performance on Qwen3-1.7B (Strategy 1). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

Method	MMLU Acc.	BBH Acc.	SQuAD F1	Macro Avg.
ZOIF_dia	57.81	41.51	82.25	60.52 $\pm$ 0.40
ZOIF_hvp	57.60	41.08	83.60	60.76 $\pm$ 0.45
ZOIF_ide	57.15	41.04	81.81	60.00 $\pm$ 1.17
ZOIF_opt	57.84	41.34	82.52	60.57 $\pm$ 0.47
FOIF_dia	58.52	42.03	83.88	61.47 $\pm$ 0.80
FOIF_hvp	56.81	40.36	83.39	60.19 $\pm$ 0.61

Appendix

Table 17: Held-out test performance on Qwen3-1.7B (Strategy 2). The macro-average entry is reported as mean

\pm

standard deviation over three independent runs.

$\alpha$ Sensitivity (fixed $K=30$ )
ZOIF Variant	$\alpha=0.001$	$\alpha=0.01$	$\alpha=0.1$
ZOIF_dia	62.40	69.36	65.19
ZOIF_hvp	59.28	69.43	64.08
ZOIF_ide	61.95	67.06	55.95
ZOIF_opt	65.22	70.58	61.02
$K$ Sensitivity (fixed $\alpha=0.001$ )

Appendix

Table 18: Performance sensitivity to perturbation radius

\alpha

(fixed

K=30

) and random directions

K

(fixed

\alpha=0.001

) on Llama-3.2-1B (SQuAD).

Method	2.5%	5.0%	10.0%	15.0%
ZOIF_dia	60.66	62.40	73.18	69.69
ZOIF_hvp	61.06	59.28	58.61	60.66
ZOIF_ide	64.80	61.95	64.96	62.91
ZOIF_opt	62.50	65.22	69.08	65.39
Random	58.80	62.41	65.23	64.33

Appendix

Table 19: Sensitivity analysis of selection budget on Llama-3.2-1B (SQuAD, Strategy 2).

Method	Top- $K$ Overlap	Spearman $\rho_{\cap}$
ZOIF_dia	16.44%	0.3277
ZOIF_hvp	19.74%	0.2725
ZOIF_ide	27.41%	0.2871
ZOIF_opt	19.72%	0.4096

Appendix

Table 20: Consistency analysis: ZOIF vs. FOIF (

K=10{,}000

Influence Functions

Influence functions estimate how training examples or interventions change model parameters, predictions, and downstream behavior.

See all 40 papers

Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning
Feb 19, 2026Sirui Chen, Yunzhe Qi, Mengting Ai +4Training Data Selection LLM Fine-Tuning
University of Illinois Urbana-Champaign
LESSER: Post-Training Data Selection with Output-Layer Gradients
Oct 2, 2026Lyuxin David Zhang, Eric Wong, Surbhi Goel +1Training Data Selection Data Selection for Language Model Fine-Tuning
University of Pennsylvania · University of Texas at Austin
Interaction-Aware Influence Functions for Group Attribution
May 15, 2026Jaeseung Heo, Kyeongheung Yun, Youngbin Choi +3Instruction-Tuning Data Selection Influence Functions
GSAI, POSTECH · CSE, POSTECH

Feb 19, 2026J/K move · Enter open · S save

Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning

Sirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun+3

University of Illinois Urbana-Champaign

Training Data Selection LLM Fine-Tuning Instruction-Tuning Data Selection Influence Functions

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits a model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce Iprox, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's influence. Experimental results across diverse LLM families and evaluation tasks show that Iprox consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with Iprox achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, Iprox achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that Iprox provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.

Abstract

Influence Functions

Explore similar work

Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning

LESSER: Post-Training Data Selection with Output-Layer Gradients

Interaction-Aware Influence Functions for Group Attribution