Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning

Sirui Chen; Yunzhe Qi; Mengting Ai; Yifan Sun; Ruizhong Qiu; Jiaru Zou; Jingrui He

cs.LGFeb 19, 2026

Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning

Authors: Sirui Chen, Yunzhe Qi, Mengting Ai, Yifan Sun, Ruizhong Qiu, Jiaru Zou, Jingrui He

Organizations: University of Illinois Urbana-Champaign

Abstract

Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits a model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce Iprox, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's influence. Experimental results across diverse LLM families and evaluation tasks show that Iprox consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with Iprox achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, Iprox achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that Iprox provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.

Figures & tables

Figure 1: For Qwen3-4B, a 1.5B IProX outperforms the Qwen3-1.7B off-the-shelf proxy, demonstrating that a smaller influence-pre-serving proxy can achieve better data selection performance.

Figure 2: Overview of IProX . In the first stage (left), IPSVD leverages hidden states and gradients to build second-moment matrices that reweight the model weights for proxy initialization. In the second stage (right), the proxy is further aligned with the target LLM through internal gradient alignment in the low-rank space and external logits anchoring for stability.

Figure 3: Loss and influence (TracIn) retention of SVD and our IPSVD under different compression sparsity.

Dataset	Task	$\mathcal{D}_{\text{test}}$	$\mathcal{D}_{\text{val}}$	# Shots	Metric
TyDiQA	Multilingual QA	1,713	9	1	Exact Match
MMLU	Multiple choice	18,721	285	5	Accuracy
BBH	Reasoning	920	81	3	Accuracy

Table 1: Statistics of the evaluation datasets for fine-tuning.

Target Model	Proxy Model	#Params	MMLU	BBH	TyDiQA	Avg.
Llama3.2-3B	Llama3.2-3B	3B	56.28	47.78	43.10	49.05
	Llama3.2-1B	1B	55.89	47.31	38.84	47.35
	IProX , $\rho=0.3$	2.5B	56.77	49.16	40.98	48.97
	IProX , $\rho=0.5$	1.8B	56.35	47.69	39.77	47.94
	IProX , $\rho=0.7$	1.3B	56.28	47.31	39.04	47.54
Gemma3-4B	Gemma3-4B	4B	59.67	47.68	28.14	45.16

Table 2: IProX compared with off-the-shelf proxies across four target model families. For each target model, we report results using the full model (shown in gray, provided only as a reference), an off-the-shelf proxy from the same family, and IProX with different sparsity levels

\rho

. Bold and underline indicate the best and second-best proxy results, respectively.

	Llama3.2-3B				Gemma3-4B
Task	Layer Extraction	Influence Scorer	IProX	$\Delta$	Layer Extraction	Influence Scorer	IProX	$\Delta$
MMLU	56.44	56.42	56.77	0.33	59.30	59.49	59.36	-0.13
BBH	46.85	46.57	49.16	2.31	48.79	47.87	49.63	0.84
TyDiQA	35.18	34.11	40.98	5.80	26.91	26.91	32.19	5.28
Avg.	46.16	45.70	48.97	2.81	45.00	44.76	47.06	1.99

Table 3: Comparison of IProX with two baselines: Layer Extraction and Influence Scorer . For IProX and Layer Extraction, we report the results based on

\rho=0.3

\Delta

denotes the performance gain of IProX over the strongest baseline. Bold indicates the best results.

	Llama3.2-3B					Gemma3-4B
Task			IProX					IProX
	Llama3.2-3B	Llama3.2-1B	$\rho\!=\!0.3$	$\rho=0.5$	$\rho=0.7$	Gemma3-4B	Gemma3-1B	$\rho=0.3$	$\rho=0.5$	$\rho=0.7$
MMLU	56.31	56.10	56.09	55.96	56.52	59.50	59.18	59.37	59.57	59.34
BBH	48.43	46.20	48.24	47.96	47.31	49.54	45.09	48.98	48.52	48.15
TyDiQA	41.88	38.13	44.35	41.57	39.05	32.48	30.01	34.18	33.94	28.44
Avg.	48.87	46.81	49.56	48.50	47.63	47.17	44.76	47.51	47.34	45.31

Table 4: Evaluation results of IProX on Influence Function. Bold and underline indicate the best and second-best results within each target group, respectively.

Figure 4: TFLOPs breakdown on Llama3.2-3B across different sparsity levels.

Model	Stage 1	Stage 2	Infl. Calc.
Llama3.2-3B	–	–	$\sim$ 90 mins
Llama3.2-1B	–	–	$\sim$ 40 mins
IProX	$\sim$ 2 mins	$\sim$ 3–5 mins	$\sim$ 38–44 mins

Table 5: Computation breakdown on Llama3.2-3B measured in single GH200 GPU hours. Infl. Calc. denotes the time for influence calculation.

Proxy Model	MMLU		BBH		TyDiQA
	SA $\uparrow$	1-NND $\uparrow$	SA $\uparrow$	1-NND $\uparrow$	SA $\uparrow$	1-NND $\uparrow$
Llama3.2-1B	29.01	13.91	20.94	13.29	18.61	13.13
IProX , $\rho=0.3$	33.39	14.04	21.78	15.67	24.59	15.26
IProX , $\rho=0.5$	33.14	14.31	21.32	15.45	20.59	16.17
IProX , $\rho=0.7$	32.19	16.07	21.32	15.63	19.72	15.82

Table 6: Similarity and diversity of selected subsets with the target model Llama3.2-3B. SA measures subspace alignment with the target task (higher is better), and 1-NND measures average nearest-neighbor distance within the selected dataset for diversity (higher is better). Bold and underline indicate the best and second-best proxy results.

Probe Size	MMLU	BBH	TyDiQA	Avg.
$0.5\times$	56.12	46.85	37.71	46.89
Default	56.28	47.31	39.04	47.54
$3\times$	56.26	47.41	39.89	47.85
$5\times$	56.41	47.50	38.76	47.55

Table 7: Impact of Probe Set Size. Increasing probe size yields diminishing returns while significantly increasing computational overhead.

Diversity Setting	MMLU	BBH	TyDiQA	Avg.
Default (Random)	56.28	47.31	39.04	47.54
10% redundancy	56.28	47.22	38.76	47.42
20% redundancy	56.15	46.76	38.40	47.10
30% redundancy	56.12	45.65	37.67	46.48

Table 8: Impact of Probe Set Diversity. Reducing diversity (via redundancy injection) while keeping size fixed leads to performance degradation.

Model	MMLU	BBH	TyDiQA
IProX , $\rho=0.3$	56.77	49.16	40.98
w/o KL anchoring	56.52	48.88	40.85
w/o alignment	56.41	48.51	39.33
IProX , $\rho=0.5$	56.35	47.69	39.77
w/o KL anchoring	56.19	47.59	39.04
w/o alignment	56.08	47.03	36.43

Table 9: Ablation study on Llama3.2-3B. Removing KL anchoring or the entire aligning stage leads to consistent drops in performance across all tasks.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen3-4B				Qwen2-7B
Task	Layer Extraction	Influence Scorer	IProX	$\Delta$	Layer Extraction	Influence Scorer	IProX	$\Delta$
MMLU	69.86	69.60	70.15	0.29	70.31	70.28	70.41	0.10
BBH	74.25	74.90	75.18	0.28	59.63	59.17	60.93	1.30
TyDiQA	46.78	46.52	50.63	3.85	44.15	45.72	53.56	7.84
Avg.	63.63	63.67	65.32	1.47	58.03	58.39	61.63	3.08

Appendix

Table 10: Additional evaluation results of IProX on Influence Function. Bold and underline indicate the best and second-best results within each target group.

	1% Data					20% Data
Task			IProX					IProX
	Llama3.2-3B	Llama3.2-1B	$\rho\!=\!0.3$	$\rho=0.5$	$\rho=0.7$	Llama3.2-3B	Llama3.2-1B	$\rho=0.3$	$\rho=0.5$	$\rho=0.7$
MMLU	56.43	56.13	56.52	56.22	56.23	55.77	55.15	56.36	55.18	55.66
BBH	46.85	46.67	48.33	47.31	47.41	47.13	45.83	47.47	46.20	46.11
TyDiQA	34.37	32.39	36.79	35.27	33.32	40.73	36.55	38.20	37.49	36.63
Avg.	45.88	45.06	47.21	46.27	45.65	47.88	45.84	47.34	46.29	46.13

Appendix

Table 11: Evaluation results of IProX on different data budgets. Bold and underline indicate the best and second-best results within each target group.

Target Model	Proxy Model	MMLU	BBH	TyDiQA	Avg.
Llama3.2-3B	IProX , $\rho=0.3$	56.77	49.16	40.98	48.97
	w/o IPSVD	56.42 (-0.35)	46.94 (-2.22)	36.53 (-4.45)	46.63 (-2.34)
	IProX , $\rho=0.5$	56.35	47.69	39.77	47.94
	w/o IPSVD	56.11 (-0.24)	46.30 (-1.39)	34.50 (-5.27)	45.64 (-2.30)
	IProX , $\rho=0.7$	56.28	47.31	39.04	47.54
	w/o IPSVD	55.97 (-0.31)	46.11 (-1.20)	32.73 (-6.31)	44.94 (-2.60)

Appendix

Table 12: Ablation of IPSVD vs. standard SVD on Llama3.2-3B. “w/o IPSVD” replaces IPSVD with standard SVD; numbers in parentheses denote the drop relative to IPSVD.

Candidate Training Data	Proxy Model	MMLU	BBH	TyDiQA	Avg.
CoT	Llama3.2-3B	56.53	48.61	47.90	51.01
	Llama3.2-1B	56.17	47.31	42.67	48.72
	IProX, $\rho=0.3$	56.96	48.80	48.72	51.49
	IProX, $\rho=0.5$	56.48	48.06	46.73	50.42
	IProX, $\rho=0.7$	56.26	47.60	43.18	49.01
BioInstruct	Llama3.2-3B	56.61	47.22	38.96	47.60

Appendix

Table 13: Performance on diverse candidate training data. IProX achieves competitive performance with the full model and outperforms the 1B proxy, with optimal results at

\rho=0.3

Target Model	Method	MMLU	BBH	TyDiQA	Avg.
Llama3.2-3B	IProX	56.17 (+0.20)	46.57 (+0.64)	37.26 (+5.25)	46.67 (+2.03)
Llama3.2-3B	Layer Extraction	55.97	45.93	32.01	44.64
Qwen2-7B	IProX	70.25 (+0.21)	60.00 (+0.56)	48.67 (+6.02)	59.64 (+2.26)
Qwen2-7B	Layer Extraction	70.04	59.44	42.65	57.38

Appendix

Table 14: Performance under Extreme Compression (

\rho=0.9

). Even at 90% sparsity, IProX consistently outperforms the Layer Extraction baseline. Gains shown in parentheses.

Target Model	Configuration	MMLU	BBH	TyDiQA	Avg.
Llama3.2-3B	IProX, $\gamma_{\text{KL}}=0.5$	56.02	46.85	37.83	46.90
	IProX, $\gamma_{\text{KL}}=0.1$	56.28	47.31	39.04	47.54
	IProX, $\gamma_{\text{KL}}=0.01$	56.12	47.04	36.61	46.59
	IProX, $\gamma_{\text{KL}}=0.001$	56.05	46.48	36.54	46.36

Appendix

Table 15: Sensitivity of KL Coefficient (

\gamma_{\text{KL}}

). A moderate coefficient (

\gamma_{\text{KL}}=0.1

) strikes the best balance between influence alignment and output stability. Experiments performed on Llama3.2-3B with

\rho=0.7

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 184 papers

ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
Oct 11, 2026Lisong He, Yunxuan Jiang, Haishan Ye +1Influence Functions
School of Management, Xi’an Jiaotong University, Xi’an, China
Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
Mar 8, 2026Fangxin Wang, Peyman Baghershahi, Langzhou He +3Data Selection for Language Model Fine-Tuning Training Data Selection
Department of Computer Science, University of Illinois Chicago
BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
Jun 17, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Training Data Selection Bilevel Optimization
University of Oxford

Oct 11, 2026J/K move · Enter open · S save

ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

Lisong He, Yunxuan Jiang, Haishan Ye, Xiangyu Chang

School of Management, Xi’an Jiaotong University, Xi’an, China

Influence Functions

Open paper AskPDF arXiv

Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.

Abstract

Training Data Selection

Explore similar work

ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training