High-Fidelity Pruning for Large Language Models

Yijun Zhu; Jianxin Wang; Chengchao Shen

cs.CLMar 9, 2026

High-Fidelity Pruning for Large Language Models

Authors: Yijun Zhu, Jianxin Wang, Chengchao Shen

Organizations: Central South University

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance across a wide range of tasks, yet their significant computational and memory requirements present major challenges for deployment. A common approach uses Taylor expansion on the loss function to estimate neuron importance. However, its reliance on one-hot cross entropy loss, a key limitation is that it narrowly assesses importance based only on the probability assigned to the single predicted next token, thereby ignoring the other potential predictions of the original model. An intuitive solution to address this is to employ self distillation criterion for importance evaluation. However, this approach introduces significant computational overhead by requiring a separate teacher model for supervision. To this end, we propose a simple but effective criterion, information entropy of the model's output distribution, to efficiently evaluate importance scores of neurons with Taylor pruning without requirement of additional teacher. Compared to plain cross entropy criterion, it provides a more holistic criterion for Taylor pruning to prune neurons with the least impact on the prediction of model in a global manner, thereby preserving the fidelity of the model's predictive capabilities. Experimental results on extensive zero-shot benchmarks demonstrate that our method consistently outperforms existing pruning methods across the LLaMA and Qwen series models. The source code and trained weights are availabel at https://github.com/visresearch/HFPrune.

Figures & tables

Figure 1: One-hot label loss criterion vs our proposed information entropy criterion. The cross entropy criterion ( left ) measures the model prediction by only label-related prediction. Based on this criterion, Taylor-based pruning only minimizes the change of label-related prediction after pruning. In contrast, our proposed information entropy ( right ) fully represent holistic predictions, thus minimizing the change of global prediction distribution for better performance preservation.

Figure 2: The overview of our high fidelity prune method. We focus on the pruning of hidden neurons

h

of parameter-intensive MLP modules of LLM Transformer. To this end, we apply information entropy of model prediction to evaluate the importance scores of hidden neurons

h

based on Taylor expansion. Then, we rank the hidden neurons

h

by the obtained importance scores and prune the least important neurons, thus reducing model size while minimizing performance degradation.

Ratio	Method	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Wino	Average
0%	Llama-2-7B	45.1	73.8	79.4	67.4	44.2	78.7	40.1	46.5	38.8	69.3	58.3
20%	LLM-pruner	40.4	70.1	80.2	61.7	38.8	75.8	39.0	47.1	43.9	64.3	56.1
	LoRAPrune	41.6	71.0	81.7	58.7	41.4	76.7	40.4	44.0	65.9	65.9	56.7
	LoRAP	38.5	66.0	70.9	–	39.6	78.1	–	–	–	65.7	–
	SDMPrune	43.9	72.3	81.7	62.1	42.0	77.0	41.3	48.5	44.9	68.4	58.2
	HFPrune (Ours)	47.1	73.8	85.2	60.2	43.2	77.3	43.3	49.5	44.7	66.2	59.0

Table 1: Zero-shot performance on LLaMA-2-7B, which are finetuned on the LaMini dataset. The average accuracy is calculated among 10 zero-shot benchmarks. The “bolded” marked results denotes the best one under the same pruning ratio.

Table 2: Zero-shot performance on smaller LLMs, including LLaMA3.2-3.2B (left) and LLaMA3.2-1.2B (right) models. The settings are identical to the ones of LLaMA-2-7B. More detailed results on each benchmark are available in the Appendix.

Model	Ratio	Method	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Wino	Average
Qwen2.5-7B	0%	Original	51.1	77.5	84.7	64.0	46.8	79.8	41.4	54.8	56.4	73.2	63.0
	20%	SDMPrune	48.1	69.7	84.6	64.2	44.4	77.6	41.7	48.0	51.8	67.2	59.7
	20%	HFPrune(Ours)	52.1	74.5	84.6	62.9	45.4	78.1	41.2	48.1	52.0	64.8	60.4
	30%	SDMPrune	40.5	63.6	81.1	62.2	38.4	74.2	39.4	44.5	49.1	60.2	55.3
	30%	HFPrune(Ours)	48.2	70.3	83.2	60.4	41.6	76.1	40.7	48.9	48.2	62.8	58.0
Qwen2.5-1.5B	0%	Original	45.4	72.0	73.0	60.9	40.4	76.0	36.5	48.9	46.6	63.5	56.3

Table 3: Zero-shot performance on Qwen series models. The settings are identical to the ones of LLaMA-2-7B.

Model	#Params	Ratio	Avg zero-shot $\uparrow$	Prefill (ms) $\downarrow$	Speedup	Decoding (tokens/s) $\uparrow$
LLaMA2-7B	6.7B	0	58.3	84.6	1.00 $\times$	473.9
	5.4B	20%	59.0	65.5	1.29 $\times$	558.7 (+17.9%)
	4.7B	30%	56.3	57.5	1.47 $\times$	643.7 (+35.8%)

Table 4: Acceleration results of the structural pruned LLMs. Prefill Latency and decoding throughput were measured on a single NVIDIA A6000 GPU without employing any acceleration frameworks.

Model	Method	Elapsed Time (s)	Peak Memory (GB)
LLaMA3.2-1.2B	HFPrune (Ours)	150.7	7.5
LLaMA3.2-1.2B	SDMPrune	317.1	12.9
LLaMA3.2-3.2B	HFPrune (Ours)	305.8	17.3
LLaMA3.2-3.2B	SDMPrune	825.2	25.7
LLaMA2-7B	HFPrune (Ours)	508.9	35.3
LLaMA2-7B	SDMPrune	1539.8	51.2

Table 5: Efficiency comparison of the pruning process. The metrics are measured using 1000 randomly sampled sequences from the C4 dataset, each with a length of 1024 tokens. All models were run on four A6000 GPUs.

Ratio	Criterion	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Wino	Avg
20%	CE Loss	37.2	62.3	71.3	58.9	39.4	74.5	36.9	43.1	36.3	65.9	52.6
	SD Loss	34.2	63.6	67.1	59.6	38.4	75.0	37.4	44.0	36.6	62.8	51.9
	IE (ours)	37.0	63.4	71.8	60.0	40.0	74.9	37.7	43.7	37.0	65.0	53.1
30%	CE Loss	31.4	49.0	63.1	54.1	32.4	69.4	32.8	40.9	37.2	57.4	46.8
	SD Loss	28.2	50.3	49.0	53.9	32.6	69.3	33.6	41.5	37.3	56.4	45.2
	IE (ours)	30.1	49.4	65.8	54.6	33.2	70.2	33.2	41.7	37.5	57.1	47.3

Table 6: Zero-Shot Performance Comparison of Different Pruning Criteria on LLaMA-2-7B (No Fine-Tuning). Where IE (Information Entropy) is our proposed criterion, while SD (Self-Distillation) and CE (Cross-Entropy) represent the baseline loss criteria.

Ratio	Criterion	JS Distance (↓)	Top-15 Jaccard (↑)
20%	CE Loss	0.243	0.439
20%	Ours (IE)	0.241	0.445
30%	CE Loss	0.362	0.588
30%	Ours (IE)	0.353	0.595

Table 7: Comparison of pruning methods based on output distribution similarity.

Ratio	Method	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	Wino	Average
20%	attn&mlp w/o tune	40.0	69.0	65.1	60.0	40.0	76.0	32.5	41.0	55.0	53.2
	attn&mlp w/ tune	48.3	73.9	84.6	60.5	42.8	76.7	41.9	49.1	65.0	60.3
	mlp w/o tune	37.0	63.4	71.8	60.0	40.0	74.9	37.7	43.7	65.0	54.8
	mlp w/ tune	50.3	75.2	85.8	60.0	45.2	77.3	44.5	50.5	67.9	61.9
30%	attn&mlp w/o tune	33.0	57.0	56.0	54.0	33.4	70.0	24.2	38.1	52.3	46.4
	attn&mlp w/ tune	44.6	73.0	82.9	56.3	41.4	76.0	39.9	48.1	60.1	58.0

Table 8: Prune different parts on LLaMA-2-7B

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Ratio	$\rho_{mlp}$	Pack	lr	beta1	beta2	Weight decay	Precision	Batch size	Device
LLaMA3.2-1.2B	20%	0.70	yes	$4\times 10^{-4}$	0.9	0.999	0	BF16	256	4 $\times$ 4090
LLaMA3.2-1.2B	30%	0.55	yes	$4\times 10^{-4}$	0.9	0.999	0	BF16	256	4 $\times$ 4090
LLaMA3.2-3.2B	20%	0.70	yes	$4\times 10^{-4}$	0.9	0.999	0	BF16	256	4 $\times$ 4090
LLaMA3.2-3.2B	30%	0.55	yes	$4\times 10^{-4}$	0.9	0.999	0	BF16	256	4 $\times$ 4090
LLaMA2-7B	20%	0.70	yes	$2\times 10^{-4}$	0.9	0.999	0	BF16	512	8 $\times$ A6000
LLaMA2-7B	30%	0.54	yes	$2\times 10^{-4}$	0.9	0.999	0	BF16	512	8 $\times$ A6000

Appendix

Table 9: Implementation Details of the structural pruned LLMs.

Ratio	Method	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Wino	Average
0%	Llama-3.2-1.2B	37.1	60.7	64.0	62.6	37.6	74.2	37.6	42.9	37.7	60.4	51.5
20%	LLM-pruner	29.6	51.0	62.1	57.5	33.6	67.2	33.7	40.9	41.7	56.6	47.4
	Compresso	27.1	48.5	59.1	58.6	27.4	67.0	33.2	40.2	43.7	55.6	46.1
	LoRAPrune	29.6	51.0	66.2	54.4	29.6	66.4	34.1	42.0	41.7	56.9	47.2
	SDMPrune	31.1	55.2	67.6	57.8	32.4	70.3	35.4	42.7	40.2	56.6	48.9
	HFPrune(Ours)	36.0	60.9	72.8	55.4	33.0	71.3	36.9	43.6	41.3	56.6	50.8

Appendix

Table 10: Zero-shot performance of the compressed LLaMA-3.2-1B. The “bolded” represents the best result under the same pruning ratio.

Ratio	Method	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Average
0%	llama-3.2-3B	73.7	63.3	40.6	77.9	38.9	46.7	39.2	54.3
20%	LLMPruner	74.7	58.9	33.8	72.1	40.1	43.7	42.6	52.2
	LoRAPrune	68.8	57.9	33.1	71.4	35.3	42.9	42.6	50.3
	SDMPrune	75.3	59.3	36.2	73.6	38.9	44.6	45.8	53.4
	HFPrune(Ours)	79.8	58.8	37.8	74.6	39.0	46.0	42.6	54.1
30%	LLMPruner	66.2	54.9	32.8	66.2	37.1	41.3	42.3	48.7

Appendix

Table 11: Zero-shot performance of the compressed llama-3.2-3B. The “bolded” represents the best result under the same pruning ratio.

Model	Ratio	Method	ARCc	ARCe	BoolQ	Crows	OBQA	PIQA	Race	SiQA	TfQA	Wino	Average
Qwen2.5-7B	0%	Dense	51.1	77.5	84.7	64.0	46.8	79.8	41.4	54.8	56.4	73.2	63.0
	20%	SDMPrune	37.5	58.5	74.3	61.2	38.7	76.1	37.1	50.4	50.2	63.5	54.8
	20%	HFPrune(Ours)	39.9	66.9	68.2	61.8	39.8	77.5	37.2	48.2	46.1	60.1	54.6
	30%	SDMPrune	30.0	52.8	66.4	59.3	32.8	71.2	33.6	42.6	44.9	58.9	49.2
	30%	HFPrune(Ours)	34.0	58.3	61.9	58.3	35.2	74.1	35.2	44.3	43.1	56.2	50.1
Qwen2.5-1.5B	0%	Dense	45.4	72.0	73.0	60.9	40.4	76.0	36.5	48.9	46.6	63.5	56.3

Appendix

Table 12: Zero-shot performance of the compressed Qwen without any fine-tuning. The “bolded” represents the best result under the same pruning ratio.

Figure 3: Sample1: You are able to depend on Local Bathroom Remodel Crew to deliver the very best expert services when it comes to Bathroom Remodeling in Williamstown, NJ. Our crew of experienced experts will provide the expert services that you require with the most innovative technologies around. We make sure

Figure 4: Sample2: At Machu Picchu, be part of up With all the travellers as part of your group who hiked the common Inca Trail. If skies are very clear, get pleasure from a spectacular views more than The traditional metropolis from your Sunlight Gate, right before going on a guided stroll round the ruins. Mobile Massage Can Save The Environment! A more pure, not to say

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM Pruning LLM Pruning
Mohamed bin Zayed University of AI · Carnegie Mellon University
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
Mar 12, 2026Xiaoyun Liu, Divya Saxena, Jiannong Cao +3LLM Pruning Structured Pruning
Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sep 10, 2026Sankar Behera, Dhruv Singh, Anshika Agnihotri +3LLM Pruning LLM Compression
CSE, IIT Jammu · Mathematics, IIT Jammu · IT, EC Bikaner +1

Nov 8, 2023J/K move · Enter open · S save

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma+1

Mohamed bin Zayed University of AI · Carnegie Mellon University

LVLM Pruning LLM Pruning Neural Network Pruning LLM Compression Transformer Pruning

Open paper AskPDF arXiv

Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.

Abstract

LLM Pruning

Explore similar work

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry