Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.
Figures & tables
Symbol
Dimension
Description
W
Rdout×din
Weight matrix of a neural network layer
X
RN×L×din
Input activation matrix across samples
G
RN×dout×din
Gradient matrix, G[k,i,j]=∂W[i,j]∂Ek
S
Rdout×din
Importance score matrix (pruning metric)
M
{0,1}dout×din
Binary pruning mask
H
Rd×d
Hessian matrix, H=∂w2∂2E
Table 1
Figure 1: Illustration of our method GBLM-Pruner . Given a weight matrix, W , a gradient matrix, G , and an input feature activation, X , weight importance is computed as an element-wise multiplication of weight magnitude and ℓ1 or ℓ2 norm of the gradients across multiple samples, denoted as ∥G∥p⋅∣W∣ , optionally, it is promotable to add the multiplication of weight and the ℓ2 norm of input activations, denoted as ∣W∣⋅∥X∥2 .
Models
Method
BoolQ
RTE
HellaSwag
WinoGrande
ARC-e
OBQA
Mean
LLaMA-1-7B
Dense
75.11
66.43
76.21
69.85
72.81
44.40
67.47
Magnitude
54.65
54.15
60.90
59.43
54.38
35.20
53.12
SparseGPT
72.87
53.07
69.77
67.88
66.46
40.60
61.77
Wanda
71.25
54.87
70.12
66.06
65.11
39.60
61.17
Ours
73.43
59.93
70.29
67.40
65.99
41.40
63.07
LLaMA-1-13B
Dense
77.98
70.40
79.07
72.77
74.75
44.80
69.96
Table 1: Zero-Shot evaluation on 50 % unstructured sparsity pruned models for different benchmarks.
Comparison Group
Perplexity
layer
7.45
input,1
10.16
input,128
7.64
output,1
6.86
output,128
7.47
Table 2: Comparison group for GBLM-Pruner .
LLaMA-2
LLaMA-1
Method
Sparsity
7B
13B
70B
7B
13B
30B
None
0
5.47
4.88
3.32
5.68
5.09
4.10
Magnitude
0.5
16.03
6.83
5.36
17.29
20.21
7.54
SparseGPT
0.5
7.00
6.03
4.25
7.22
6.19
5.32
Wanda
0.5
6.92
5.97
4.22
7.26
6.15
5.24
GBLM-Pruner ℓ1
0.5
6.86
5.88
4.17
7.15
6.11
5.18
Table 3: WikiText perplexity of pruned LLaMA-1 and LLaMA-2 family of models.
Model
Methods
MMLU
GPQA-D
GPQA
MATH
GSM8K
Mean
(5-s)
(5-s)
(5-s)
(4-s)
(5-s)
Llama-3.1-8B
Base
65.20
24.24
27.68
8.54
51.55
35.44
SparseGPT
53.21
23.23
22.32
7.76
16.60
24.63
Wanda
49.34
24.18
23.66
7.92
13.57
23.73
Ours
50.16
25.25
25.00
8.54
14.86
24.76
Qwen3 8B
Base
74.95
19.70
28.57
25.70
87.34
47.25
Table 4: Performance comparison of base and pruned language models with 50% unstructured sparsity on STEM and general purpose reasoning benchmarks.
Model
Method
MBPP
HumanEval
Mean
(3-shot)
(0-shot)
Qwen3-8B
Base
65.60
62.80
64.20
SparseGPT
37.20
12.80
25.00
Wanda
43.40
26.22
34.81
Ours
44.00
28.05
36.02
Qwen3-14B
Base
71.60
55.49
63.54
Table 5: Performance comparison of base and pruned language models on coding benchmarks. All pruned models use 50% unstructured sparsity.
Model
Method
MMMU
MTVQA
Mean
Qwen2.5-VL-3B
Base
48.60
24.34
36.47
SparseGPT
30.33
19.66
25.00
Wanda
37.50
19.96
28.73
Ours
36.81
20.97
28.89
Qwen2.5-VL-7B
Base
49.55
29.59
39.57
SparseGPT
39.33
27.14
33.24
Table 6: Performance comparison of base and pruned vision-language models on multimodal benchmarks at 50% unstructured sparsity. Values are shown as percentages. Bold indicates best performance among pruning methods, underlined indicates second best.
Method
Sparsity
7B
13B
Magnitude
0.5
16.03
6.83
SparseGPT
0.5
7.00
6.03
Wanda
0.5
6.92
5.97
∣W∣⋅∥G∥1 (Ours)
0.5
7.17
6.15
∣W∣⋅∥G∥2 (Ours)
0.5
7.09
5.96
Table 7: Ablation on the pruning metrics.
Method
Sparsity
Perplexity
∣W∣⋅∣Gacc∣
0.5
119.72
∣W∣⋅∥G∥1
0.5
7.17
∣W∣⋅∥G∥2
0.5
7.09
(∣W∣⋅∥X∥2)2 + α⋅∣W∣⋅∥G∥1
0.5
6.90
(∣W∣⋅∥X∥2)2 + α⋅∣W∣⋅∥G∥2
0.5
6.88
(∣W∣⋅∥X∥2)2 - α⋅∣W∣⋅∥G∥1
0.5
9743.65
Table 8: Pruning metric on weight, gradient, activation.
Figure 2: Illustration of learned pruning pattern.
Model
Memory (GB)
Time (seconds)
LLaMA-7B
62
71.3
LLaMA-13B
96
102.1
Table 9: Computation requirements on an A100 GPU.
Method
LLaMA-7B
LLaMA-13B
SparseGPT
256,732.95
423,550.22
Wanda
257.04
455.39
GBLM-Pruner
403.76
561.30
Table 10: Pruning metric computation and weight removal time (milliseconds) on an A100 GPU.
LLaMA-1 layer
Dense
2:4
Speedup
q/k/v/o_proj
3.49
2.14
1.63 ×
up/gate_proj
9.82
6.10
1.61 ×
down_proj
9.92
6.45
1.54 ×
Table 11: Inference speedup for 2:4 sparsity on an A6000 GPU across transformer layers of LLaMA-1.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Sparsity
Wanda
Magnitude
Ours ℓ1
Ours ℓ2
0
75.40
75.40
75.40
75.40
0.5
64.54
59.48
64.64
64.86
0.6
43.65
29.98
44.15
44.23
0.7
7.92
1.88
8.89
8.02
0.8
0.20
0.18
0.32
0.24
Appendix
Table 12: ViT-B model pruning.
Scaling Factor, ( α )
Perplexity
0.001
6.920
0.01
6.919
0.1
6.921
1
6.912
10
6.890
100
6.858
Appendix
Table 13: Ablation of scaling factor.
Figure 3: Robustness to calibration samples.
Figure 4: Sparsity variation results for a large and a small model where we compare the performance of our method against other baseline methods.
Method
Sparsity
LLaMA-2-7B-chat
LLaMA-13B-chat
None
0
7.08
6.11
Magnitude
0.5
22.82
8.49
SparseGPT
0.5
8.66
7.26
Wanda
0.5
8.78
7.50
GBLM-Pruner ℓ2
0.5
8.52
7.27
GBLM-Pruner ℓ1
0.5
8.40
7.10
Appendix
Table 14: WikiText validation perplexity of different pruning methods for LLaMA 2 chat models.
Weight Update
Method
Datasplit
no
yes
Magnitude
Calib
18.14
12.93
Valid
17.29
12.55
Wanda
Calib
7.52
7.61
Valid
7.26
7.36
Ours
Calib
7.54
7.64
Appendix
Table 15: OBS weight update.
Method
Sparsity
Perplexity
Method
Sparsity
Perplexity
∣W∣⋅∣Gacc∣
0.5
119.72
(∣W∣⋅∥X∥2)2 + α⋅W⋅Gacc
0.5
7.04
∣W∣⋅∥G∥1
0.5
7.17
(∣W∣⋅∥X∥2)2 + α⋅W⋅∥G∥1
0.5
180490.19
∣W∣⋅∥G∥2
0.5
7.09
(∣W∣⋅∥X∥2)2 + α⋅W⋅∥G∥2
0.5
91781.49
∣W∣⋅∥X∥2⋅∣Gacc∣
0.5
69.59
(∣W∣⋅∥X∥2)2 - α⋅W⋅Gacc
0.5
7.14
∣W∣⋅∥X∥2⋅∥G∥1
0.5
7.31
(∣W∣⋅∥X∥2)2 - α⋅W⋅∥G∥1
0.5
246846.28
∣W∣⋅∥X∥2⋅∥G∥2
0.5
7.31
(∣W∣⋅∥X∥2)2 - α⋅W⋅∥G∥2
0.5
283620.75
Appendix
Table 16: Pruning metric.
Model
Sparsity
Wanda
GBLM-Pruner
Qwen3-14B
0.7
33.12
32.13
0.8
254.90
192.85
0.9
10902.73
5225.87
0.95
16872.77
11845.89
LLaMA-3.1-8B
0.7
113.59
111.48
0.8
1480.43
835.78
Appendix
Table 17: WikiText perplexity at higher sparsity levels. Bold indicates best performance.
Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning.
Ziyang Wang, Jiangfeng Xiao, Chuan Xiao +3
Beijing Institute of Technology, Zhuhai · Shenzhen University · Osaka University
Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. DIET does not require large costs from pre-computation or training. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET; for example, at 20% sparsity on Gemma-2 2B, DIET achieves near 10% average accuracy improvement, compared to previous state-of-the-art structured pruning methods. This advantage persists across various sparsity levels and model scales, positioning DIET as a practical and robust choice for structured LLM pruning.
Large Language Models (LLMs) have experienced significant growth and development in recent years. However, performing inference on LLMs remains costly, especially for long-context inference or in resource-constrained devices. This motivates the development of new post-training pruning (PTP) methods. These methods reduce LLMs' requirements by removing a substantial part of the model's parameters. The discarded weights are selected depending on their impact on the models performance. Current PTP methods prune the models by removing the less informative hidden nodes from the FFN layers, and the least important attention layers. We propose Putri, a PTP method that introduces three changes to the State-of-the-art. First, we update the un-pruned weights of the FFN to compensate for the introduced pruning error. Second, the FFN layers are pruned sequentially, taking into account the updates done to the previous layers. Third, instead of removing full attention layers, we remove individual attention-heads. We extend this method such that it can also address Grouped-Query Attention. In summary, Putri is a structure pruning method which remains simple while showing SOTA performance. Pruning experiments on multiple models with a wide variety of sparsity ranges and on different datasets, validate the generality of Putri. Notably, we demonstrate that, unlike previous methods, Putri can prune LLMs on extreme sparsity ratios. The code is available at: https://github.com/Coello-dev/Putri.
Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-Thieme
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1