Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA2-7B on a single H200, where ELSA uses four, with 5.0× smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost.
Figures & tables
Figure 1: Cost–quality trade-off for sparse LLaMA-2-7B. Both panels report WikiText-2 perplexity.
LLaMA-2-7B
LLaMA-2-13B
Sparsity
Method
Wiki ↓
C4 ↓
Avg. ↑
Wiki ↓
C4 ↓
Avg. ↑
0%
Dense
5.47
7.26
60.21
4.88
6.72
63.37
70%
Magnitude
5.0e4
2.8e4
33.55
218.1
196.4
33.91
Wanda
77.08
81.51
35.29
45.19
55.31
38.44
SparseGPT
27.77
31.37
41.70
19.65
23.25
45.94
ALPS
18.88
20.18
45.38
14.13
16.35
49.44
Table 1: LLaMA-2-7B and LLaMA-2-13B at 70%–90% sparsity: perplexity ( ↓ ) and average zero-shot accuracy ( ↑ , %). The Elsa 13-B results are taken from ( Lee et al., 2026 ) .
Model
Method
90%
95%
99%
Wiki
C4
Wiki
C4
Wiki
C4
LLaMA-2-7B Dense: 5.47 / 7.26
Wanda
6477
5099
9558
7951
1.24e5
1.44e5
SparseGPT
1369
1033
5075
5628
1.51e4
9158
Wanda + LoRA
43.67
34.33
75.52
48.41
249.6
91.25
Wanda + Full
31.27
24.30
69.45
39.50
195.75
68.62
Wanda + Seft
38.71
30.73
79.49
47.37
160.3
64.85
Table 2: Extreme sparsity perplexity on LLaMA-2-7B.
Gemma-2-27B
DeepSeek-67B
LLaMA-2-70B
Method
Wiki
C4
Wiki
C4
Wiki
C4
Dense
5.68
9.26
7.11
11.06
3.32
5.71
Magnitude
4.6e12
4.2e12
5.1e5
5.0e5
1.5e5
9.0e4
Wanda
1.5e4
1.1e4
1.3e6
1.2e6
5319.8
4375.0
SparseGPT
404.6
337.1
245.6
305.0
268.4
143.4
Elsa -L
28.47
29.27
–
–
–
–
Table 3: Perplexity ( ↓ ) at 90% unstructured sparsity across 27B–70B parameter models. Elsa -L utilizes quantized optimizer and auxiliary state and is reproduced on Gemma-2-27B (its maximum reported scale); 67B and 70B settings are compared against scalable one-shot baselines.
Perplexity ( ↓ )
Resource Overhead
Memory (GiB)
Method
Wiki
C4
GPUs
Wall-clock (h)
GPU-h
Persistent State
Magnitude
4.8e4
5.4e4
1
< 0.01
< 0.01
12.55
Wanda
6477
5099
1
0.01
0.01
12.55
SparseGPT
1369
1033
1
0.06
0.06
12.55
ALPS
232.2
174.5
1
0.92
0.92
4.04
Wanda + Seft
38.71
30.73
1
1.70
1.70
14.43
Table 4: Resource overhead and perplexity ( ↓ ) for 90%-sparse LLaMA-2-7B on NVIDIA H200 NVL GPUs. Persistent state denotes aggregate batch-independent model, optimizer, auxiliary, and saliency memory (excluding activations).
50%
70%
90%
Route
Wiki
C4
Wiki
C4
Wiki
C4
Progressive ( Etaps )
6.66
8.20
10.99
10.87
20.98
17.39
Direct projection
6.54
8.23
11.47
11.34
34.17
23.14
Table 5: Direct versus progressive pruning on LLaMA-2-7B (perplexity ↓ ).
50%
70%
90%
Allocation
Wiki
C4
Wiki
C4
Wiki
C4
Increasing ( Etaps )
6.66
8.20
10.99
10.87
20.98
17.39
Uniform
6.68
8.24
11.18
11.05
23.19
18.63
Table 6: Adaptation-interval allocation on LLaMA-2-7B (perplexity ↓ ).
50%
70%
90%
Score Variant
Wiki
C4
Wiki
C4
Wiki
C4
Mean absolute gradient ( Etaps )
6.66
8.20
10.99
10.87
20.98
17.39
Magnitude only
6.71
8.56
12.06
12.56
32.64
24.52
Absolute mean gradient
9.31
10.68
17.55
15.66
33.22
25.05
Table 7: Saliency-score variants on LLaMA-2-7B (perplexity ↓ ).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Smax≤60%
Smax>60%
Initial sparsity S0
0
0
Pruning boundaries P
8
32
Optimizer steps T
1024 †
4096
Effective batch size
32 †
8
Sparse-delta budget dϕ
LoRA rank- 32 -equivalent over prunable modules
Gradient-statistic accumulation
full adaptation interval ki
Appendix
Table 8: Etaps optimization protocol by target-sparsity regime.
Table 10: Prunable parameter count dθ and nominal LoRA-equivalent sparse-delta budget dϕ(r) . A fixed r is not a fixed fraction of the model: for the default r=32 , dϕ/dθ decreases from 6.25% on OPT-125M to 0.61% on LLaMA-2-70B.
Perplexity ( ↓ )
Tasks ( ↑ )
Pattern
Method
Wiki
C4
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
RTE
Winogrande
Avg.
–
Dense
5.47
7.26
42.75
75.55
79.33
57.13
33.40
63.54
69.77
60.21
2:4
MaskLLM
7.33
9.13
35.24
69.23
71.47
51.28
28.40
57.76
64.48
53.98
2:4
Etaps
7.50
9.41
33.62
67.59
71.83
50.40
28.20
59.57
64.96
53.74
4:8
Etaps
7.01
8.89
33.79
67.68
74.43
51.70
28.40
65.34
66.06
55.34
Appendix
Table 11: LLaMA-2-7B semi-structured sparsity: perplexity and seven-task zero-shot accuracy.
Model
Allocation
50%
60%
70%
80%
90%
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
LLaMA-3.2-3B
Uniform
10.00
14.16
12.55
16.76
17.51
20.45
24.51
26.03
37.39
35.46
Global
10.14
13.97
12.85
16.61
17.12
19.93
23.82
25.48
36.01
35.15
LLaMA-2-7B
Uniform
6.58
8.30
7.53
9.17
11.33
11.22
15.25
13.49
20.88
17.78
Global
6.66
8.20
7.69
9.04
10.99
10.87
14.17
13.09
20.98
17.39
Appendix
Table 12: Global versus uniform module-wise sparsity allocation (perplexity ↓ ).
50%
70%
90%
Model
Route
Wiki
C4
Wiki
C4
Wiki
C4
LLaMA-2-7B
Progressive ( Etaps )
6.66
8.20
10.99
10.87
20.98
17.39
Direct projection
6.54
8.23
11.47
11.34
34.17
23.14
Direct + delta reallocation
6.53
8.23
11.34
11.33
35.81
23.12
LLaMA-3.2-3B
Progressive ( Etaps )
10.14
13.97
17.12
19.93
36.01
35.15
Direct projection
10.67
14.73
19.84
22.39
60.40
49.97
Appendix
Table 13: Direct versus progressive pruning across three models (perplexity ↓ ).
P
1 (one-shot)
4
8
16
32 (default)
Wiki
34.17
22.04
22.02
21.91
20.98
C4
23.14
18.12
17.68
17.45
17.39
S0
0%
25%
50%
70%
90%
Wiki
20.98
21.13
20.61
22.48
34.17
C4
17.39
17.27
17.18
17.45
23.14
Appendix
Table 14: Boundary-count and start-sparsity controls on LLaMA-2-7B at 90% (perplexity ↓ ).
50%
70%
90%
Step allocation
Wiki
C4
Wiki
C4
Wiki
C4
Increasing ( Etaps )
10.14
13.97
17.12
19.93
36.01
35.15
Uniform
10.17
14.04
17.38
20.48
39.66
37.84
Appendix
Table 15: Adaptation-interval allocation on LLaMA-3.2-3B (perplexity ↓ ).
50%
70%
90%
Model
Profile
Wiki
C4
Wiki
C4
Wiki
C4
LLaMA-2-7B
Cubic ( Etaps )
6.66
8.20
10.99
10.87
20.98
17.39
Geometric
6.72
8.25
11.40
11.22
23.98
18.91
Linear
6.65
8.27
12.14
11.52
29.64
23.15
LLaMA-3.2-3B
Cubic ( Etaps )
10.14
13.97
17.12
19.93
36.01
35.15
Geometric
10.18
14.14
18.09
21.01
40.41
38.32
Appendix
Table 16: Sparsity schedule at fixed step and boundary budgets (perplexity ↓ ).
Figure 2: Allocated delta slots (faded) and realized active capacity (solid, dϕ,i ) on LLaMA-2-7B; the gap corresponds to inactive capacity. Per-layer budgets Bℓ are fixed at initialization, so slots retire as the surviving pruning support contracts. (a) Evolution across pruning boundaries at 99% sparsity for three budgets. (b) Final active capacity at 90%, 95%, and 99%.
Model
Sparsity
Mean
Min
Median
Max
CV
Most / least preserved module
OPT-125M
50%
50%
18.5%
50.8%
75.0%
0.23
L0.fc1
L3.fc2
70%
30%
8.4%
28.0%
62.2%
0.35
L0.fc1
L3.fc2
90%
10%
1.8%
8.9%
34.1%
0.60
L0.fc1
L3.k_proj
OPT-1.3B
50%
50%
0.9%
52.6%
78.3%
0.40
L23.k_proj
L1.q_proj
70%
30%
0.7%
26.4%
55.0%
0.52
L13.v_proj
L1.q_proj
90%
10%
0.4%
8.3%
37.9%
0.63
L0.fc1
L1.k_proj
Appendix
Table 20: Final module-density statistics under global selection.
Perplexity ( ↓ )
Tasks ( ↑ )
Sparsity
Method
Wiki
C4
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
RTE
Winogrande
Avg.
0%
Dense
5.47
7.26
42.75
75.55
79.33
57.13
33.40
63.54
69.77
60.21
70%
Magnitude
5.0e4
2.8e4
24.23
28.58
38.10
25.91
15.40
53.43
49.17
33.55
Wanda
77.08
81.51
18.43
31.02
53.61
28.04
12.60
52.71
50.59
35.29
SparseGPT
27.77
31.37
21.25
44.61
63.73
32.95
18.20
53.07
58.09
41.70
ALPS
18.88
20.18
24.74
53.32
64.34
37.85
21.80
53.43
62.19
45.38
Appendix
Table 21: LLaMA-2-7B perplexities and task-wise results at 70%–90% sparsity.
Perplexity ( ↓ )
Tasks ( ↑ )
Sparsity
Method
Wiki
C4
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
RTE
Winogrande
Avg.
0%
Dense
4.88
6.72
47.10
78.91
82.35
60.21
34.60
68.23
72.22
63.37
70%
Magnitude
218.1
196.4
20.39
31.69
38.17
27.75
16.20
52.71
50.43
33.91
Wanda
45.19
55.31
18.52
38.47
62.23
29.51
13.80
52.71
53.83
38.44
SparseGPT
19.65
23.25
26.79
54.00
66.73
36.91
20.60
52.71
63.85
45.94
ALPS
14.13
16.35
29.10
59.68
71.25
41.11
25.20
54.15
65.59
49.44
Appendix
Table 22: LLaMA-2-13B perplexities and task-wise results at 70%–90% sparsity. Elsa resultes are taken from ( Lee et al., 2026 ) .
Model
Method
50%
60%
70%
80%
90%
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
OPT-125M Dense: 27.60 / 26.53 5M delta budget
Magnitude
200.3
146.9
937.9
609.3
3855
2323
4882
3209
6513
4355
Wanda
38.99
34.85
75.51
62.19
344.4
253.8
1873
1007
4333
3113
SparseGPT
36.98
33.55
58.76
48.92
233.6
161.3
2090
1142
6987
3418
ALPS
32.73
30.92
43.13
37.13
89.29
65.68
493.65
265.35
2720
1142
Elsa
36.44
31.55
39.42
33.32
47.56
38.83
62.63
46.42
95.92
61.68
Appendix
Table 23: Perplexity across model families at 50%–90% sparsity.
Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the pruning metric, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguingly, by incorporating gradients, unstructured pruning with our method tends to reveal some structural patterns, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-series, Qwen-Series and Deepseek models across zero-shot, STEM-Reasoning and Coding benchmarks show that GBLM-Pruner surpasses magnitude pruning and Wanda by significant margins and performs competitively against SparseGPT which involves intensive weight updates. We further extend our approach on Vision Transformer and Vision Language Models. Our code and models are available at https://github.com/VILA-Lab/GBLM-Pruner.
Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2
Mohamed bin Zayed University of AI · Carnegie Mellon University
Large language models (LLMs) are often considered fragile under aggressive sparsification, and maintaining reliable performance typically requires sticking to moderate sparsity levels. However, recent studies suggest that LLMs are more resilient to high sparsity than previously thought, reframing the problem as a design challenge rather than a fundamental limitation. In this work, we challenge the perceived limits of unstructured post-training LLM pruning by revisiting elementary pruning strategies that have remained relatively underexplored at this scale. Through a progressive sparsification framework with second-order saliency and continued training coordinated with sparsity progression, we show that pretrained LLMs can retain strong performance far beyond commonly studied sparsity regimes. Across LLaMA-2 and Qwen-3 model families, our approach improves perplexity and downstream accuracy up to 99% sparsity, surpassing both the current state-of-the-art and representative baselines. Precisely, on LLaMA-2-7B, our approach achieves WikiText-2 perplexities of 13.48 and 19.67 at 95% and 99% sparsity, respectively, while delivering 3.23× decoding speedup and 6.21× memory savings at 95% sparsity. Taken together, our results show that LLMs can be pushed into extreme sparsity while retaining strong performance, providing a foundation for further improving sparse models in this regime.
Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.