Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity
Organizations: University of Bath · Aalto University · University of Texas at Austin · Google
Abstract
Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA2-7B on a single H200, where ELSA uses four, with smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost.
Figures & tables
| LLaMA-2-7B | LLaMA-2-13B | ||||||
| Sparsity | Method | Wiki | C4 | Avg. | Wiki | C4 | Avg. |
| 0% | Dense | 5.47 | 7.26 | 60.21 | 4.88 | 6.72 | 63.37 |
| 70% | Magnitude | 5.0e4 | 2.8e4 | 33.55 | 218.1 | 196.4 | 33.91 |
| Wanda | 77.08 | 81.51 | 35.29 | 45.19 | 55.31 | 38.44 | |
| SparseGPT | 27.77 | 31.37 | 41.70 | 19.65 | 23.25 | 45.94 | |
| ALPS | 18.88 | 20.18 | 45.38 | 14.13 | 16.35 | 49.44 | |
| Model | Method | 90% | 95% | 99% | |||
| Wiki | C4 | Wiki | C4 | Wiki | C4 | ||
| LLaMA-2-7B Dense: 5.47 / 7.26 | Wanda | 6477 | 5099 | 9558 | 7951 | 1.24e5 | 1.44e5 |
| SparseGPT | 1369 | 1033 | 5075 | 5628 | 1.51e4 | 9158 | |
| Wanda + LoRA | 43.67 | 34.33 | 75.52 | 48.41 | 249.6 | 91.25 | |
| Wanda + Full | 31.27 | 24.30 | 69.45 | 39.50 | 195.75 | 68.62 | |
| Wanda + Seft | 38.71 | 30.73 | 79.49 | 47.37 | 160.3 | 64.85 | |
| Gemma-2-27B | DeepSeek-67B | LLaMA-2-70B | ||||
| Method | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| Dense | 5.68 | 9.26 | 7.11 | 11.06 | 3.32 | 5.71 |
| Magnitude | 4.6e12 | 4.2e12 | 5.1e5 | 5.0e5 | 1.5e5 | 9.0e4 |
| Wanda | 1.5e4 | 1.1e4 | 1.3e6 | 1.2e6 | 5319.8 | 4375.0 |
| SparseGPT | 404.6 | 337.1 | 245.6 | 305.0 | 268.4 | 143.4 |
| Elsa -L | 28.47 | 29.27 | – | – | – | – |
| Perplexity ( ) | Resource Overhead | Memory (GiB) | ||||
| Method | Wiki | C4 | GPUs | Wall-clock (h) | GPU-h | Persistent State |
| Magnitude | 4.8e4 | 5.4e4 | 1 | 0.01 | 0.01 | 12.55 |
| Wanda | 6477 | 5099 | 1 | 0.01 | 0.01 | 12.55 |
| SparseGPT | 1369 | 1033 | 1 | 0.06 | 0.06 | 12.55 |
| ALPS | 232.2 | 174.5 | 1 | 0.92 | 0.92 | 4.04 |
| Wanda + Seft | 38.71 | 30.73 | 1 | 1.70 | 1.70 | 14.43 |
| 50% | 70% | 90% | ||||
| Route | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| Progressive ( Etaps ) | 6.66 | 8.20 | 10.99 | 10.87 | 20.98 | 17.39 |
| Direct projection | 6.54 | 8.23 | 11.47 | 11.34 | 34.17 | 23.14 |
| 50% | 70% | 90% | ||||
| Allocation | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| Increasing ( Etaps ) | 6.66 | 8.20 | 10.99 | 10.87 | 20.98 | 17.39 |
| Uniform | 6.68 | 8.24 | 11.18 | 11.05 | 23.19 | 18.63 |
| 50% | 70% | 90% | ||||
| Score Variant | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| Mean absolute gradient ( Etaps ) | 6.66 | 8.20 | 10.99 | 10.87 | 20.98 | 17.39 |
| Magnitude only | 6.71 | 8.56 | 12.06 | 12.56 | 32.64 | 24.52 |
| Absolute mean gradient | 9.31 | 10.68 | 17.55 | 15.66 | 33.22 | 25.05 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | ||
| Initial sparsity | 0 | 0 |
| Pruning boundaries | 8 | 32 |
| Optimizer steps | 1024 † | 4096 |
| Effective batch size | 32 † | 8 |
| Sparse-delta budget | LoRA rank- -equivalent over prunable modules | |
| Gradient-statistic accumulation | full adaptation interval | |
| Peak learning rate | Models |
| OPT-125M; LLaMA-2-7B; Gemma-2-27B; DeepSeek-67B; LLaMA-2-70B | |
| LLaMA-3.2-3B; LLaMA-2-13B | |
| OPT-1.3B; Gemma-2-2B |
| Model | |||||
| OPT-125M | 84,934,656 | 1,327,104 | 2,654,208 | 5,308,416 | 10,616,832 |
| OPT-1.3B | 1,207,959,552 | 7,077,888 | 14,155,776 | 28,311,552 | 56,623,104 |
| Gemma-2-2B | 2,024,275,968 | 10,383,360 | 20,766,720 | 41,533,440 | 83,066,880 |
| LLaMA-3.2-3B | 2,818,572,288 | 12,156,928 | 24,313,856 | 48,627,712 | 97,255,424 |
| LLaMA-2-7B | 6,476,005,376 | 19,988,480 | 39,976,960 | 79,953,920 | 159,907,840 |
| Perplexity ( ) | Tasks ( ) | ||||||||||
| Pattern | Method | Wiki | C4 | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | RTE | Winogrande | Avg. |
| – | Dense | 5.47 | 7.26 | 42.75 | 75.55 | 79.33 | 57.13 | 33.40 | 63.54 | 69.77 | 60.21 |
| MaskLLM | 7.33 | 9.13 | 35.24 | 69.23 | 71.47 | 51.28 | 28.40 | 57.76 | 64.48 | 53.98 | |
| Etaps | 7.50 | 9.41 | 33.62 | 67.59 | 71.83 | 50.40 | 28.20 | 59.57 | 64.96 | 53.74 | |
| Etaps | 7.01 | 8.89 | 33.79 | 67.68 | 74.43 | 51.70 | 28.40 | 65.34 | 66.06 | 55.34 | |
| Model | Allocation | 50% | 60% | 70% | 80% | 90% | |||||
| Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | ||
| LLaMA-3.2-3B | Uniform | 10.00 | 14.16 | 12.55 | 16.76 | 17.51 | 20.45 | 24.51 | 26.03 | 37.39 | 35.46 |
| Global | 10.14 | 13.97 | 12.85 | 16.61 | 17.12 | 19.93 | 23.82 | 25.48 | 36.01 | 35.15 | |
| LLaMA-2-7B | Uniform | 6.58 | 8.30 | 7.53 | 9.17 | 11.33 | 11.22 | 15.25 | 13.49 | 20.88 | 17.78 |
| Global | 6.66 | 8.20 | 7.69 | 9.04 | 10.99 | 10.87 | 14.17 | 13.09 | 20.98 | 17.39 | |
| 50% | 70% | 90% | |||||
| Model | Route | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| LLaMA-2-7B | Progressive ( Etaps ) | 6.66 | 8.20 | 10.99 | 10.87 | 20.98 | 17.39 |
| Direct projection | 6.54 | 8.23 | 11.47 | 11.34 | 34.17 | 23.14 | |
| Direct delta reallocation | 6.53 | 8.23 | 11.34 | 11.33 | 35.81 | 23.12 | |
| LLaMA-3.2-3B | Progressive ( Etaps ) | 10.14 | 13.97 | 17.12 | 19.93 | 36.01 | 35.15 |
| Direct projection | 10.67 | 14.73 | 19.84 | 22.39 | 60.40 | 49.97 | |
| 1 (one-shot) | 4 | 8 | 16 | 32 (default) | |
| Wiki | 34.17 | 22.04 | 22.02 | 21.91 | 20.98 |
| C4 | 23.14 | 18.12 | 17.68 | 17.45 | 17.39 |
| 0% | 25% | 50% | 70% | 90% | |
| Wiki | 20.98 | 21.13 | 20.61 | 22.48 | 34.17 |
| C4 | 17.39 | 17.27 | 17.18 | 17.45 | 23.14 |
| 50% | 70% | 90% | ||||
| Step allocation | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| Increasing ( Etaps ) | 10.14 | 13.97 | 17.12 | 19.93 | 36.01 | 35.15 |
| Uniform | 10.17 | 14.04 | 17.38 | 20.48 | 39.66 | 37.84 |
| 50% | 70% | 90% | |||||
| Model | Profile | Wiki | C4 | Wiki | C4 | Wiki | C4 |
| LLaMA-2-7B | Cubic ( Etaps ) | 6.66 | 8.20 | 10.99 | 10.87 | 20.98 | 17.39 |
| Geometric | 6.72 | 8.25 | 11.40 | 11.22 | 23.98 | 18.91 | |
| Linear | 6.65 | 8.27 | 12.14 | 11.52 | 29.64 | 23.15 | |
| LLaMA-3.2-3B | Cubic ( Etaps ) | 10.14 | 13.97 | 17.12 | 19.93 | 36.01 | 35.15 |
| Geometric | 10.18 | 14.14 | 18.09 | 21.01 | 40.41 | 38.32 | |
| Model | Score variant | 50% | 70% | 90% | |||
| Wiki | C4 | Wiki | C4 | Wiki | C4 | ||
| LLaMA-3.2-3B | Mean absolute gradient | 10.14 | 13.97 | 17.12 | 19.93 | 36.01 | 35.15 |
| Magnitude only | 11.20 | 15.56 | 22.87 | 24.57 | 58.04 | 47.77 | |
| Absolute mean gradient | 17.20 | 21.60 | 32.25 | 32.30 | 58.27 | 48.50 | |
| OPT-1.3B | Mean absolute gradient | 14.84 | 16.65 | 20.26 | 20.33 | 54.64 | 42.30 |
| Magnitude only | 17.62 | 17.70 | 24.50 | 22.97 | 118.31 | 69.91 | |
| Budget | 50% | 70% | 90% | 95% | 99% |
| (no adaptation) | 14.62 / 20.27 | 53.41 / 67.00 | 1783 / 1230 | 2.1e4 / 1.1e4 | 1.5e4 / 9710 |
| 10.38 / 14.20 | 17.73 / 20.93 | 42.08 / 41.23 | 62.32 / 56.26 | 122.2 / 95.04 | |
| 10.25 / 14.05 | 17.24 / 20.35 | 38.31 / 37.46 | 55.34 / 48.87 | 100.6 / 77.41 | |
| (default) | 10.14 / 13.97 | 17.12 / 19.93 | 36.01 / 35.15 | 47.63 / 43.01 | 86.28 / 66.78 |
| 10.10 / 13.82 | 16.73 / 19.54 | 32.95 / 31.82 | 43.40 / 38.87 | 77.07 / 60.13 |
| Budget | 50% | 70% | 90% | 95% | 99% |
| (no adaptation) | 8.31 / 10.60 | 22.52 / 25.92 | 463.2 / 349.5 | 5438 / 2243 | 8889 / 9852 |
| 6.62 / 8.20 | 10.19 / 10.73 | 20.70 / 18.80 | 31.29 / 25.46 | 69.39 / 44.15 | |
| 6.62 / 8.18 | 10.46 / 10.74 | 20.67 / 18.01 | 30.88 / 23.56 | 56.61 / 37.10 | |
| (default) | 6.66 / 8.20 | 10.99 / 10.87 | 20.98 / 17.39 | 28.56 / 22.12 | 46.13 / 33.22 |
| 6.70 / 8.25 | 12.14 / 11.14 | 21.75 / 16.98 | 27.71 / 20.81 | 42.29 / 29.74 |
| Model | Sparsity | Mean | Min | Median | Max | CV | Most / least preserved module | |
| OPT-125M | 50% | 50% | 18.5% | 50.8% | 75.0% | 0.23 | L0.fc1 | L3.fc2 |
| 70% | 30% | 8.4% | 28.0% | 62.2% | 0.35 | L0.fc1 | L3.fc2 | |
| 90% | 10% | 1.8% | 8.9% | 34.1% | 0.60 | L0.fc1 | L3.k_proj | |
| OPT-1.3B | 50% | 50% | 0.9% | 52.6% | 78.3% | 0.40 | L23.k_proj | L1.q_proj |
| 70% | 30% | 0.7% | 26.4% | 55.0% | 0.52 | L13.v_proj | L1.q_proj | |
| 90% | 10% | 0.4% | 8.3% | 37.9% | 0.63 | L0.fc1 | L1.k_proj | |
| Perplexity ( ) | Tasks ( ) | ||||||||||
| Sparsity | Method | Wiki | C4 | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | RTE | Winogrande | Avg. |
| 0% | Dense | 5.47 | 7.26 | 42.75 | 75.55 | 79.33 | 57.13 | 33.40 | 63.54 | 69.77 | 60.21 |
| 70% | Magnitude | 5.0e4 | 2.8e4 | 24.23 | 28.58 | 38.10 | 25.91 | 15.40 | 53.43 | 49.17 | 33.55 |
| Wanda | 77.08 | 81.51 | 18.43 | 31.02 | 53.61 | 28.04 | 12.60 | 52.71 | 50.59 | 35.29 | |
| SparseGPT | 27.77 | 31.37 | 21.25 | 44.61 | 63.73 | 32.95 | 18.20 | 53.07 | 58.09 | 41.70 | |
| ALPS | 18.88 | 20.18 | 24.74 | 53.32 | 64.34 | 37.85 | 21.80 | 53.43 | 62.19 | 45.38 | |
| Perplexity ( ) | Tasks ( ) | ||||||||||
| Sparsity | Method | Wiki | C4 | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | RTE | Winogrande | Avg. |
| 0% | Dense | 4.88 | 6.72 | 47.10 | 78.91 | 82.35 | 60.21 | 34.60 | 68.23 | 72.22 | 63.37 |
| 70% | Magnitude | 218.1 | 196.4 | 20.39 | 31.69 | 38.17 | 27.75 | 16.20 | 52.71 | 50.43 | 33.91 |
| Wanda | 45.19 | 55.31 | 18.52 | 38.47 | 62.23 | 29.51 | 13.80 | 52.71 | 53.83 | 38.44 | |
| SparseGPT | 19.65 | 23.25 | 26.79 | 54.00 | 66.73 | 36.91 | 20.60 | 52.71 | 63.85 | 45.94 | |
| ALPS | 14.13 | 16.35 | 29.10 | 59.68 | 71.25 | 41.11 | 25.20 | 54.15 | 65.59 | 49.44 | |
| Model | Method | 50% | 60% | 70% | 80% | 90% | |||||
| Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | ||
| OPT-125M Dense: 27.60 / 26.53 5M delta budget | Magnitude | 200.3 | 146.9 | 937.9 | 609.3 | 3855 | 2323 | 4882 | 3209 | 6513 | 4355 |
| Wanda | 38.99 | 34.85 | 75.51 | 62.19 | 344.4 | 253.8 | 1873 | 1007 | 4333 | 3113 | |
| SparseGPT | 36.98 | 33.55 | 58.76 | 48.92 | 233.6 | 161.3 | 2090 | 1142 | 6987 | 3418 | |
| ALPS | 32.73 | 30.92 | 43.13 | 37.13 | 89.29 | 65.68 | 493.65 | 265.35 | 2720 | 1142 | |
| Elsa | 36.44 | 31.55 | 39.42 | 33.32 | 47.56 | 38.83 | 62.63 | 46.42 | 95.92 | 61.68 | |
LLM Pruning
Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.
See all 191 papers