Non-uniform allocation can improve post-training pruning of large language models, yet quality can still degrade sharply at high sparsity. To control degradation as sparsity increases, we introduce LADDER (Layer-Wise Allocation via Damage-Driven Evaluation and Refinement), which allocates sparsity incrementally. At each step, it evaluates pruning an additional fraction of weights from each eligible layer and commits the layer update that causes the least end-to-end degradation per removed parameter. This process produces non-uniform sparsity profiles and exactly nested masks across multiple budgets. We evaluate LADDER on five OPT, Llama, and Qwen models across 10-90% sparsity, using SparseGPT as its candidate-generation backend and comparing against uniform and adaptive allocation baselines. Results reveal a clear operating-regime trade-off: LADDER is comparable or slightly weaker at benign sparsity but becomes substantially more robust from 70% onward. This high-sparsity advantage largely persists after posterior distillation across four of five models and translates into improved retention of aggregate zero-shot accuracy.
Figures & tables
Figure 1: Overview of LADDER. At each iteration, each eligible layer proposes a candidate pruning update using a fixed pruning step and a chosen local pruning backend. Each candidate is evaluated on calibration data, and the candidate with the least degradation per removed parameter is committed, producing an exactly nested mask trajectory. In our main experiments, the local backend is SparseGPT.
Figure 2: Pre-distillation relative loss advantage of LADDER over each baseline. The bars are the arithmetic mean of the relative advantages on WikiText-2, PTB, and C4. Positive values favor LADDER.
Figure 3: Language-modeling test loss before (solid) and after (dashed) posterior distillation on C4 and WikiText-2. Dataset panels are paired by model; the shared right axis gives the corresponding perplexity.
Figure 4: Distilled zero-shot accuracy across seven downstream tasks. Rows share task accuracy limits across models. Shaded regions show the task standard errors reported by the evaluation harness. Horizontal references mark dense and random performance.
Table 1: Pre-distillation language-modeling quality. Each entry reports test loss followed by perplexity in parentheses for each dataset. Lower is better, and bold indicates the best method for each model, dataset, and sparsity level.
Figure 5: Backend ablation on OPT-125M. For each local pruning rule, we compare the standalone method with LADDER allocation.
Figure 6: Initializer ablation on OPT-125M, averaged over WikiText-2, PTB, and C4. “Continued” applies LADDER from each baseline’s 70% mask. Dashed lines denote standard LADDER.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Shared protocol
Models
Pretrained OPT-125M, OPT-350M, OPT-1.3B, Qwen3-0.6B, and Llama-3.2-1B; no additional training before pruning.
Pruning
Unstructured transformer-block Linear weights; biases and output heads excluded; requested sparsity from 10% to 90% in 10-point increments.
Calibration and local pruning
128 C4 training sequences of 2,048 tokens, batch size 1, seed 0; SparseGPT block size 128 and mean-diagonal Hessian damping 0.01.
Evaluation
WikiText-2 and PTB test splits and C4 validation, using contiguous 2,048-token windows; seven zero-shot lm-eval tasks with batch size 1.
Posterior distillation
Dense teacher; logit KL; T=2 ; AdamW with learning rate 10−4 ; gradient clipping 1; five epochs over the nine fixed masks.
Appendix
Table 2: Pruning, evaluation, and ablation configurations.
Figure 7: Mean post-distillation zero-shot accuracy over seven tasks for Qwen3-0.6B, Llama-3.2-1B, and OPT-1.3B. Curves compare methods from 50% to 90% sparsity; solid horizontal lines mark dense accuracy, while the dotted line marks the seven-task mean random accuracy (35.7%). Shading denotes propagated lm-eval uncertainty.
Sparsity
Local rule
Base
LADDER
WT2
PTB
C4
WT2
PTB
C4
50%
Mag.
5.26
5.38
4.94
4.18
4.22
3.97
SparseGPT
3.65
3.89
3.53
3.59
3.87
3.50
Wanda
3.69
3.92
3.58
3.78
3.95
3.62
60%
Mag.
6.82
6.74
6.38
4.87
4.86
4.50
SparseGPT
4.21
4.38
3.96
4.02
4.33
3.79
Appendix
Table 3: Each local pruning rule used alone (Base) and within LADDER. Values are test loss; bold marks the better matched result.
Configuration
80%
90%
WT2
PTB
C4
WT2
PTB
C4
Standard LADDER
6.14
6.57
5.66
7.43
7.40
6.85
OWL (direct)
7.48
7.30
6.64
8.68
8.31
8.06
OWL → LADDER
6.33
6.41
5.65
7.57
7.52
6.91
DSA (direct)
7.70
7.49
6.89
8.56
8.30
7.78
DSA → LADDER
6.60
6.88
5.89
7.86
7.90
7.25
Appendix
Table 4: Dataset-level test loss for direct baselines, LADDER continuation from their 70% masks, and the standard dense-to-sparse LADDER trajectory.
Figure 8: Fraction of the pruning-induced test-loss gap removed by posterior distillation, 100(Lpre−Lpost)/(Lpre−Ldense) . Higher is better; zero indicates no recovery.
Table 5: Complete pre-distillation language-modeling results at 70–90% sparsity. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration.
Table 6: Complete post-distillation language-modeling results at 70–90% sparsity. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration.
Figure 9: Qwen3-0.6B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 10: Llama-3.2-1B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 11: OPT-1.3B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 12: OPT-350M zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 13: OPT-125M zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 14: Support across adjacent sparsity levels for the five evaluated models. Each bar partitions the weights kept at the higher sparsity into those retained from the next lower sparsity and those reintroduced after having been pruned.
Figure 15: Layer allocations selected by LADDER with SparseGPT. Each bar gives the percentage of prunable weights removed from a transformer block; overlaid bars show the allocations from 50% to 90% global sparsity.
One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to ±5% around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, p<0.001). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.
Semi-structured pruning compresses large language models (LLMs) while keeping a regular sparse structure, but the prevailing N:M pattern fixes the same local sparsity ratio in every layer. Layer-adaptive sparsity allocation improves unstructured pruning, yet it has been reported to be less effective under N:M sparsity, leaving open whether adaptive allocation is of limited value for semi-structured pruning in general or only under the fine-grained N:M pattern. We examine this question with group-level sparsity, which partitions each weight matrix into regular groups, retains or prunes each group as a whole, and allows each layer's sparsity ratio to vary under a global budget. We propose GroupMask, which generates the group selectors of all layers with a lightweight hypernetwork, relaxes them with a Gumbel-Sigmoid parameterization and a straight-through estimator, and learns them through sparsity-budget regularization and self-distillation while keeping the pretrained weights frozen. On LLaMA-2-7B at 50% sparsity with the same 1×256 group size, learned layer-adaptive allocation reduces WikiText-2 perplexity from 10.02 to 8.30 and raises the average zero-shot accuracy from 0.455 to 0.496 relative to a uniform per-layer ratio. GroupMask obtains the lowest WikiText-2 perplexity on LLaMA-2-7B and the highest average zero-shot accuracy with Alpaca calibration among the evaluated baselines on five LLaMA and Qwen models. Our code is available at https://github.com/ZhengaoLi/GroupMask.
Zhengao Li, Shuoqiu Li, Xiaofang Zhang +7
Florida State University · University of Connecticut · College of William & Mary +1
Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overheads, yet existing approaches face challenges: unstructured sparsity, where nonzeros can appear anywhere, preserves accuracy but yields irregular access patterns that prevent GPU acceleration, while semi-structured 2:4 sparsity is hardware-friendly but enforces a rigid 50% pattern that degrades model quality. To bridge this gap, we introduce PATCH, a hybrid sparsity framework that enables a continuous sparsity ratio between 0% and 50%. PATCH partitions weight matrices into tiles, assigning each tile to be either dense or 2:4 sparse via a learnable mask selection mechanism. This design provides fine-grained control over accuracy-acceleration tradeoffs and supports non-uniform sparsity across layers, leading to superior overall quality. Across models from 0.5B to 13B parameters, PATCH consistently narrows the gap to dense accuracy while delivering practical speedups. For instance, on LLaMA-2 7B with an A6000 GPU, PATCH achieves 1.18x-1.38x end-to-end speedup over dense baselines while improving accuracy by 0.37%-2.96% compared to the state-of-the-art 2:4 pruning method, MaskLLM.
Younes Hourri, Mohammad Mozaffari, Maryam Mehri Dehnavi
Department of Computer Science University of Toronto