cs.LGOct 20, 2026
SaveBuilding a Nested LADDER for LLM Unstructured Post-Training Pruning
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Abstract
Non-uniform allocation can improve post-training pruning of large language models, yet quality can still degrade sharply at high sparsity. To control degradation as sparsity increases, we introduce LADDER (Layer-Wise Allocation via Damage-Driven Evaluation and Refinement), which allocates sparsity incrementally. At each step, it evaluates pruning an additional fraction of weights from each eligible layer and commits the layer update that causes the least end-to-end degradation per removed parameter. This process produces non-uniform sparsity profiles and exactly nested masks across multiple budgets. We evaluate LADDER on five OPT, Llama, and Qwen models across 10-90% sparsity, using SparseGPT as its candidate-generation backend and comparing against uniform and adaptive allocation baselines. Results reveal a clear operating-regime trade-off: LADDER is comparable or slightly weaker at benign sparsity but becomes substantially more robust from 70% onward. This high-sparsity advantage largely persists after posterior distillation across four of five models and translates into improved retention of aggregate zero-shot accuracy.
Figures & tables
Figure 1: Overview of LADDER. At each iteration, each eligible layer proposes a candidate pruning update using a fixed pruning step and a chosen local pruning backend. Each candidate is evaluated on calibration data, and the candidate with the least degradation per removed parameter is committed, producing an exactly nested mask trajectory. In our main experiments, the local backend is SparseGPT.
Figure 2: Pre-distillation relative loss advantage of LADDER over each baseline. The bars are the arithmetic mean of the relative advantages on WikiText-2, PTB, and C4. Positive values favor LADDER.
Figure 3: Language-modeling test loss before (solid) and after (dashed) posterior distillation on C4 and WikiText-2. Dataset panels are paired by model; the shared right axis gives the corresponding perplexity.
Figure 4: Distilled zero-shot accuracy across seven downstream tasks. Rows share task accuracy limits across models. Shaded regions show the task standard errors reported by the evaluation harness. Horizontal references mark dense and random performance.
| Model | Method | 70% | 80% | 90% | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| WT2 | PTB | C4 | WT2 | PTB | C4 | WT2 | PTB | C4 | ||
| Qwen3-0.6B | LADDER | 4.34 (76.4) | 5.00 (147.7) | 4.40 (81.8) | 5.34 (208.7) | 6.26 (524.5) | 5.29 (197.6) | 8.83 (6849.8) | 10.26 (28430.2) | 7.74 (2292.0) |
| OWL | 5.59 (267.0) | 6.56 (707.6) | 5.60 (271.5) | 8.71 (6065.7) | 9.66 (15680.2) | 7.72 (2251.0) | 11.60 (109443.9) | 11.24 (75885.4) | 12.43 (250240.0) | |
| DSA | 5.15 (173.0) | 5.64 (280.5) | 5.17 (176.7) | 8.41 (4500.9) | 9.67 (15803.2) | 7.08 (1185.7) | 10.41 (33179.1) | 10.79 (48424.5) | 10.60 (40099.6) | |
| LSA | 5.20 (180.6) | 5.86 (349.8) | 5.13 (169.6) | 7.39 (1612.8) | 8.64 (5627.5) | 6.79 (887.2) | 12.06 (173103.3) | 12.27 (213937.1) | 12.08 (176410.1) | |
| Dense: WT2=2.54 (12.7), PTB=3.09 (21.9), C4=2.99 (19.9) | ||||||||||
Table 1: Pre-distillation language-modeling quality. Each entry reports test loss followed by perplexity in parentheses for each dataset. Lower is better, and bold indicates the best method for each model, dataset, and sparsity level.
Figure 5: Backend ablation on OPT-125M. For each local pruning rule, we compare the standalone method with LADDER allocation.
Figure 6: Initializer ablation on OPT-125M, averaged over WikiText-2, PTB, and C4. “Continued” applies LADDER from each baseline’s 70% mask. Dashed lines denote standard LADDER.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Shared protocol | |
|---|---|
| Models | Pretrained OPT-125M, OPT-350M, OPT-1.3B, Qwen3-0.6B, and Llama-3.2-1B; no additional training before pruning. |
| Pruning | Unstructured transformer-block Linear weights; biases and output heads excluded; requested sparsity from 10% to 90% in 10-point increments. |
| Calibration and local pruning | 128 C4 training sequences of 2,048 tokens, batch size 1, seed 0; SparseGPT block size 128 and mean-diagonal Hessian damping 0.01. |
| Evaluation | WikiText-2 and PTB test splits and C4 validation, using contiguous 2,048-token windows; seven zero-shot lm-eval tasks with batch size 1. |
| Posterior distillation | Dense teacher; logit KL; ; AdamW with learning rate ; gradient clipping 1; five epochs over the nine fixed masks. |
Table 2: Pruning, evaluation, and ablation configurations.
Figure 7: Mean post-distillation zero-shot accuracy over seven tasks for Qwen3-0.6B, Llama-3.2-1B, and OPT-1.3B. Curves compare methods from 50% to 90% sparsity; solid horizontal lines mark dense accuracy, while the dotted line marks the seven-task mean random accuracy (35.7%). Shading denotes propagated lm-eval uncertainty.
| Sparsity | Local rule | Base | LADDER | ||||
|---|---|---|---|---|---|---|---|
| WT2 | PTB | C4 | WT2 | PTB | C4 | ||
| 50% | Mag. | 5.26 | 5.38 | 4.94 | 4.18 | 4.22 | 3.97 |
| SparseGPT | 3.65 | 3.89 | 3.53 | 3.59 | 3.87 | 3.50 | |
| Wanda | 3.69 | 3.92 | 3.58 | 3.78 | 3.95 | 3.62 | |
| 60% | Mag. | 6.82 | 6.74 | 6.38 | 4.87 | 4.86 | 4.50 |
| SparseGPT | 4.21 | 4.38 | 3.96 | 4.02 | 4.33 | 3.79 | |
Table 3: Each local pruning rule used alone (Base) and within LADDER. Values are test loss; bold marks the better matched result.
| Configuration | 80% | 90% | ||||
|---|---|---|---|---|---|---|
| WT2 | PTB | C4 | WT2 | PTB | C4 | |
| Standard LADDER | 6.14 | 6.57 | 5.66 | 7.43 | 7.40 | 6.85 |
| OWL (direct) | 7.48 | 7.30 | 6.64 | 8.68 | 8.31 | 8.06 |
| OWL LADDER | 6.33 | 6.41 | 5.65 | 7.57 | 7.52 | 6.91 |
| DSA (direct) | 7.70 | 7.49 | 6.89 | 8.56 | 8.30 | 7.78 |
| DSA LADDER | 6.60 | 6.88 | 5.89 | 7.86 | 7.90 | 7.25 |
Table 4: Dataset-level test loss for direct baselines, LADDER continuation from their 70% masks, and the standard dense-to-sparse LADDER trajectory.
Figure 8: Fraction of the pruning-induced test-loss gap removed by posterior distillation, . Higher is better; zero indicates no recovery.
| Model | Method | 70% | 80% | 90% | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| WT2 | PTB | C4 | WT2 | PTB | C4 | WT2 | PTB | C4 | ||
| OPT-125M | LADDER | 4.79 (120.6) | 5.22 (184.6) | 4.48 (88.5) | 6.14 (465.4) | 6.57 (712.8) | 5.66 (286.6) | 7.43 (1693.2) | 7.40 (1629.2) | 6.85 (947.3) |
| OWL | 5.54 (254.3) | 5.49 (243.3) | 4.98 (145.2) | 7.48 (1772.1) | 7.30 (1473.4) | 6.64 (764.1) | 8.68 (5862.5) | 8.31 (4082.0) | 8.06 (3179.8) | |
| DSA | 5.58 (264.2) | 5.55 (256.5) | 5.10 (164.4) | 7.70 (2205.9) | 7.49 (1793.7) | 6.89 (983.1) | 8.56 (5236.4) | 8.30 (4029.8) | 7.78 (2403.5) | |
| LSA | 6.37 (581.6) | 6.58 (718.1) | 5.58 (264.7) | 8.24 (3798.7) | 7.92 (2739.3) | 7.47 (1760.6) | 8.92 (7447.4) | 8.62 (5535.1) | 8.47 (4765.3) | |
| Dense: WT2=3.32 (27.7), PTB=3.41 (30.2), C4=3.28 (26.6) | ||||||||||
Table 5: Complete pre-distillation language-modeling results at 70–90% sparsity. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration.
| Model | Method | 70% | 80% | 90% | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| WT2 | PTB | C4 | WT2 | PTB | C4 | WT2 | PTB | C4 | ||
| OPT-125M | LADDER | 4.52 (91.5) | 4.60 (100.0) | 4.13 (61.9) | 5.52 (250.6) | 5.62 (277.1) | 4.84 (126.3) | 7.00 (1099.1) | 6.99 (1087.3) | 5.96 (388.2) |
| OWL | 4.74 (114.0) | 4.74 (114.0) | 4.23 (68.9) | 6.23 (507.8) | 6.06 (427.6) | 5.30 (200.8) | 7.94 (2798.5) | 7.92 (2740.9) | 6.85 (939.9) | |
| DSA | 4.70 (109.5) | 4.69 (108.7) | 4.23 (68.6) | 6.21 (496.3) | 6.13 (461.1) | 5.32 (204.8) | 7.89 (2674.5) | 7.81 (2457.7) | 6.78 (882.0) | |
| LSA | 5.38 (217.5) | 5.36 (212.8) | 4.53 (93.0) | 6.64 (766.3) | 6.74 (845.2) | 5.64 (281.8) | 7.82 (2491.2) | 7.68 (2161.0) | 6.82 (914.9) | |
| Dense: WT2=3.32 (27.7), PTB=3.41 (30.2), C4=3.28 (26.6) | ||||||||||
Table 6: Complete post-distillation language-modeling results at 70–90% sparsity. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration.
Figure 9: Qwen3-0.6B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 10: Llama-3.2-1B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 11: OPT-1.3B zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 12: OPT-350M zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 13: OPT-125M zero-shot accuracy degradation before and after posterior distillation. Lower is better.
Figure 14: Support across adjacent sparsity levels for the five evaluated models. Each bar partitions the weights kept at the higher sparsity into those retained from the next lower sparsity and those reintroduced after having been pruned.
Figure 15: Layer allocations selected by LADDER with SparseGPT. Each bar gives the percentage of prunable weights removed from a transformer block; overlaid bars show the allocations from 50% to 90% global sparsity.
More in this topic
LLM Pruning
Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.
See all 178 papers