cs.LGDec 15, 2025

OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

Authors: Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi, Amir Yazdanbakhsh

Organizations: University of Toronto · Google DeepMind

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Post-training pruning cuts the inference cost of large language models, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are computationally infeasible at modern scale. One-shot methods such as SparseGPT sit in between and apply efficient but approximate weight updates. To close this gap, we introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability. OPTIMA casts layer-wise weight reconstruction after mask selection as independent, column-wise quadratic programs (QPs) that share one Hessian per weight matrix. Solving these QPs yields, at solver convergence, the per-column minimizer of the layer-wise reconstruction objective under the shared Hessian, and the shared Hessian makes the QPs easy to batch on accelerators. We implement an accelerator-friendly QP solver that accumulates one Hessian per weight matrix and solves many small QPs in parallel with a convergence certificate per column, enabling one-shot post-training pruning at scale on a single accelerator without fine-tuning. OPTIMA integrates with existing mask selectors and improves average zero-shot accuracy in 40 of 49 (model, mask selector, sparsity) settings across the LLaMA and Gemma families, with mean gains of +0.61, +0.56, and +1.01 points at 50% unstructured, 2:4, and 60% unstructured sparsity, and up to +3.97 points at 60% sparsity, while matching ADMM-based exact reconstruction on its own masks. On an NVIDIA H100, OPTIMA prunes an 8B-parameter transformer end-to-end in 40 hours and 60 GB of peak GPU memory. This shows that principled column-wise QP reconstruction is a practical alternative to heuristic weight updates in one-shot post-training pruning.

Figures & tables

Figure 1: Overview of OPTIMA . One Hessian H\mathbf{H} per weight matrix is estimated from the calibration data; after the mask selector fixes pruned entries, each column’s remaining weights are updated by a small QP with H\mathbf{H} , and the identically structured QPs are solved in parallel on an accelerator.
  \displaystyle\min_{{\color[rgb]{0.5,0,0.5}\Delta\mathbf{w}_{j}}}{\color[rgb]{0.5,0,0.5}\Delta\mathbf{w}_{j}^{T}}{\color[rgb]{0.5,0,0.5}\mathbf{H}_{\mathbf{W}}}{\color[rgb]{0.5,0,0.5}\Delta\mathbf{w}_{j}} 
  \displaystyle\text{s.t. }({\color[rgb]{0.5,0,0.5}\Delta\mathbf{w}_{j}})_{i}=-({\color[rgb]{0.5,0,0.5}\mathbf{w}_{j}})_{i},\;\forall i\in{\color[rgb]{0.5,0,0.5}\mathcal{S}_{j}} 
Algorithm 1 Layer-wise pruning with batched column-wise quadratic programming
Figure 2: Ratio of OPTIMA ’s layer-wise output error to that of the Wanda, SparseGPT, and Thanos updates on LLaMA 3.2 1B; values below 1 mean OPTIMA lowers the error, and values near 1 mark weight matrices where the fallback kept the selector’s update.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
LLaMA 3.1 8BDense–5.8463.5780.0981.4451.3773.4833.4063.89
Wanda–9.6447.7975.6872.5640.7070.0927.4055.70
WandaOPTIMA9.3748.8576.7173.8242.3270.3228.2056.70
SparseGPTSparseGPT9.3051.3276.1773.0241.3070.8829.4057.02
SparseGPTOPTIMA9.3349.3176.6174.2842.8370.8828.2057.02
Table 1: WikiText2 perplexity and zero-shot accuracy at 50% unstructured sparsity. OPTIMA improves the average accuracy in 12 of 15 (model, mask selector) pairs and ties one. Within each block, the better of the two Weight Update rows is bolded.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
LLaMA 3.1 8BDense–5.8463.5780.0981.4451.3773.4833.4063.89
Wanda–13.5443.4273.1869.2335.3267.3225.8052.38
WandaOPTIMA12.5845.4573.3969.5736.1868.9025.2053.12
SparseGPTSparseGPT12.3745.6273.8369.1535.8469.2225.6053.21
SparseGPTOPTIMA12.5446.0473.7269.9536.7769.6127.0053.85
Table 2: WikiText2 perplexity and zero-shot accuracy at 2:4 sparsity applied to the MLP layers only (38% to 41% end-to-end sparsity). OPTIMA improves the average accuracy in 17 of 19 (model, mask selector) pairs. Within each block, the better of the two Weight Update rows is bolded.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Calibration Samples128
Tokens per Sample2048
Dataset for CalibrationC4
Relative Tolerance (rAPDHG)0.01
Absolute Tolerance (rAPDHG)0.01
Maximum Iterations (rAPDHG)100,000
Appendix
Table 3: Key hyperparameters of OPTIMA and of the ADAM baseline of subsection G.1 (last two rows).
SparsityModelWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
50% unstructuredLLaMA 3.2 1B–25.3525.9765.4050.7222.8754.4621.2040.10
ADMM17.4227.0068.8855.5127.1356.2723.0042.97
OPTIMA17.4226.9369.5955.6827.1356.3522.4043.01
LLaMA 3.2 3B–14.0338.3772.7465.2434.1363.0625.4049.82
ADMM11.5843.1973.9467.4234.4766.6928.6052.38
Appendix
Table 4: Matched-protocol comparison with ADMM-based reconstruction ( Boža, 2024 ) . ADMM selects the mask; on that same mask and the same calibration data, the rows compare no weight update (–), ADMM’s own update, and OPTIMA . Within each block, the better perplexity and average accuracy of the two updates are bolded.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
LLaMA 3.1 8BDense–5.8463.5780.0981.4451.3773.4833.4063.89
Wanda–21.6531.9869.5361.1127.3061.0921.4045.40
WandaOPTIMA17.5633.9671.6063.7629.3566.0622.6047.89
SparseGPTSparseGPT15.4435.3271.5562.8831.6668.1924.2048.96
SparseGPTOPTIMA15.6432.4471.8763.9733.1167.5624.6048.93
Appendix
Table 5: Model perplexity on WikiText2 and accuracy on zero-shot downstream tasks for 60% unstructured sparsity. OPTIMA improves the average accuracy in 11 of 15 (model, mask selector) pairs. Within each (Model, Mask Selection) block, the better of the two Weight Update rows is bolded.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
Qwen 2.5 0.5BDense–13.0847.3669.9764.1829.1855.8024.4048.48
Wanda–24.0030.5264.0957.4124.0654.3819.8041.71
WandaOPTIMA22.7026.1464.5857.7925.2656.0422.0041.97
SparseGPTSparseGPT20.3329.3864.7456.5224.1556.2020.6041.93
SparseGPTOPTIMA19.5427.6865.1356.9924.6655.3320.6041.73
Appendix
Table 6: Qwen-2.5 family perplexity on WikiText2 and accuracy on zero-shot downstream tasks for 50% unstructured sparsity. OPTIMA improves the average accuracy in 11 of 15 (model size, mask selector) pairs. Within each (Model, Mask Selection) block, the better of the two Weight Update rows is bolded.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAverage
Qwen 2.5 0.5BDense–13.0847.3669.9764.1829.1855.8024.4048.48
Wanda–83.4223.0259.9643.8118.0950.2812.8034.66
WandaOPTIMA51.9723.1660.7246.2520.1451.7816.4036.41
SparseGPTSparseGPT40.5622.9061.5948.4021.2552.8016.8037.29
SparseGPTOPTIMA36.7723.0662.1348.7421.3353.9917.4037.77
Appendix
Table 7: Qwen-2.5 perplexity on WikiText2 and accuracy on zero-shot downstream tasks for 60% unstructured sparsity. OPTIMA improves the average accuracy in 11 of 15 (model size, mask selector) pairs. Within each (Model, Mask Selection) block, the better of the two Weight Update rows is bolded.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
Qwen 2.5 0.5BDense–13.0847.3669.9764.1829.1855.8024.4048.48
Wanda–41.3027.8861.7548.9923.3852.7214.0038.12
WandaOPTIMA27.6125.5763.7651.1822.1853.2815.8038.63
SparseGPTSparseGPT27.1524.8362.7949.4122.3552.3317.2038.15
SparseGPTOPTIMA25.7723.3862.9551.3022.9554.7017.0038.71
Appendix
Table 8: Qwen-2.5 perplexity on WikiText2 and accuracy on zero-shot downstream tasks for 2:4 sparsity. In this experiment, only the layers in the MLP part of the transformer are pruned, and the self-attention layers are dense, resulting in an end-to-end sparsity ratio of 38% to 41%. OPTIMA improves the average accuracy in 9 of 15 (model size, mask selector) pairs. Within each (Model, Mask Selection) block, the better of the two Weight Update rows is bolded.
Mask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
Wanda–23.47 ±\pm 0.1126.38 ±\pm 0.0664.97 ±\pm 0.1751.96 ±\pm 0.0223.74 ±\pm 0.0654.66 ±\pm 0.2318.04 ±\pm 0.0839.96 ±\pm 0.05
WandaOPTIMA18.77 ±\pm 0.0927.65 ±\pm 0.1367.11 ±\pm 0.2752.57 ±\pm 0.0524.74 ±\pm 0.1055.89 ±\pm 0.1320.16 ±\pm 0.0841.35 ±\pm 0.18
Appendix
Table 9: Mean ±\pm standard deviation over five independent draws of 128 C4 calibration samples for LLaMA 3.2 1B at 50% unstructured sparsity.
Figure 3: Sensitivity analysis for the number of calibration samples for different pruning methods.
ModelMask SelectionWeight UpdatePerplexityMetrics (%)
MMLUPIQAArc-EArc-CWinoOpenQAAvg.
Gemma 3 1BDense–14.1724.9574.8171.9335.4158.7228.8049.10
Wanda–32.9622.9767.1961.0326.3755.7220.0042.21
WandaADAM29.2523.1669.0462.7127.7357.4622.2043.72
WandaOPTIMA28.9023.9669.4862.8428.5856.8322.4044.01
SparseGPTSparseGPT28.3424.8568.8860.9426.6255.4921.4043.03
Appendix
Table 10: Comparison of OPTIMA with other optimizers without convergence guarantees (ADAM). ADAM can lead to suboptimal solutions (Gemma 3 1B) or divergence of the model (OPT 125M).
Mask / Weight UpdateLLaMA 3.2 1BLLaMA 3.2 3B
PerplexityAvg. (%)PerplexityAvg. (%)
Dense9.7549.097.8157.95
Wanda, no update23.5140.0112.9249.95
Wanda + OPTIMA18.8441.3312.2451.37
Wanda + ADAM (1)23.5039.9512.9249.96
Wanda + ADAM (2)22.8240.0212.7649.88
Appendix
Table 11: Recovery fine-tuning with ADAM on the 128 calibration samples, with the mask held fixed, for 1 to 128 iterations (in parentheses), at 50% unstructured sparsity. Avg. is the six-task average accuracy.
ModelMask10−210^{-2} (default)10−310^{-3}10−410^{-4}
LLaMA 3.2 1BWanda18.84 / 41.3318.63 / 41.5618.64 / 41.27
LLaMA 3.2 1BSparseGPT18.09 / 42.7217.91 / 42.2117.86 / 42.12
LLaMA 3.2 1BThanos18.77 / 41.9418.79 / 42.3018.38 / 42.54
Gemma 3 1BWanda28.90 / 44.0128.86 / 43.6029.07 / 43.64
Gemma 3 1BSparseGPT27.35 / 43.7627.05 / 43.4727.08 / 43.51
Gemma 3 1BThanos28.14 / 44.0527.97 / 43.6627.55 / 43.78
Appendix
Table 12: Solver-tolerance ablation at 50% unstructured sparsity. Each cell reports WikiText2 perplexity / six-task average accuracy (%) for the given relative and absolute rAPDHG tolerance.
SparsityMask SelectionWeight UpdatePerplexityAvg. (%)
Dense––2.8170.35
50% unstructuredWanda–5.7067.04
50% unstructuredWandaOPTIMA5.6867.30
60% unstructuredWanda–8.0460.74
60% unstructuredWandaOPTIMA8.2661.37
Appendix
Table 13: LLaMA-3.1-70B pruned with Wanda masks on a single NVIDIA H100 (about 350 hours end-to-end). Avg. is the six-task average accuracy.
ModelWanda (s)SparseGPT (s)OPTIMA (h)
LLaMA 3.2 1B15652.5
Gemma 2 2B211205.5
LLaMA 3.2 3B271557.0
LLaMA 3.1 8B5030840.0
LLaMA 3.1 70B––≈\approx 350
Appendix
Table 14: Mask-selection time of Wanda and SparseGPT versus OPTIMA ’s end-to-end pruning time on a single NVIDIA H100, under the identical protocol (128 calibration samples of 2048 tokens); mask selection is below 1% of OPTIMA ’s time.
More in this topic

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 172 papers

Explore similar work

CardsList
Mar 6, 2026

ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning

Pruning is widely recognized as an effective method for reducing the parameters of large language models (LLMs), potentially leading to more efficient deployment and inference. One classic and prominent path of LLM one-shot pruning is to leverage second-order gradients (i.e., Hessian), represented by the pioneering work SparseGPT. However, the predefined left-to-right pruning order in SparseGPT leads to suboptimal performance when the weights exhibit columnar patterns. This paper studies the effect of pruning order under the SparseGPT framework. The analyses lead us to propose ROSE, a reordered SparseGPT method that prioritizes weights with larger potential pruning errors to be pruned earlier. ROSE first performs pre-pruning to identify candidate weights for removal, and estimates both column and block pruning loss. Subsequently, two-level reordering is performed: columns within each block are reordered in descending order of column loss, while blocks are reordered based on block loss. We introduce the relative range of block loss as a metric to identify columnar layers, enabling adaptive reordering across the entire model. Substantial empirical results on prevalent LLMs (LLaMA2-7B/13B/70B, LLaMA3-8B, Mistral-7B) demonstrate that ROSE surpasses the original SparseGPT and other counterpart pruning methods. Our code is available at https://github.com/mingluo-su/ROSE.

Mingluo Su, Huan Wang

Westlake University

LLM PruningNeural Network PruningLLM CompressionTransformer Pruning
AskPDF
Apr 14, 2026

MOONSHOT : A Framework for Multi-Objective Pruning of Vision and Large Language Models

Weight pruning is a common technique for compressing large neural networks. We focus on the challenging post-training one-shot setting, where a pre-trained model is compressed without any retraining. Existing one-shot pruning methods typically optimize a single objective, such as a layer-wise reconstruction loss or a second-order Taylor approximation of the training loss. We highlight that neither objective alone is consistently the most effective across architectures and sparsity levels. Motivated by this insight, we propose MOONSHOT, a general and flexible framework that extends any single-objective pruning method into a multi-objective formulation by jointly optimizing both the layer-wise reconstruction error and second-order Taylor approximation of the training loss. MOONSHOT acts as a wrapper around existing pruning algorithms. To enable this integration while maintaining scalability to billion-parameter models, we propose modeling decisions and introduce an efficient procedure for computing the inverse Hessian, preserving the efficiency of state-of-the-art one-shot pruners. When combined with state-of-the-art pruning methods on Llama-3.2 and Llama-2 models, MOONSHOT reduces C4 perplexity by up to 32.6% at 2:4 sparsity and improves zero-shot mean accuracy across seven classification benchmarks by up to 4.9 points. On Vision Transformers, it improves accuracy on ImageNet-1k by over 5 points at 70% sparsity, and on ResNet-50, it yields a 4-point gain at 90% sparsity.

Gabriel Afriat, Xiang Meng, Shibal Ibrahim +2

Operations Research Center Massachusetts Institute of Technology · Google · OpenAI +1

LLM PruningNeural Network PruningTransformer Pruning
AskPDF
Jun 9, 2026

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Snowflake AI Research* · Seoul National University†

LLM PruningHigh-Performance ComputingStructured SparsityLLM Inference AccelerationNeural Network Pruning
AskPDF