Sparsity Induction for Accurate Post-Training Pruning of Large Language Models

Minhao Jiang; Zhikai Li; Xuewen Liu; Jing Zhang; Mengjuan Chen; Qingyi Gu

cs.CLFeb 25, 2026

Sparsity Induction for Accurate Post-Training Pruning of Large Language Models

Authors: Minhao Jiang, Zhikai Li, Xuewen Liu, Jing Zhang, Mengjuan Chen, Qingyi Gu

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Abstract

Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our method further enhances sparsity-friendliness, achieving superior pruning performance over existing approaches.

Figures & tables

Figure 1: Top: Importance distributions before (left) and after (right) Sparsity Induction . SI sharpens the landscape by boosting salient channels and suppressing background mass, enlarging the gap between important and unimportant weights and yielding a more sparsity-friendly distribution. Bottom: We attach lightweight channel-wise scalers $s$ to Attention and FFN; backbone weights are frozen and only $s$ are trained. Dashed modules are merged back after training, introducing no extra modules or inference-time overhead.

		OPT				LLaMA-1
Sparsity	Method	125M	350M	1.3B	2.7B	7B	13B
0%	Dense	27.65	22.00	14.62	12.47	5.68	5.09
50%	Magnitude	193.36	97.78	1712.82	265.21	17.28	20.21
	+SI (Ours)	52.85	49.55	26.39	18.86	12.08	18.45
	Wanda	38.99	36.19	18.40	14.22	7.26	6.15
	+SI (Ours)	38.00	34.78	18.27	14.19	7.04	6.04

Table 1: Perplexity

\downarrow

on the WikiText-2 dataset for OPT and LLaMA-1 models. Bold indicates the lowest PPL within each (Sparsity, Model) block. Abbrev.: SI = Sparsity Induction ; rows marked “ +SI ” mean the base method + SI.

		LLaMA-1		LLaMA-2
Sparsity	Method	7B	13B	7B	13B
0%	Dense	54.79	57.82	55.95	58.25
50%	Magnitude	43.17	46.27	48.04	53.40
	+SI (Ours)	47.52	46.85	48.68	54.27
	Wanda	51.47	55.45	53.63	56.10
	+SI (Ours)	52.50	55.32	53.91	56.08

Table 2: Average zero-shot accuracy (%)

\uparrow

on ARC-Easy, ARC-Challenge, HellaSwag, BoolQ, Winogrande, and OpenBookQA; models are LLaMA-1/2 at 7B and 13B.

Method	Update time (s)	Avg. time/iter (s)	Speedup
Classical Recompute	345.91	2.70	$1.00\times$
Fast Update (Ours)	15.27	0.12	$\textbf{22.65}\times$

Table 3: Fast Hessian update vs. classical activation-recompute for refreshing the Wanda metric on LLaMA-7B in a epoch of 128 samples(batch size = 1).

Pattern	E2E latency (ms)	Speedup
Dense	312	$1.00\times$
$2{:}4$ (Wanda)	251	$1.24\times$
$2{:}4$ (Wanda+SI)	251	$1.24\times$

Table 4: End-to-end latency under

2{:}4

sparsity on LLaMA-7B. Our SI is fully absorbable and adds zero runtime latency compared with wanda.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

The Sparsity Whisperer
Aug 6, 2026Linghao Kong, Inimai Subramanian, Micah Adler +3LLM Pruning LLM Inference Acceleration
MIT · IST Austria · IBM +1
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
Nov 8, 2023Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun +2LVLM Pruning LLM Pruning
Mohamed bin Zayed University of AI · Carnegie Mellon University
Hidden in Plain Sight: The Overlooked Significance of Canonical Elements for Extreme LLM Sparsity
Sep 6, 2026Hyeondo Jang, Kwanhee Lee, Dongyeop Lee +1LLM Pruning LLM Compression
POSTECH

Aug 6, 2026J/K move · Enter open · S save

The Sparsity Whisperer

Linghao Kong, Inimai Subramanian, Micah Adler, Dan Alistarh+2

MIT · IST Austria · IBM+1

LLM Pruning LLM Inference Acceleration Neural Network Pruning

Open paper AskPDF arXiv

Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly sparsity-sensitive neurons in the MLP up and gate projections: separating similar inputs into dissimilar outputs. This suggests that effective pruning should preserve not only activations, but also the differences between outputs more broadly. We introduce a family of difference-informed pruning methods built upon this principle. Wisp is a first-order, update-free method that scores weights using input-difference norms, and Wisp+ refines this score neuronwise using the input pairs each neuron separates most strongly. Finally, Whisper is a second-order method that uses a lightly regularized difference Hessian as its reconstruction objective. Across Llama 2 and 3.1 models from 7B to 405B parameters, our second-order variant consistently improves over strong reconstruction-based baselines, while our update-free variants improve over activation-aware baselines, especially in constrained settings. The improvements over Wanda and SparseGPT extend to structured sparsity, downstream evaluations, and other model families. Augmenting stronger techniques such as RIA and ALPS with our difference-informed criteria yields further improvements, shifting the overall accuracy-runtime frontier outward at negligible additional cost. These results suggest that preserving output differences is a broadly useful and composable signal for post-training LLM sparsification.

Abstract

LLM Pruning

Explore similar work

The Sparsity Whisperer

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

Hidden in Plain Sight: The Overlooked Significance of Canonical Elements for Extreme LLM Sparsity