Deterministic Differentiable Structured Pruning for Large Language Models

Weiyu Huang; Pengle Zhang; Xiaolu Zhang; Jun Zhou; Jun Zhu; Jianfei Chen

cs.LGMar 9, 2026

Deterministic Differentiable Structured Pruning for Large Language Models

Authors: Weiyu Huang, Pengle Zhang, Xiaolu Zhang, Jun Zhou, Jun Zhu, Jianfei Chen

Organizations: Department of Computer Science and Technology, Tsinghua University, Beijing, China · Ant Group, Beijing, China

Abstract

Structured pruning reduces LLM inference cost by removing low-importance architectural components. This can be viewed as learning a multiplicative gate for each component under an l0 sparsity constraint. Due to the discreteness of the l0 norm, prior work typically adopts stochastic hard-concrete relaxations to enable differentiable optimization; however, this stochasticity can introduce a train--test mismatch when sampled masks are discretized for deployment and restricts masks to a bounded, near-binary range. To address this, we propose Deterministic Differentiable Pruning (DDP), a mask-only optimization method that eliminates stochasticity by directly optimizing a deterministic soft surrogate of the discrete l0 objective. Compared with prior approaches, DDP offers greater expressiveness, reduced train--test mismatch, and faster convergence. We apply our method to several dense and MoE models, including Qwen3-32B and Qwen3-30B-A3B, achieving a performance loss as small as 1% on downstream tasks while outperforming previous methods at 20% sparsity. We further demonstrate end-to-end inference speedups in realistic deployment settings with vLLM.

Figures & tables

Submodule	Attention	MLP
Masks	$\boldsymbol{m}^{\text{attn}}\in\mathbb{R}^{H}$ ( $\times L$ )	$\boldsymbol{m}^{\text{mlp}}\in\mathbb{R}^{C}$ ( $\times L$ )

Table 1: Mask variables for attention and MLP pruning.

Figure 1: Deterministic Differentiable Pruning overview. Left: Masked formulation for dense and MoE models. For dense models, we prune attention heads and MLP channels; for MoE models, we prune expert channels only. Right: Mask-only optimization with decoupled forward masks and retention scores for regularization, enabling deterministic training and an expanded mask range.

Figure 2: Deterministic surrogate mapping in DDP. Annealing $\mu$ sharpens the soft sigmoid projection, progressively approximating $\ell_{0}$ regularization.

Model	LLaMA-7B				LLaMA-2-7B				LLaMA-13B
Sparsity Ratio	20%		50%		20%		50%		20%		50%
Method	Wiki2 $\downarrow$	Mean Acc $\uparrow$	Wiki2 $\downarrow$	Mean Acc $\uparrow$	Wiki2 $\downarrow$	Mean Acc $\uparrow$	Wiki2 $\downarrow$	Mean Acc $\uparrow$	Wiki2 $\downarrow$	Mean Acc $\uparrow$	Wiki2 $\downarrow$	Mean Acc $\uparrow$
Original (Dense)	12.62	65.96	12.62	65.96	12.18	66.63	12.18	66.63	11.58	68.79	11.58	68.79
LoRAPrune	16.80	60.05	30.12	49.71	–	–	–	–	–	–	–	–
LoRAP	16.35	61.70	30.90	52.17	14.67	61.20	26.26	52.31	13.58	64.71	22.66	58.20
SlimLLM	15.55	62.41	26.71	53.16	15.28	61.70	27.29	52.02	13.35	64.21	25.64	54.75

Table 2: Performance of pruned dense LLMs under different methods. All baseline methods are evaluated after fine-tuning. Bold indicates the best performance at the same pruning ratio.

Model	DeepSeekMoE-16B-Base						Qwen3-30B-A3B
Sparsity Ratio	20%		40%		60%		20%		40%		60%
Method	C4 $\downarrow$	Mean Acc $\uparrow$	C4 $\downarrow$	Mean Acc $\uparrow$	C4 $\downarrow$	Mean Acc $\uparrow$	C4 $\downarrow$	Mean Acc $\uparrow$	C4 $\downarrow$	Mean Acc $\uparrow$	C4 $\downarrow$	Mean Acc $\uparrow$
Original (Dense)	9.05	62.28	9.05	62.28	9.05	62.28	12.13	71.08	12.13	71.08	12.13	71.08
NAEE	10.07	60.51	12.80	54.94	29.44	45.28	12.44	69.94	13.87	65.77	19.37	57.13
$\text{D}^{2}$ -MoE	12.62	58.97	17.22	54.32	34.54	46.72	18.52	66.35	35.48	62.77	68.36	52.02
Camera-P	9.84	61.03	11.68	58.58	18.10	51.62	12.25	69.94	15.58	67.46	24.48	59.03

Table 3: Performance of pruned Mixture-of-Experts LLMs under different methods. Bold indicates the best performance at the same pruning ratio.

	LLaMA-7B		DeepSeekMoE-16B
Method	PPL $\downarrow$	Mean Acc $\uparrow$	PPL $\downarrow$	Mean Acc $\uparrow$
Dense (0%)	12.62	65.96	9.05	62.28
HC	16.52	59.95	9.55	61.58
Det. HC	16.30	61.74	9.52	61.65
Det. HC + EM	15.36	63.92	9.42	61.80
Ours	15.20	64.13	9.38	61.84

Table 4: Ablation on mask-learning components at 20% sparsity. We report perplexity and zero-shot mean accuracy on LLaMA-7B and DeepSeekMoE-16B. HC: stochastic hard-concrete; Det. HC: deterministic hard-concrete; +EM: expanded-mask parameterization. Bold indicates the best performance among pruned variants at the same sparsity.

	LLaMA-7B	DeepSeekMoE-16B
Metric	Layer-wise	Layer-wise	Expert-wise
$\Delta$ Perplexity $\downarrow$	+0.04	+0.47	+1.17
$\Delta$ Mean Acc. (pp) $\uparrow$	-0.48	-1.09	-1.97

Table 5: Sparsity granularity ablation at 20% sparsity. We report the change in perplexity and zero-shot mean accuracy relative to global sparsity for LLaMA-7B and DeepSeekMoE-16B.

Panel of Figure 3 — Figure 3: Effect of different training tokens on perplexity and zero-shot mean accuracy on different models.

	LLaMA-7B		DeepSeekMoE-16B
Metric	C4	LaMini	C4	LaMini
$\Delta$ WikiText-2 PPL $\downarrow$	+0.35	-0.10	+0.21	+0.11
$\Delta$ Mean Acc. (pp) $\uparrow$	-1.14	-0.53	-0.68	-0.87

Table 6: Dataset ablation at 20% sparsity. We report the change in WikiText-2 perplexity and zero-shot mean accuracy relative to training masks on our default dataset (FineWeb-Edu) for LLaMA-7B and DeepSeekMoE-16B. Negative

\Delta

PPL and positive

\Delta

Acc indicate improvement.

Model	Method	Tokens	PPL $\downarrow$	Mean Acc. $\uparrow$
LLaMA-2-7B	LoRAP	0M	15.02	59.44
	LoRAP	30M	14.69	61.24
	LoRAP	60M	14.48	62.77
	LoRAP	120M	14.46	62.98
	DDP (Ours)	30M	14.39	64.82
DeepSeekMoE	HEAPr	0M	9.85	60.62

Table 7: Comparison between mask-only optimization and training-free pruning followed by LoRA recovery. PPL is evaluated on WikiText-2 for LLaMA-2-7B and on C4 for DeepSeekMoE-16B. DDP uses 30M mask-optimization tokens and keeps pretrained weights frozen.

Model	Method	12.5%		25%		37.5%		50%
Model	Method	Wiki PPL $\downarrow$	Mean Acc. $\uparrow$	Wiki PPL $\downarrow$	Mean Acc. $\uparrow$	Wiki PPL $\downarrow$	Mean Acc. $\uparrow$	Wiki PPL $\downarrow$	Mean Acc. $\uparrow$
LLaMA-2-7B	Týr-the-Pruner	5.84	56.98	7.51	54.64	10.29	52.21	16.17	47.41
LLaMA-2-7B	DDP (Ours)	5.72	61.99	6.87	60.29	8.17	55.98	10.34	52.17
LLaMA-2-13B	Týr-the-Pruner	5.03	62.66	5.79	61.16	7.17	58.67	9.59	54.58
LLaMA-2-13B	DDP (Ours)	5.03	65.62	5.75	64.51	6.65	60.88	8.14	58.43
Mistral-7B-v0.3	Týr-the-Pruner	5.61	63.05	7.08	60.22	10.25	52.34	15.53	46.21

Table 8: Additional comparison with Týr-the-Pruner on dense models under different sparsity ratios. We report WikiText-2 perplexity and mean zero-shot accuracy. Bold indicates the best result at the same model and sparsity ratio.

Method	Params	Memory	FLOPs	Time / Step
DDP	0.35M	18 GB	0.97E	2.1 s
DDP (distill)	0.35M	19 GB	1.254E	2.5 s
LoRA	20M	18 GB	0.97E	2.4 s
Full FT	7B	83 GB	1.40E	2.9 s

Table 9: Per-step training cost comparison on LLaMA2-7B with global batch size 16 and sequence length 2,048. Memory denotes the sum of activation memory, model memory, and gradient/optimizer memory. For LoRA, we use rank

r=8

Model	GPU		0%	20%	50%
LLaMA-7B	RTX 5090	Through.	10.88	14.75	23.98
LLaMA-7B	RTX 5090	Speedup	1.00×	1.36x	2.20x
LLaMA-13B	B200	Through.	27.20	32.14	38.56
LLaMA-13B	B200	Speedup	1.00×	1.18x	1.42x

Table 10: End-to-end speedup results on dense models. We report Throughput (requests per second) and speedup ratio.

Model	GPU		0%	20%	40%	60%
Qwen3 30B-A3B	B200	Through.	29.86	33.25	37.42	45.16
Qwen3 30B-A3B	B200	Speedup	1.00×	1.11x	1.25x	1.51x

Table 11: End-to-end speedup results on MoE models. We report Throughput (requests per second) and speedup ratio.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Ratio	Method	Wiki2	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
LLaMA-7B
Dense	Original	12.62	76.50	79.35	76.00	69.53	72.10	44.28	44.00	65.96
20%	LoRAPrune	16.80	65.62	79.31	70.00	62.76	65.87	37.69	39.14	60.05
	LoRAP	16.35	72.94	76.93	70.90	65.75	64.31	39.93	41.20	61.70
	SlimLLM	15.55	74.71	76.61	71.23	66.54	66.96	40.61	40.20	62.41
	Ours	15.20	74.10	77.97	73.20	69.22	69.65	42.41	42.40	64.13

Appendix

Table 12: Detailed performance of pruned dense LLMs under different pruning methods. All pruning baseline methods are evaluated after fine-tuning. Bold indicates the best performance at the same pruning ratio.

Ratio	Method	Wiki2	BoolQ	OBQA	RTE	WinoGrande	HellaSwag	PIQA	MathQA	ARC-e	ARC-c	Average
Qwen3-32B
Dense	Original	7.61	86.33	46.20	76.53	72.93	82.57	82.15	58.32	83.21	60.84	72.12
20%	Ours	7.25	81.77	46.80	76.17	75.37	80.44	80.58	52.26	83.63	62.80	71.09
50%	Ours	9.78	82.81	44.80	74.37	71.19	73.31	77.80	37.76	80.89	55.38	66.48
Qwen3-14B
Dense	Original	8.65	89.30	46.20	77.62	72.45	78.82	80.03	56.85	82.87	60.32	71.61

Appendix

Table 13: Detailed performance of pruned Qwen3 dense LLMs using our method.

Ratio	Method	C4	BoolQ	OBQA	RTE	WinoGrande	HellaSwag	PIQA	MathQA	ARC-e	ARC-c	Average
DeepSeekMoE-16B-Base
Dense	Original	9.05	72.45	44.00	63.54	70.24	77.37	80.68	31.52	72.89	47.86	62.28
20%	NAEE	10.07	67.83	42.40	62.09	69.53	74.63	78.34	30.99	72.56	46.24	60.51
	$\text{D}^{2}$ -MoE	12.62	69.32	41.40	61.01	69.22	69.87	76.44	29.45	71.29	42.75	58.97
	Camera-P	9.84	68.01	44.00	64.62	70.17	75.02	78.62	31.46	71.80	45.56	61.03
	HEAPr	9.85	64.77	44.60	56.68	70.72	76.59	79.87	31.83	73.91	46.59	60.62

Appendix

Table 14: Detailed performance of compressed Mixture of Experts LLMs under different pruning methods. Bold indicates the best performance at the same pruning ratio.

Panel of Figure 4 — Figure 4: Dense-model sparsity patterns (LLaMA-7B, 20% sparsity). Left: layer-wise MLP channel sparsity. Right: learned head sparsity map.

Panel of Figure 5 — Figure 5: Dense-model sparsity patterns (LLaMA-7B, 50% sparsity). Sparsity increases toward later layers, and head pruning becomes more selective under higher compression.

Panel of Figure 6 — Figure 6: MoE expert sparsity patterns (DeepSeekMoE-16B). Expert-wise sparsity maps at different target sparsities, showing that pruning increasingly concentrates on rarely activated experts.

Panel of Figure 7 — Figure 7: Magnitude distributions of learned mask values. The learned masks are centered near 1 but span a wider continuous range, demonstrating the effect of DDP’s expanded mask parameterization.

LLM Pruning

Large language model pruning removes less useful weights or components to reduce memory and inference costs while preserving model capabilities.

See all 178 papers

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Mar 25, 2026Jimyung Hong, Jaehyung KimLLM Pruning Structured Pruning
Yonsei University
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Feb 11, 2025Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2LLM Pruning Structured Pruning
MBZUAI · ETH Zurich · ISTA +1
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1

Mar 25, 2026J/K move · Enter open · S save

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

Jimyung Hong, Jaehyung Kim

Yonsei University

LLM Pruning Structured Pruning Neural Network Pruning LLM Compression

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. At 20% sparsity on Gemma-2 2B, DIET improves average accuracy by 2.1 percentage points over SliceGPT and by 9.2 percentage points over PuDDing. On Gemma-2 9B, DIET remains competitive with SliceGPT while consistently outperforming PuDDing at 10% and 20% sparsity.

Abstract

LLM Pruning

Explore similar work

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

DarwinLM: Evolutionary Structured Pruning of Large Language Models

Prune, Update and Trim: Robust Structured Pruning for Large Language Models