S$^3$: Structured Sparsity Specification

Ayoub Ghriss

cs.LGApr 13, 2026

S $^3$ : Structured Sparsity Specification

Authors: Ayoub Ghriss

Organizations: Department of Computer Science, Colorado University, Boulder

Abstract

We introduce the Structured Sparsity Specification (S $^3$ ), an algebraic framework for defining, composing, and implementing structured sparse patterns. S $^3$ specifies sparsity through three components: a View that reshapes the tensor via layout composition, a Block specification that defines the atomic pruning unit, and the sparsity decision Scope. Both Block and Scope support Coupling across tensors for coordinated sparsification. S $^3$ enables precise specification of diverse sparsity structures, from fine-grained N:M patterns to coarse channel pruning, and integrates seamlessly with Optimal Brain Damage (OBD) and Surgeon (OBS). We formalize the framework mathematically, demonstrate its expressiveness on canonical patterns, and validate it experimentally via structured OBS and OBD implementations built entirely on S $^3$ , which surpasses well-established second order heuristics on output reconstruction across common configurations.

Figures & tables

Panel of Figure 1 — Figure 1: S 3 can target low-level hardware patterns to avoid any slowdowns due to memory access conflicts. For m16n8k16 matrix multiplication patterns, we can target the thread-group as one block.

Panel of Figure 2 — Figure 2: Experimental sparsity patterns: structure (top) and pruned result (bottom). Cell fill color encodes block identity; colored outlines delimit scopes; dark gray cells are pruned to zero. All four patterns are expressed by the same three primitives (View, Block, Scope) with different shapes, and pruned by the same S-OBS algorithm.

Model	Method	PPL ( $\downarrow$ )	PPL Increase	Time
Qwen3-0.6B	Dense	26.13	—	—
	S-OBD	369.78	$+$ 1315.1%	79s
	SparseGPT	126.23	$+$ 383.0%	165s
	S-OBS	154.16	$+$ 489.9%	223s
Qwen3-1.7B	Dense	21.05	—	—
	S-OBD	125.75	$+$ 497.4%	167s

Table 1: End-to-end 2:4 pruning of Qwen3 models. WikiText-2 word perplexity, 1024 C4 calibration samples. S-OBS achieves lower perplexity on Qwen3-1.7B and Qwen3-4B but not on Qwen3-0.6B, despite lower per-linear errors on all three ( Appendices F and G ).

Panel of Figure 3 — Figure 3: End-to-end 2:4 pruning of Qwen3-1.7B. Figure 3(a) : WikiText-2 perplexity after pruning every fourth decoder layer. S-OBS maintains a growing advantage throughout, reaching 46.3 vs. 56.4 for SparseGPT. Figure 3(b) : Per-layer decoder output error. S-OBS achieves lower error in 25 of 28 layers; SparseGPT is slightly better only in the final three.

Config	S-OBS	SparseGPT	Impr.	Time
2:4	11.87%	14.12%	$+$ 16.0%	53s
Coupled 2:4	15.75%	19.01%	$+$ 17.1%	217s
4:8	15.92%	19.00%	$+$ 16.2%	279s
16-col block	26.91%	33.46%	$+$ 19.6%	108s

Table 2: Structured OBS via S 3 vs. SparseGPT on the first decoder layer of Qwen3-4B across four sparsity configurations (50% sparsity). S-OBS reports the best variant (True OBS with per-row Schur updates). “Impr.” is relative loss reduction

(1-\text{S-OBS}/\text{SparseGPT})\times 100

Panel of Figure 4 — Figure 4: Per-linear analysis of Qwen3-1.7B 2:4 pruning. Figure 4(a) : Output error scatter (H-based metric). Points below the diagonal indicate S-OBS achieves lower error. S-OBS wins on 177 of 196 linear layers. Figure 4(b) : Relative improvement heatmap across layers and sublayer types. Blue indicates S-OBS is better. S-OBS dominates except for a few MLP projections in the final layers.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathcal{T}$	A dense tensor
${\bm{s}}=(s_{0},\ldots,s_{n-1})$	Shape tuple (extents)
${\bm{d}}=(d_{0},\ldots,d_{n-1})$	Stride tuple
$\mathcal{L}={\bm{s}}:{\bm{d}}$	Layout specification
$\|\mathcal{L}\|$	Size of a layout
$\text{cosize}(\mathcal{L})$	Cosize (memory footprint)

Appendix

Table 3: Notation summary.

Figure 5: Standard 2:4 (top) vs. Coupled 2:4 (bottom) over a single A-fragment row of mma.m16n8k16 (16 k-columns). Cell color encodes the owning TID; faded cells are pruned. Arcs link the column pair

\{c,\;c\!+\!8\}

; solid arcs are retained pairs, dashed arcs are pruned pairs. Standard 2:4 requires independent masks for each k-half. Coupled 2:4 uses one mask for both halves, halving the metadata and improving the compression ratio from

9/16

17/32

( Section E.5 ).

Pattern	View $\mathcal{V}$	Block ${\bm{b}}$	Scope ${\bm{s}}$	Keep
Unstructured	$(M,\;K):(K,1)$	$(1,1)$	$(M,\;K)$	$k$
N:M (2:4)	$(M,\;K):(K,1)$	$(1,1)$	$(1,\;4)$	$2$
Block $b\!\times\!b$	$(M,\;K):(K,1)$	$(b,\;b)$	$(1,\;K/b)$	$k$
Channel	$(C_{\text{out}},\;C_{\text{in}}HW)$	$(1,\;C_{\text{in}}HW)$	$(C_{\text{out}},)$	$k$
Head	$(h,\;d/h,\;d)$	$(1,\;d/h,\;d)$	$(h,\;1,\;1)$	$k$

Appendix

Table 4: Canonical sparsity patterns expressed in S 3 . Strides are implied by standard row-major order. The Keep column is the number of blocks retained per scope.

Figure 6: Thread-fragment assignment for mma.sync.aligned.m16n8k16 . Cell color encodes the owning thread (hue = thread block

g=\lfloor t/4\rfloor

, brightness = within-block offset

t\bmod 4

). Dashed lines mark the k-half boundary (

k\!=\!8

). Each thread owns 8 fp16 elements of A, 4 of B, and 4 fp32 elements of C. The 2-column k-pair structure of A and B is the hardware reason why 2:4 and 4:8 sparsity patterns are natural: surviving elements fill exactly the k-col pairs owned by a single thread.

Method	Dense PPL	PPL ( $\downarrow$ )	PPL Increase	Time
S-OBD	26.13	369.78	$+$ 1315.1%	79s
SparseGPT	26.13	126.23	$+$ 383.0%	165s
S-OBS	26.13	154.16	$+$ 489.9%	223s

Appendix

Table 5: End-to-end 2:4 pruning of Qwen3-0.6B (28 layers). WikiText-2 word perplexity, 1024 C4 calibration samples. S-OBD (no compensation) performs substantially worse than both SparseGPT and S-OBS, confirming that compensation is critical.

Panel of Figure 7 — Figure 7: End-to-end 2:4 pruning of Qwen3-0.6B. Figure 7(a) : WikiText-2 perplexity after pruning every fourth decoder layer. SparseGPT maintains lower perplexity throughout, reaching 126.2 vs. 154.2 for S-OBS. Figure 7(b) : Per-layer decoder output error. S-OBS achieves lower error in early-to-mid layers, but SparseGPT becomes substantially better in later layers.

Panel of Figure 8 — Figure 8: Per-linear analysis of Qwen3-0.6B 2:4 pruning. Figure 8(a) : Output error scatter. S-OBS wins on 178 of 196 linear layers, comparable to the 177/196 advantage on Qwen3-1.7B. Figure 8(b) : Improvement heatmap. S-OBS dominates early-to-mid layers but loses in the final layers, particularly for projections with large input dimension ( down_proj , $K\!=\!3072$ ).

Panel of Figure 9 — Figure 9: Per-projection output error and decoder error across decoder layers (2:4 pruning, 1024 C4 calibration samples). Solid = S-OBS, dashed = SparseGPT. Top row: attention projections (Q/K/V/O). Middle row: MLP projections (Gate/Up/Down). Bottom row: full decoder output error. Left column: Qwen3-1.7B. Right column: Qwen3-0.6B. Per-linear metric (top/middle): $\sqrt{\operatorname{tr}(\Delta\mathbf{W}\,\mathbf{H}\,\Delta\mathbf{W}^{T})\,/\,\operatorname{tr}(\mathbf{W}\,\mathbf{H}\,\mathbf{W}^{T})}$ where $\Delta\mathbf{W}=\hat{\mathbf{W}}-\mathbf{W}$ , $\mathbf{H}=\frac{1}{N}\mathbf{X}^{T}\mathbf{X}$ . Decoder metric (bottom): $\left\|\hat{\mathbf{Y}}-\mathbf{Y}\right\|_{\mathrm{F}}\,/\,\left\|\mathbf{Y}\right\|_{\mathrm{F}}$ where $\mathbf{Y}$ and $\hat{\mathbf{Y}}$ are the decoder layer outputs before and after pruning. S-OBS achieves lower per-linear error across nearly all layers and projection types, but the decoder-level advantage reverses in later layers, more severely for the smaller model.

Panel of Figure 10 — Figure 10: Qwen3-4B (36 layers): per-projection output error and decoder error (2:4 pruning, 1024 C4 calibration samples). Same format as Figure 9 . S-OBS wins on all 252 linear projections. The decoder error spike at layer 16 is shared by both methods, suggesting an inherently sensitive layer.

Structured Sparsity

Structured sparsity organizes zeros into meaningful groups, blocks, channels, or patterns to simplify models and computation.

See all 59 papers

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†
RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch
Feb 16, 2026Isam Vrce, Andreas Kassler, Gökçe AydosStructured Sparsity Neural Network Compression
Deggendorf Institute of Technology, Dieter-Görlitz-Platz 1, Deggendorf, Germany · Department of Engineering Technology, Technical University of Denmark, Ballerup, Denmark
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakCross-Modal Representation Learning Sparse Mixture-of-Experts
Seoul National University, Seoul, South Korea.

Jun 9, 2026J/K move · Enter open · S save

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Snowflake AI Research* · Seoul National University†

LLM Pruning High-Performance Computing Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Open paper AskPDF arXiv

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

Abstract

Structured Sparsity

Explore similar work

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch

Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts