SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity

Hanyong Shao; Yingbo Hao; Ting Song; Yan Xia; Di Zhang; Shaohan Huang; Xun Wu; Songchen Xu

cs.LGMar 5, 2026

SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity

Authors: Hanyong Shao, Yingbo Hao, Ting Song, Yan Xia, Di Zhang, Shaohan Huang, Xun Wu, Songchen Xu,

Organizations: Peking University · Microsoft Research · South China University of Technology · Shanghai Jiao Tong University ·

Abstract

NVIDIA's 2:4 Sparse Tensor Cores deliver 2x throughput but demand strict 50% pruning -- a ratio that collapses LLM reasoning accuracy (Qwen3: 54% to 15%). Milder $(2N-2):2N$ patterns (e.g., 6:8, 25% pruning) preserve accuracy yet receive no hardware support, falling back to dense execution without any benefit from sparsity. We present SlideSparse, the first system to unlock Sparse Tensor Core acceleration for the $(2N-2):2N$ model family on commodity GPUs. Our Sliding Window Decomposition reconstructs any $(2N-2):2N$ weight block into $N-1$ overlapping 2:4-compliant windows without any accuracy loss; Activation Lifting fuses the corresponding activation rearrangement into per-token quantization at near-zero cost. Integrated into vLLM, SlideSparse is evaluated across various GPUs (A100, H100, B200, RTX 4090, RTX 5080, DGX-spark), precisions (FP4, INT8, FP8, BF16, FP16), and model families (Llama, Qwen, BitNet). On compute-bound workloads, the measured speedup ratio (1.33x) approaches the theoretical upper-bound $N/(N-1)=4/3$ at 6:8 weight sparsity in Qwen2.5-7B, establishing $(2N-2):2N$ as a practical path to accuracy-preserving LLM acceleration. Code available at https://github.com/bcacdwk/vllmbench.

Figures & tables

Figure 1: SlideSparse extends 2:4 Sparse Tensor Cores to the

\mathbf{(2N{-}2):2N}

sparsity family. (a) SlideSparse transforms 6:8 weights into 2:4-compliant blocks, enabling sparsity acceleration. (b) End-to-end speedup on A100 (INT8, seq_len

=

8K) approaches the theoretical limit

S_{\max}=N/(N{-}1)=3/2,4/3,5/4,...

(§ 3 ).

Figure 2: Reasoning accuracy of Qwen3 ( Yang et al., 2025 ) under different sparsity. 6:8 preserves near-dense performance (51.6% vs. 54.0%); 2:4 collapses to 15.3%.

Figure 3: Two-dimensional compression space for LLM acceleration. X-axis: quantization precision (16-bit to 1.58-bit BitNet, up to

8\times

speedup). Y-axis: sparsity (dense to 2:4, up to

2\times

speedup). Gray dots mark existing hardware support—limited to dense or 2:4 extremes. Green dots show

(2N{-}2):2N

patterns that SlideSparse enables, filling the Acceleration Gap and unlocking fine-grained sparsity–precision trade-offs.

Figure 4: Sliding window decomposition for 6:8 sparsity. Three stride-2 windows (each size 4) cover all 8 positions. Overlap regions allow non-zeros to spill into the next windows when one reaches capacity, converting any

(2N{-}2):2N

pattern into concatenated 2:4 blocks for Sparse Tensor Core acceleration.

Figure 5: SlideSparse system overview. Offline : Weight preprocessing transforms

(2N{-}2):2N

sparse weights into slided format with

\gamma\times

expansion. Initialization : cuSPARSELt compresses weights into 2:4 format at model load time. Online : Per-request inference executes fused quantization-slide kernel followed by sparse GEMM.

Figure 6: Kernel-level speedup at

\mathbf{M{=}16384}

across varied precisions and GPUs. Top row: B200 (INT8, FP8, BF16); Bottom row: A100 INT8, RTX 4090 FP8, RTX 5080 BF16. B200 INT8 achieves

4

–

6\times

due to suboptimal cuBLASLt INT8 performance on Blackwell; other configurations approach theoretical

S_{\text{eff}}

bounds. Comprehensive kernel results are shown in Appendix D.3 .

Figure 7: Kernel speedup vs.

\mathbf{M}

on A100 and B200 (Qwen-7B, INT8). (a) A100: speedup increases with

M

, approaching the theoretical limit for 6:8. (b) B200: consistently high speedup across all

M

(note different Y-axis scale).

Figure 8: End-to-end inference speedup. Top: Decode; Bottom: Prefill. A100/B200: INT8 (

M{=}512

decode,

M{=}16384

prefill); RTX 4090: FP8 (

M{=}512

decode,

M{=}8192

prefill; 24GB memory limits exclude Qwen-14B). For all E2E results see Appendix D.4 .

Figure 9: Efficiency: actual speedup vs. expected speedup (Qwen2.5-7B, Prefill,

M{=}8192

). (a) INT8; (b) FP8 (A100 lacks FP8 support). Expected speedup

=

(2:4 baseline)

\times

\gamma^{-1}

. Values

>

100% indicate SlideSparse’s fused kernel unlocks additional performance beyond 2:4 predictions.

Figure 10: Speedup vs.

\mathbf{M}

on B200 (Qwen-7B, INT8). (a) Decode (

M\in\{128,256,512\}

); (b) Prefill (

M\in\{4\text{K},8\text{K},16\text{K},32\text{K}\}

). All

(2N{-}2){:}2N

patterns achieve consistent speedups across

M

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

GPU	$\mathbf{M}$	Quant-only	Quant+Slide	Overhead	$\Delta$ ( $\mathbf{\mu}$ s)
A100	2048	18.4	27.7	+50%	9.3
A100	4096	30.7	44.0	+43%	13.3
A100	8192	60.4	77.8	+29%	17.4
A100	16384	109.6	141.3	+29%	31.7
H100	4096	26.6	34.7	+30%	8.1
H100	8192	47.3	64.2	+36%	16.9

Appendix

Table 1: Fused kernel latency (

\mu

s) for 6:8 sparsity. Overhead is relative to quant-only baseline.

GPU	Arch.	CC	Memory	Platform	Host Arch.
A100 80GB PCIe	Ampere	sm80	80GB HBM2e	Datacenter	x86_64
H100 80GB PCIe	Hopper	sm90	80GB HBM3	Datacenter	x86_64
B200 180GB SXM	Blackwell	sm100	180GB HBM3e	Datacenter	x86_64
RTX 4090	Ada Lovelace	sm89	24GB GDDR6X	Consumer	x86_64
RTX 5080	Blackwell	sm120	16GB GDDR7	Consumer	x86_64
DGX Spark (GB10)	Blackwell	sm121	128GB Unified	Embedded	aarch64

Appendix

Table 2: Hardware specifications for all evaluated GPUs.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†
Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference
Oct 20, 2026Xiuyi QinLLM Inference Acceleration Structured Sparsity
Independent Researcher, San Jose, CA, USA.
PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM Pruning LLM Compression
Department of Computer Science University of Toronto

Jun 9, 2026J/K move · Enter open · S save

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Snowflake AI Research* · Seoul National University†

LLM Pruning High-Performance Computing Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Open paper AskPDF arXiv

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

Abstract

LLM Inference Acceleration

Explore similar work

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference

PATCH: Learnable Tile-level Hybrid Sparsity for LLMs