Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

Jafar Badour; Maurice van Keulen; Elena Mocanu

cs.LGOct 2, 2026

Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU

Authors: Jafar Badour, Maurice van Keulen, Elena Mocanu

Organizations: University of Twente

Abstract

Deep neural networks continue to grow in parameter count, driving up training and inference cost on GPUs. Sparse neural networks and Dynamic Sparse Training (DST) promise to reduce these costs, but most implementations rely on binary masks over dense tensors and recover little of the theoretical compute, memory, or energy savings. We propose SNACK, a truly sparse GPU layer that stores and computes only non-zero connections. SNACK exposes a simple PyTorch API for restructuring connections and backpropagating gradients entirely in the sparse paradigm, and ships SNACK-COO, a custom COO-format SpMM CUDA kernel with a batch-to-Streaming-Multiprocessor mapping tuned for the small-batch, high-sparsity regime typical of large-model training and single-stream inference. At the kernel level, SNACK is up to 7x faster than the masked dense baseline (Dense+Mask) and competitive with cuSPARSE, Sputnik, and FlashSparse at 95% sparsity. At 90% sparsity, a single SNACK layer accelerates training by 8x and 3.7x, and inference by 4x and 2x, over Dense+Mask and fully dense layers, respectively, while using 72% less memory than dense and substantially less energy. End-to-end, SNACK reduces GPT-2 peak training memory by up to 40% and graph-style inference latency by 4.8x over Dense+Mask at 99% sparsity.

Figures & tables

Method	Truly Sparse	Unstructured	Hardware-Aware	Sparsity Phase	CPU/GPU/IPU
Truly SNNs at Scale [ 6 ]	✓	✓		Training	CPU
SET-MLP [ 29 ]	✓	✓		Training	CPU
QuickSelection [ 2 ]	✓	✓		Training (Autoencoder)	CPU
Sparse-IFT [ 49 ]			✓	Training & Inference	GPU
PopSparse [ 26 ]	✓		✓	Training & Inference	IPU
BLaST [ 39 ]			✓	Training & Inference	GPU

Table 1: DST-oriented methods that maintain a sparse representation throughout training. “Truly Sparse” means no dense weight tensor is materialized at any point; “Unstructured” means the method does not require a block,

N{:}M

, or other hardware-friendly sparsity pattern; “Hardware-Aware” means the method requires structured patterns or specialized accelerator support. SNACK is, to our knowledge, the first truly sparse PyTorch building block on commodity GPUs that supports unstructured sparsity end-to-end and exposes both forward and backward passes in the sparse paradigm.

Figure 1: SNACK layer overview. Two stacked SNACK blocks ( $A$ and $B$ ) act as drop-in replacements for standard nn.Linear layers. Each block stores its weight matrix $\mathrm{SpM}(\cdot)$ on the GPU as the COO triple $(\mathbf{A}_{v},\mathbf{A}_{r},\mathbf{A}_{c})$ over its non-zeros only: the highlighted (blue) edges are the active sparse subnetwork that consumes memory and FLOPs, while the faded (grey) edges depict the underlying dense linear layer that SNACK never materializes during training or inference. The forward pass is computed by SNACK-COO directly on the triple, the backward pass uses a sparse outer product over $\mathrm{supp}(W)$ , and DST methods may add or drop active connections in place, so SNACK and dense blocks can be freely interleaved in a network without changing any caller code.

Figure 2: SpMM CUDA time vs. sparsity at fixed batch sizes

\{1,2\}

on a

5000\times 5000

matrix. Each panel sweeps sparsity for one batch size. SNACK-COO (ours, orange) stays below the Dense+Mask baseline across these small-batch settings and remains competitive with the dense reference.

Method	is sparse	CUDA (ms)
		500x500	1000x1000	5000x5000	10000x10000
SpMM_CSR(Torch)	Yes	27.05	26.84	231.83	885.33
SpMM_CSR(Cupy)		11.95	11.66	12.27	17.90
SpMM_COO(Torch)		27.41	29.10	341.60	1396.26
SNACK-COO (ours)		2.43	2.38	7.28	15.17
SpMM_COO(Cupy)		207.86	215.45	1231.38	4946.64

Table 2: Performance comparison of sparse matrix multiplication implementations. SpM sparsity is 95%, batch size of 1. Our SNACK-COO method outperforms other methods (7x reduction in time vs DenseMM+Mask). † Simulates sparsity.

Figure 3: Layer-level GPU profile,

10000{\times}10000

projection (

100

SGD iterations). Top row: per-iteration CUDA time (log

\mu

s; the multiple lines per method are mini-batches

b\!\in\!\{1,2,4,8,16,32,64\}

, annotated bz=

b

); average total energy (mWh); total energy per batch (mWh, same

b

-sweep). Bottom row: total parameters (sparse values + COO indices for SNACK), trainable parameters, peak GPU memory (MB), and steady-state power (W); these four panels report a single curve per method. Only SNACK’s parameter, memory, and energy curves fall linearly with

1-\eta

; Dense+Mask stays flat because it still allocates and multiplies through a dense

W

. CUDA-time and energy axes are log-scaled because of the wide dynamic range across

\eta

Type	Sparsity	Params	Memory	CUDA $\tau$	Power	Energy	CO 2
Type	%	(k)	(MB)	( $\mu s$ )	(W)	(mWh)	(mg)
Dense+Mask	90	200,000	4371	1085.6	94.6	3.08	1.38
SNACK (ours)	90	30,000	1173	484.4	81.7	0.85	0.38
Dense+Mask	95	200,000	4140	1085.6	94.7	3.15	1.42
SNACK (ours)	95	15,000	947	254.6	71.6	0.45	0.20
Dense+Mask	99	200,000	4447	1085.3	94.4	3.07	1.38

Table 3: Performance and environmental impact comparison of different implementations on a

10000{\times}10000

matrix at batch size 4. Energy (mWh) and CO 2 (mg) use local emission factors. Dense (Only) is sparsity-independent and is reported once at the bottom as a reference (mean over the three sparsity-block runs; per-run variation is

<2\%

on every column except memory).

Figure 4: Memory vs. input size at hidden dimension

10000

. Dense and Dense+Mask OOM at input

\geq 15000

; Dense+Mask at

\eta=0.9

fails earlier (

\geq 12500

) due to lazy allocation. SNACK scales gracefully because no step ever materializes a dense

W

Sparsity	Method	Perplexity	Memory	Energy_spent	Latency_token_ms
0.25	Dense+Mask	29.92	2260.84	6916.40	41.64
0.25	SNACK(Sputnik) (ours)	29.73	2506.92	345236.72	582.63
0.50	Dense+Mask	36.50	2260.84	7021.94	41.59
0.50	SNACK(Sputnik) (ours)	36.89	2126.79	190115.90	453.23
0.75	Dense+Mask	43.08	2260.84	7127.47	41.53
0.75	SNACK(Sputnik) (ours)	43.05	1746.65	34995.07	332.92

Table 4: GPT-2 (small) dynamic sparse training across target sparsity levels. We compare Dense+Mask and SNACK(Sputnik) (ours), reporting validation perplexity (WikiText-103 validation split), peak GPU memory (MB), per-step energy (mJ), and per-token training latency (ms); dense is the no-sparsity reference. Best per-metric per-sparsity value is in bold .

Sparsity	Method	Memory	Energy_spent	Latency_token_ms
0.25	Dense+Mask	1255.59	761.16	5.42
0.25	SNACK(Sputnik) (ours)	1609.92	2237.78	11.91
0.5	Dense+Mask	1255.59	749.48	5.33
0.5	SNACK(Sputnik) (ours)	1284.49	1507.93	9.31
0.75	Dense+Mask	1255.59	737.79	5.25
0.75	SNACK(Sputnik) (ours)	959.05	778.07	6.70

Table 5: GPT-2 (small) inference across target sparsity levels at batch size 1 on a consumer-class RTX 3070 (6 GB). We compare Dense+Mask and SNACK(Sputnik) (ours), reporting peak GPU memory (MB), per-token energy (mJ), and per-token latency (ms); dense is the no-sparsity reference. Best per-metric per-sparsity value is in bold .

Method	Sparsity	Memory	Energy_spent	Inference_latency_ms
Dense+Mask	0.9	408.56	52.79	1.02
SNACK(SNACK-COO) (ours)	0.9	128.60	11.82	0.25
Dense+Mask	0.95	408.56	53.63	1.01
SNACK(SNACK-COO) (ours)	0.95	118.60	10.14	0.21
Dense+Mask	0.99	309.89	50.86	1.01
SNACK(SNACK-COO) (ours)	0.99	110.60	9.82	0.21

Table 6: Graph-style sparse inference on a

5000{\times}5000

GAMLP-shaped workload at batch size 1 , averaged over 2,000 samples after warm-up. We compare Dense+Mask and SNACK(SNACK-COO) (ours), reporting peak memory (MB), per-sample energy (mJ), and inference latency (ms); dense is the no-sparsity reference. Best per-metric per-sparsity value is in bold .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Format	Data	Indices	Index Pointer
COO	[3, 4, 5]	row = [0, 1, 2]	col = [2, 0, 1]
CSR	[3, 4, 5]	col = [2, 0, 1]	indptr = [0, 1, 2, 3]
CSC	[4, 5, 3]	row = [1, 2, 0]	indptr = [0, 1, 2, 3]

Appendix

Table 7: Sparse matrix representations in COO, CSR, and CSC formats.

Sparsity	RigL*	RigL (SNACK) (ours)	Dense
(%)			ResNet-50
80	$74.9\pm 0.1$	$74.7\pm 0.2$
90	$72.8\pm 0.1$	$72.8\pm 0.1$	$76.7$
95	$69.6\pm 0.1$	$69.6\pm 0.2$	$\pm 0.1$
99	$51.4\pm 0.2$	$51.4\pm 0.3$	$\pm 0.1$

Appendix

Table 8: Top-1 ImageNet test accuracy of ResNet-50 trained with RigL at high sparsities. RigL* is the method used by [ 23 ] ; RigL (SNACK) is the same method but using SNACK instead of a dense layer with masked weights.

Sparsity	RigL*	RigL	Dense
%		(SNACK) (ours)	ResNet-50
80	$5.170\pm 0.007$	$5.217\pm 0.010$	$4.26\pm 0.003$
90	$5.172\pm 0.005$	$5.045\pm 0.010$	$4.25\pm 0.004$
95	$5.156\pm 0.006$	$5.033\pm 0.015$	$4.24\pm 0.003$
99	$5.112\pm 0.007$	$5.017\pm 0.020$	$4.25\pm 0.003$

Appendix

Table 9: Energy consumption (in kWh) of ResNet-50 trained with RigL at various sparsities. RigL* are from [ 23 ] ; RigL (SNACK) replaces dense masked layers with truly sparse CUDA kernels.

Type	Sparsity	Params	Memory	CUDA $\tau$	Power	Energy	CO 2
Type	%	(k)	(MB)	( $\mu s$ )	(W)	(mWh)	(mg)
Dense+Mask	90	250,000	5995	1358.3	102.8	4.36	1.96
SNACK (ours)	90	37,500	1863	163.1	100.2	0.39	0.18
Dense+Mask	95	250,000	6473	1357.4	98.9	4.20	1.89
SNACK (ours)	95	18,750	1023	83.0	111.5	0.43	0.19
Dense+Mask	99	250,000	7429	1358.3	103.3	4.38	1.97

Appendix

Table 10: Performance and environmental impact comparison of different implementations. All experiments use a 12500×10000 matrix with batch size of 1. Energy consumption (in mWh).

Figure 5: SNACK performance when SNACK layer dimension is 2500x10000

Figure 6: SNACK performance when SNACK layer dimension is 5000x10000

Figure 7: SNACK performance when SNACK layer dimension is 7500x10000

Figure 8: SNACK performance when SNACK layer dimension is 10000x10000

Figure 9: SNACK performance when SNACK layer dimension is 12500x10000

Method	is sparse	CUDA (ms)
		500	1000	5000	7500	8500	10000
SpMM_CSR(Torch)	Yes	24.83	25.83	215.16	443.18	576.99	811.52
SpMM_CSR(Cupy)		11.35	11.05	11.63	11.45	14.47	15.68
SpMM_COO(Torch)		24.55	26.59	321.53	693.69	901.41	1260.55
SNACK-COO (ours)		2.72	2.85	6.24	8.12	13.39	13.93
SpMM_COO(Cupy)		166.27	175.73	1153.05	2520.67	3410.45	4893.86

Appendix

Table 11: A100 replication of Table 2 . CUDA time (ms) for sparse matrix multiplication implementations on an NVIDIA A100, sparsity

95\%

, batch size

1

, square matrices, mean of

1{,}000

multiplications. Best per-column value among sparse methods is bold ; the DenseMM baseline is also bold where it remains the global best. † Simulates sparsity.

Method	is sparse	CUDA (ms)
		500	1000	5000	7500	8500	10000
SpMM_CSR(Torch)	Yes	27.46	28.25	248.35	528.04	675.79	950.01
SpMM_CSR(Cupy)		12.15	12.19	12.81	21.32	27.07	37.01
SpMM_COO(Torch)		27.37	30.03	342.17	767.56	979.77	1392.60
SNACK-COO (ours)		2.98	3.05	21.12	36.90	52.17	65.54
SpMM_COO(Cupy)		180.59	185.85	1267.72	2800.83	3618.27	5160.39

Appendix

Table 12: A100 SpMM benchmark at batch size

4

. CUDA time (ms) for sparse matrix multiplication implementations on an NVIDIA A100, sparsity

95\%

, batch size

4

, square matrices, mean of

1{,}000

multiplications. Same method set as Table 11 ; SpMM_BSR(Jax) is omitted because the JAX/CUDA toolchain on the A100 host could not be brought up at the matched library versions. Best per-column value among sparse methods is bold ; the DenseMM baseline is also bold where it remains the global best. † Simulates sparsity.

Method	Best val acc	Final val acc	Latency_step_ms	Energy_step_mJ	Peak Mem (MB)
Dense	0.9232	0.9191	17.11	1700.3	6823
Dense+Mask	0.9174	0.9140	17.34	1694.3	6840
SNACK-COO (ours)	0.9172	0.9101	160.39	17177.1	6784

Appendix

Table 13: GAMLP-style dynamic sparse training on ogbn-products at sparsity

\eta=0.9

, batch size

4096

40

epochs on a single GPU. We report best and final validation accuracy, mean per-step training latency (ms), per-step energy (mJ), and peak GPU allocation (MB). The SNACK-COO (ours) row reflects training executed through the SNACK layer with the SNACK-COO backend, included to expose the

n\ll m

regime mismatch at

b=4096

predicted in Appendix F ; under the recommended per-regime backend choice the SNACK layer is trained with the Sputnik backend and only swapped to SNACK-COO at inference time (Table 6 ). Best per-column value is in bold .

Dynamic Sparse Training

See all 10 papers

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation Compression Memory Compression
University of Cambridge · University of Oxford
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
Jun 13, 2026Tao Lu, Haoyu Wang, Zonghui Wang +3LLM Pruning GPU Kernel Optimization
National University of Singapore · Zhejiang University
SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†

Oct 15, 2026J/K move · Enter open · S save

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu

University of Cambridge · University of Oxford

Activation Compression Memory Compression Memory-Efficient Training Neural Network Compression Activation Sparsity

Open paper AskPDF arXiv

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse

\relu

and

\relu^2

activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31% and 30% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

Abstract

Dynamic Sparse Training

Explore similar work

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference