Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.
Figures & tables
Figure 1: Comparison of ELL with our new TwELL and Hybrid sparse formats designed for LLM inference and training, respectively.
Figure 2Figure 3Figure 4
Figure 5: Training speedups and peak memory reduction from our sparse LLM training kernels across L1 regularization levels.
Model scale
Sparse
Mean task accuracy
Forward execution [-1pt] (input tokens/ms)
Energy per token [-1pt] (mJ)
Training step [-1pt] (input token/ms)
Peak memory [-1pt] (GB)
0.5B params [-1pt] 10B tokens
✗
40.4%
410 +0 (0.0%)
1.63 +0 (0.0%)
97.3 - (0.0%)
26.2 +0 (0.0%)
✓
40.4%
480 (+17.0%)
1.43 (-11.8%)
95.9 (-1.5%)
21.2 (-19.2%)
1B params [-1pt] 20B tokens
✗
44.6%
185 +0 (0.0%)
3.71 +0 (0.0%)
48.6 +0 (0.0%)
44.5 +0 (0.0%)
✓
44.7%
219 (+18.1%)
3.17 (-14.6%)
52.1 (+7.1%)
33.1 (-25.5%)
1.5B params [-1pt] 30B tokens
✗
46.4%
119 +0 (0.0%)
5.73 +0 (0.0%)
31.8 +0 (0.0%)
62.8 +0 (0.0%)
✓
46.2%
141 (+18.8%)
4.87 (-15.0%)
35.5 (+11.6%)
45.1 (-28.1%)
Table 1: Comparison of performance and efficiency statistics of sparse LLMs leveraging our kernels with traditional models.
Figure 6: Sparsity statistics and speedup contributions across different layers of our sparse LLMs.
Figure 7: Sparsity statistics across LLM input tokens and positions.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Gated LLM
Non-Gated LLM
Model architecture
Hidden size
2048
2048
Hidden MLP size (intermediate)
5632
8192
Gated MLP
true
false
Hidden activation
ReLU
ReLU
Number of hidden layers
8/18/28/38
8/18/28/38
Appendix
Table 2: Default Hyperparameters for Pretraining Sparse and Non-Sparse LLMs.
Model scale
Activation
Sparse
L1 coeff.
Mean task accuracy
Cross-entropy
# non-zeros
Forward execution [-1pt] (input tokens/ms)
Energy per token [-1pt] (mJ)
1.5B params [-1pt] 30B tokens
ReLU
✗
0
46.4%
2.255
911
117.1 +0 (0.0%)
5.77 +0 (0.0%)
SiLU
✗
0
47.1%
2.240
5632
116.5 0 (-0.5%)
5.82 0 (+0.1%)
ReLU
✓
2×10−5
46.2%
2.297
29
138.0 (+17.9%)
5.07 (-12.1%)
Appendix
Table 3: Comparison of performance and efficiency statistics of sparse LLMs leveraging our kernels with traditional gated models using both ReLU and SiLU activations ( ??? ).
Model scale
Gated
Sparse
L1 coefficient
Mean task accuracy
Forward execution [-1pt] (input tokens/ms)
Energy per token [-1pt] (mJ)
1.5B params [-1pt] 30B tokens
✓
✗
0
46.36%
117.1 +0 (0.0%)
5.79 +0 (0.0%)
✓
2×10−5
46.20%
138.0 (+17.9%)
5.07 (-12.5%)
✓
3×10−5
44.83%
147.0 (+25.5%)
4.75 (-18.0%)
1.5B params [-1pt] 30B tokens
✗
✗
0
46.57%
125.8 +0 (0.0%)
5.52 +0 (0.0%)
✓
2×10−5
46.46%
139.9 (+11.2%)
5.03 (-8.8%)
✓
3×10−5
44.71%
142.3 (+13.1%)
4.86 (-12.0%)
Appendix
Table 4: Comparison of performance and efficiency statistics of sparse LLMs leveraging our kernels with traditional baselines, considering both gated models ( ? ), and their original non-gated counterparts used in the original transformer ( ? ).
Model scale
Training [-1pt] modification
Sparse
L1 coefficient
Mean task accuracy
Cross-entropy
# non-zeros
Forward execution [-1pt] (input tokens/ms)
Energy per token [-1pt] (mJ)
1.5B params [-1pt] 30B tokens
–
✗
0
46.4%
2.255
911
117.1 +0 (0.0%)
5.77 +0 (0.0%)
–
✓
2×10−5
46.2%
2.297
29
138.0 (+17.9%)
5.07 (-12.1%)
Dead neuron reinit.
✓
2×10−5
46.6%
2.298
29
139.4 (+19.1%)
4.96 (-14.0%)
Sparsity warmup
✓
3×10−4
45.9%
2.293
108
119.3 (+1.9%)
5.76 (-0.1%)
Appendix
Table 5: Comparison of performance and efficiency statistics of sparse LLMs leveraging our kernels with traditional baselines trained using our standard recipe, or with dead neuron mitigation strategies such as warming up the coefficient of the L1 loss and applying targeted reinitialization to the gate projection’s weights.
Figure 8: Number of non-zeros and fraction of dead neurons of LLMs with different strategies for dead neuron mitigation throughout training.
Figure 9: Number of non-zeros and fraction of dead neurons of LLMs across L1 regularization levels throughout training.
Model scale
Sparse
Mean Accuracy
HellaSwag
CQA
PIQA
Winogrande
ARC-easy
ARC-challenge
OpenBookQA
0.5B params [-1pt] 10B tokens
✗
0 40.4%
0 33.7%
0 20.9%
0 64.5%
0 50.9%
0 64.1%
0 28.1%
0 20.8%
✓
0 40.4%
0 34.0%
0 22.3%
0 66.4%
0 53.8%
0 60.5%
0 27.5%
0 18.0%
1B params [-1pt] 20B tokens
✗
0 44.6%
0 39.2%
0 20.0%
0 68.7%
0 54.4%
0 72.6%
0 34.0%
0 23.6%
✓
0 44.7%
0 39.8%
0 18.6%
0 68.1%
0 54.8%
0 71.6%
0 35.3%
0 24.4%
1.5B params [-1pt] 30B tokens
✗
0 46.4%
0 41.0%
0 20.8%
0 70.2%
0 55.9%
0 72.5%
0 36.7%
0 27.4%
✓
0 46.2%
0 41.1%
0 21.0%
0 69.1%
0 54.4%
0 74.3%
0 37.5%
0 26.0%
Appendix
Table 6: Granular comparison of per-task downstream performance across model scales to complement Table 1 .
Figure 10: Sparsity statistics and speedup contributions across different layers of non-sparse LLMs.
Figure 11: Sparsity statistics and speedup contributions across different layers of an LLM with high regularization L1=104 .
Figure 12: Training speedups from our sparse LLM training kernels across L1 regularization levels for both H100 and RTX6000 devices.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.