Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

Di Zhang; Xun Wu; Shaohan Huang; Yudong Wang; Hanyong Shao; Yingbo Hao; Zewen Chi; Li Dong

cs.CLMar 5, 2026

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

Authors: Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong,

Organizations: Microsoft Research · Peking University · South China University of Technology

Abstract

Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models. To study this effect, we propose Sparse-BitNet, a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30X. These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet

Figures & tables

Figure 1: Intrinsic Sparsity in 1.58-bit BitNet. We present the aggregated weight statistics averaged across all linear layers of the pre-trained 1.58-bit BitNet (2B) model ( Ma et al., 2025b ) . (A) The distribution of normalized latent weights exhibits a distinct quantization-valley structure, where the majority of values fall within the [-0.5, 0.5] rounding interval. (B) Consequently, the quantized discrete states are dominated by zeros (approx. 42.3%), confirming that BitNet naturally converges to a highly sparse representation without explicit pruning .

Figure 2: Comparison of N:M sparsity friendliness between 1.58-bit BitNet and full-precision models. Normalized PPL increase relative to each method’s dense (8:8) counterpart. The dashed line marks a 10% degradation threshold. At 2:4 (50% sparsity; same ratio as 4:8), BF16 exceeds the threshold (+18.8%) while BitNet remains below it (+5.7%), indicating 1.58-bit BitNet is more sparsity-friendly than full-precision models.

Method	Task Accuracy (%)					Average
Method	HellaSwag	ARC-E	PIQA	BoolQ	COPA	Score	$\Delta$
Qwen2.5-0.5B
Dense BF16	40.45	43.31	69.04	60.12	71.00	56.78	–
Sparse BF16 (6:8)	39.21	39.84	66.43	57.32	66.00	53.76	-3.02
Dense BitNet	35.27	40.70	65.07	59.24	69.00	53.86	–
Sparse BitNet (6:8)	34.95	37.63	63.87	59.08	68.00	52.71	-1.15

Table 1: Downstream task performance evaluation. We compare the performance of Dense and Sparse models across varying scales (0.5B, 1.5B, and 3B) on five benchmarks. The

\Delta

column highlights the performance drop post-sparsification. Notable Trend: BitNet consistently demonstrates superior resilience to sparsification (smaller

\Delta

drop) compared to BF16 across all model scales .

Method	PPL $\downarrow$
Method	0.5B	1.5B	3B
Dense BF16	21.91	18.10	16.03
Sparse BF16 (6:8)	23.11 (+1.20)	18.70 (+0.60)	16.48 (+0.45)
Dense BitNet	25.99	20.11	17.70
Sparse BitNet (6:8)	26.31 (+0.32)	20.35 (+0.24)	17.87 (+0.17)

Table 2: Perplexity (PPL) degradation analysis. While BitNet has a higher baseline PPL due to quantization, its increase in PPL after sparsification (values in parentheses) is significantly smaller than that of BF16, highlighting its robustness.

	Throughput (k tok/s)		Speedup
(M)	Dense	Sparse	Speedup
Prefill (A100)
512	9.7k	10.6k	1.09 $\times$
1,024	20.3k	21.3k	1.05 $\times$
2,048	37.7k	42.5k	1.13 $\times$
4,096	40.9k	52.2k	1.28 $\times$

Table 3: End-to-End Performance: Qwen2.5-3B (Dense vs. 6:8 Sparse). Throughput is in thousands (k) of tokens/s. M = SeqLen (Prefill) / BatchSize (Decode).

Panel of Figure 3 — Figure 3: Ablation on training design choices for dynamic $6{:}8$ sparsity (Qwen2.5-0.5B). (a) Validation perplexity curves under different training design choices. (b) Corresponding mask flip rate $r_{t}$ (Eq. 6 ), reflecting the stability of sparsity pattern evolution during training.

Panel of Figure 4 — Figure 4: Polarization trend under ternary QAT. (a) BF16 maintains a concentration around zero. (b) BitNet shows decreasing near-zero mass, indicating strong polarization over time.

Figure 5: Weight Distribution. Global histogram of linear-layer master weights at the final checkpoint. Unlike the unimodal distribution of BF16, BitNet displays a structured, multi-modal magnitude landscape, confirming the intrinsic sparsity property.

Panel of Figure 6 — Figure 6: Overlay of $|w|$ density and per-block threshold $t$ (dense training). Blue: density of normalized magnitudes $|w|$ ; Orange: density of thresholds $t=|w|_{(N+1)}$ for $6{:}8$ selection. BitNet shows a more pronounced higher-magnitude population in mid/late layers, while thresholds concentrate mainly in the low-to-mid regime, suggesting that $N{:}M$ selection primarily operates within lower-magnitude candidates.

Sparse ratio $\rho$ (%)	Val PPL $\downarrow$
25	27.48
50	27.39
75	26.71
100 (sparse-from-scratch)	26.31
dense training	25.99

Table 4: Effect of dense-to-sparse schedule on validation PPL (6:8, Qwen2.5-0.5B ).

\rho

denotes the fraction of total training steps using

6{:}8

sparse training after an initial dense phase.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
$\beta_{1},\beta_{2},\epsilon$	[ $0.9$ , $0.95$ , $1e-5$ ]
Learning rate	1e-5
Schedule	cosine
Warmup ratio	0.5
Weight decay	0.1

Appendix

Table 5: Training hyperparameters for all experiments.

Method	8:8	7:8	6:8	5:8	2:4	3:8	2:8
BitNet	25.99	26.12	26.31	26.71	27.48	29.80	33.12
BF16	21.91	22.27	23.11	23.42	26.03	28.66	31.70

Appendix

Table 6: Raw validation PPL for the

N{:}8

sparsity sweep on Qwen2.5-0.5B .

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 350 papers

SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask
May 7, 2026Liu Hanzuo, Chaofan Lin, Weixuan Sun +4LLM Pruning Sparse Recovery
†Tsinghua University · ‡Tencent
Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling
Aug 24, 2026Ha Dinh, Xuan Duy Ta, Khoat Than +1LLM Pruning LLM Compression
Viettel AI, Viettel Group, Vietnam · Hanoi University of Science and Technology, Hanoi, Vietnam
SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†

May 7, 2026J/K move · Enter open · S save

SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask

Liu Hanzuo, Chaofan Lin, Weixuan Sun, Yulong Wang+3

†Tsinghua University · ‡Tencent

LLM Pruning Sparse Recovery Structured Sparsity Neural Network Pruning

Open paper AskPDF arXiv

Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers from substantial quality degradation due to strong structural coupling. Existing methods rely on large-scale sparse retraining to recover accuracy, resulting in high computational cost. We propose SparseForge, a post-training framework that improves recovery efficiency by directly optimizing the sparsity mask rather than scaling up retraining tokens. SparseForge combines Hessian-aware importance estimation with progressive annealing of soft masks into hardware-executable structured sparsity, enabling stable and efficient sparse recovery. On LLaMA-2-7B under 2:4 sparsity, SparseForge achieves 57.27% average zero-shot accuracy with only

\textbf{5B}

retraining tokens, surpassing the dense model's 56.43% accuracy and approaching the 57.52% result of a state-of-the-art method using

\textbf{40B}

tokens. Such improvements on the accuracy-efficiency trade-off from SparseForge are shown to be consistent across model families.

Abstract

LLM Inference Efficiency

Explore similar work

SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask

Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference