Semi-structured N:M sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible N:M patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to O(M). As a result, RoI requires 1.5-8.75× fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive N:M sparsity patterns, offering a practical path toward efficient LLM deployment.
Figures & tables
Figure 1: Learnable semi-structured N : M sparsity methods: a) modeling the mask selection process using a categorical distribution over feasible masks, and b) our proposed method by learning to sample subsets without replacement of model parameters. The proposed method is more memory efficient than previous works for most practical N : M sparsity patterns. The memory advantage becomes more pronounced as M increases or when N is around M/2 .
Figure 2: Overview of the RoI framework for semi-structure sparsity learning via differentiable subset sampling, illustrating the training and inference phases.
Figure 3: RoI demonstrates better scalability than MaskLLM and ProxSparse, with WikiText-2 perplexity (on the Qwen2.5-1.5B model) improving more consistently as the number of training tokens increases.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Mask difference comparison between RoI and previous works on the Qwen2.5-7B model. Besides the name of each baseline, place an overlapping percentage indicating the similarity of the produced masks between that baseline and RoI. Bold pixels indicate masks corresponding to retained weights.