Large Language Models (LLMs) offer strong capabilities but incur high inference costs due to dense computation and memory access. Training-free activation sparsity is a promising approach for efficient LLM inference, yet existing methods often rely solely on activation information and uniform sparsity ratios. This overlooks the critical interplay with weights and inter-block sensitivity variation, leading to suboptimal performance. We identify two key phenomena in modern LLMs: 1) less significant activations may align with highly important weights, and 2) sparsity sensitivity varies non-monotonically across model blocks. We propose Weight-aware Mixed-Granularity Training-free Activation Sparsity (WiSparse), which leverages both activation and weight information for adaptive sparsity allocation. Specifically, we introduce a weight-aware mechanism integrating activation magnitudes with precomputed weight norms to accurately identify salient channels. This is combined with a mixed-granularity allocation scheme: a global budget is distributed across blocks via evolutionary search to protect sensitive regions, then refined within blocks to minimize reconstruction error. We improve sparse kernels and demonstrate effectiveness on three representative models. Notably, at 50% sparsity, WiSparse preserves 97% of Llama3.1's dense performance, surpassing the strongest baseline by 2.23 percentage points while achieving a 21.4% acceleration in end-to-end inference speed. Our research advances the limits of training-free approaches for efficient LLM inference, pushing the boundaries of achievable speedup without training.
Figures & tables
Figure 1: The overall framework of WiSparse. The process starts by calculating importance scores for each layer based on activation values and weight norms. These scores generate sparsity masks to prune less important channels. Block-level sparsity is optimized via an evolutionary search on a small calibration dataset, followed by refinement of layer-level sparsities using a greedy allocation strategy. The final configuration is applied during inference to improve computational efficiency by reducing unnecessary operations.
Figure 2: Distribution of activation and weight magnitudes for the self_attn.o_proj layer in block 17 of Llama-3.1-8B. The plot shows that channels with low activation magnitudes can have high-magnitude weights (e.g., channel 2244), demonstrating the limitations of using activation-only metrics to assess channel importance.
Figure 3: Block-wise sensitivity to sparsification. The plot reports the relative change in validation perplexity ( Δ PPL vs the dense model, in %) when sparsifying one block at a time while keeping all other blocks dense. Curves correspond to 40%, 50%, and 60% sparsity.
Model
Sparsity
Method
SIQA
GSM8K
WiC
HumanEval
MMLU
CSQA
Average
Llama-3.1-8B
0
Baseline
43.19
82.87
52.35
67.07
69.07
78.87
65.57
30
R-Sparse
41.40
82.56
51.41
68.90
68.45
78.21
65.16
TEAL
41.71
83.85
51.25
64.63
68.01
78.54
64.67
Ours
43.76
82.26
52.63
67.07
68.22
78.46
65.40
40
R-Sparse
42.68
80.82
50.66
61.59
66.86
77.56
63.36
TEAL
41.71
82.03
50.47
63.41
67.10
78.13
63.81
Table 1: Accuracy comparison of WiSparse with baseline methods on six tasks. The best performance in each group is shown in bold , while the second-best is underlined .
Figure 4: Achieved TLOPS (left) and end-to-end inference speed in tokens/s (right) under different sparsity levels with WiSparse on Llama‑3.1‑8B, Mistral‑7B, and Qwen‑2.5‑7B.
Method
Sparsity
SIQA
GSM8K
WiC
HumanEval
MMLU
CSQA
Average
Baseline
0
43.19
82.87
52.35
67.07
69.07
78.87
65.57
Activation only
0.5
42.32
74.07
42.01
56.10
63.04
74.28
58.64
+ Weight importance
0.5
43.60
76.42
50.47
60.98
63.56
74.37
61.57
+ Coarse search
0.5
42.99
76.42
51.10
62.20
64.00
75.84
62.10
+ Fine search
0.5
43.14
78.77
50.63
65.85
65.23
77.81
63.57
Table 2: Ablation studies : the effect of different components proposed in the paper. The experiment is conducted over Llama-3.1-8B on average metrics.
Figure 5: Per-block and per-module (self-attention and MLP) sparsity distributions for (a) Llama-3.1-8B and (b) Qwen-2.5-7B, as determined by our search algorithm targeting 50% overall sparsity.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Visualization of the calibrated α across model layers. The left panel displays the α values for attention projection matrices, while the right panel shows them for MLP projection matrices. The x-axis represents the layer index.
Activation sparsity studies how neural networks can use mostly inactive units to reduce computation, memory, and energy while preserving useful representations.
Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not tightly control the realised sparsity, while TopK-based methods such as WINA enforce a fixed sparsity level but use the same sparsity budget for every token. Both also apply the same budget across transformer blocks, despite large differences in block sensitivity. We introduce TopK-Guided, a training-free method that addresses both limitations by combining bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation. Across Llama-2 and Llama-3 models, TopK-Guided consistently improves perplexity and downstream accuracy over TEAL and WINA while preserving essentially the same sparsitydependent projection compute as WINA, with the largest gains at high sparsity. Ablations show that both components provide complementary improvements.
Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.
Wenwen Hou, Xinyuan Song, Wenqian Weng +1
The Hong Kong University of Science and Technology(Guangzhou) · Emory University · ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems
Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present Prox, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a 1.99× end-to-end decoding speedup at 70% FFN sparsity, and is compatible with quantization and sparse attention.
Jinyi Liu, Wei Chen, Pengyu Chen +4
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications