Activation sparsity accelerates LLM decoding by skipping unnecessary weight reads and computation. However, SiLU-based LLMs exhibit little natural sparsity, and imposing high sparsity can substantially degrade accuracy. We propose RoShift, a training-free method that transforms activations into sparsity-friendly representations. RoShift introduces lazy shift, which recenters SiLU activation distributions around zero and defers correction to the end of the FFN to obtain additional sparsity. It also reduces sparsification-induced output error through rotation that jointly accounts for activation and weight information. By fusing the corrections required for shifting and rotation, RoShift translates these gains into decoding speedup. At 50% and 60% sparsity, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, across five models. It achieves up to 1.72× decoding speedup over the dense model on an RTX 5090 at 50% sparsity and yields a better accuracy-speedup Pareto frontier than TEAL. Code is available at https://github.com/hulahula3247/RoShift
Figures & tables
Figure 1: Activation distributions of post-SiLU h1 before and after recentering. Light shading denotes the 50% of components removed by magnitude in each representation; dark shading denotes retained components. Dashed lines indicate zero.
Figure 2: Overview of RoShift, which introduces lazy shift and weight-aware rotation, tightly integrating both with efficient correction fusion for activation sparsification.
Diagnostic
No Rotation
PCA
AW-Rot
Leading 50% activation variance
49.51%
83.69%
80.49%
Leading 50% weight energy
49.97%
58.44%
78.92%
Leading 50% output-contribution variance
49.59%
90.77%
94.85%
Removed contribution proxy
4.41%
1.62%
0.89%
Output NMSE
2.11%
1.33%
0.70%
Table 1: Statistics and sparsification errors for No Rotation, PCA, and AW-Rot on Llama-3.1-8B at 50% sparsity. Values are means over the Q/K/V/Gate projections across 32 layers (128 projections). “Leading 50%” denotes the first half of coordinates in descending eigenvalue order (original order for No Rotation), not the Top-K support. Removed contribution proxy is the token-average fraction of total squared activations, weighted by squared weight-row norms, removed by the actual B128 mask.
Figure 3: Layer-wise RMS distributions of FFN inputs before RMSNorm in Llama-3.1-8B, measured over 32×4096 tokens.
Scope
LaRoSA
TEAL
WAS
RoShift
Projection-wise sparsity
△
✓
✓
✓
Layer-wise sparsity
×
×
△
✓
Table 2: Sparsity allocation flexibility.
Method / Model
Llama-3.1-8B
Llama-2-7B
Mistral-7B
Qwen2.5-7B
Qwen3-8B
Avg
Dense (0%)
63.87
47.14
59.67
73.28
76.43
64.08
TEAL 40%
61.83 (-2.05)
45.89(-1.26)
58.56(-1.11)
71.97(-1.32)
74.54 (-1.89)
62.56 (-1.52)
LaRoSA 40%
61.21(-2.66)
44.99(-2.15)
57.83(-1.84)
71.04(-2.25)
74.02(-2.40)
61.82(-2.26)
WINA 40%
61.71(-2.16)
46.39(-0.75)
58.60 (-1.07)
71.26(-2.02)
73.97(-2.45)
62.39(-1.69)
RoShift 40%
62.26(-1.62)
46.03 (-1.12)
59.24(-0.43)
71.86 (-1.42)
75.16(-1.27)
62.91(-1.17)
TEAL 50%
59.21 (-4.67)
44.34(-2.81)
56.95 (-2.72)
70.46 (-2.82)
73.49 (-2.94)
60.89 (-3.19)
Table 3: Accuracy under non-uniform sparsity at matched Transformer MAC-reduction targets, excluding the LM-head. Each model entry reports mean accuracy over 11 benchmarks. Full results are in Appendix A.9 .
GPU
Sparsity
Llama-3.1 8B
Llama-2 7B
Mistral 7B
Qwen2.5 7B
Qwen3 8B
RTX 5090
Dense
100.58(1.00 × )
104.62(1.00 × )
94.66(1.00 × )
96.72(1.00 × )
97.43(1.00 × )
25%
120.03(1.19 × )
121.64(1.16 × )
121.63(1.28 × )
129.81( 1.34 × )
113.52(1.17 × )
50%
159.86(1.59 × )
159.20(1.52 × )
162.66( 1.72 × )
159.37(1.65 × )
146.86(1.51 × )
75%
238.18(2.37 × )
235.79(2.25 × )
246.03( 2.60 × )
239.96(2.48 × )
217.42(2.23 × )
Jetson AGX Orin 64GB
Dense
11.49(1.00 × )
12.58(1.00 × )
12.12(1.00 × )
12.08(1.00 × )
11.34(1.00 × )
25%
14.09( 1.23 × )
14.78(1.17 × )
14.25(1.18 × )
14.62(1.21 × )
13.45(1.19 × )
Table 4: Batch-size-1 decoding throughput (tokens/s; speedup over the dense baseline in parentheses) of RoShift with uniform sparsity across Transformer layers and projections. Measurement details are in Appendix A.5 .
Figure 4: Accuracy-speedup trade-offs for three models on the RTX 5090. Accuracy is averaged over 11 benchmarks at uniform sparsity; speedup is the end-to-end decoding speedup over each model’s Dense baseline.
Figure 5: Decoding latency versus fused-matrix cache size for Llama-3.1-8B at 50% sparsity. The x-axis shows the average number of fused matrices per layer. The GiB annotations exclude the fixed 2 GiB for transition and correction matrices.
Method
40%
50%
60%
Dense
63.87
63.87
63.87
Top-K
58.81
52.44
41.28
WA-Top-K
58.75
52.97
42.57
A-Rot. + Top-K
58.55
53.21
39.40
A-Rot. + WA-Top-K
58.69
52.63
37.60
AW-Rot. + WA-Top-K
59.47
54.34
44.92
Table 5: Accuracy ablation on Llama-3.1-8B at uniform sparsity. Top-K uses activation magnitudes; WA-Top-K also uses weight norms. A-Rot. and AW-Rot. denote activation-only and activation- and weight-aware rotations.
Figure 6: Cumulative decoding speedup over Dense at 50% uniform sparsity.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Three-stage LM-head candidate selection. Only shaded regions are computed. Subsequent computations for pruned candidates are skipped, and their logits are set to −∞ .
Model
Sparsity allocation
Rotation search
Llama-3.1-8B
5.2
0.8
Llama-2-7B
6.3
0.6
Mistral-7B
4.0
0.7
Qwen2.5-7B
4.8
0.7
Qwen3-8B
2.6
0.8
Model average
4.6
0.7
Appendix
Table 13: Approximate search time (hours) per sparsity setting on a single RTX 5090, averaged over 40%, 50%, and 60% sparsity using combined local and server run records.
Model
40%
50%
60%
Llama-3.1-8B
38.78
48.24
57.89
Llama-2-7B
40.00
49.43
59.14
Mistral-7B-v0.1
39.89
49.42
59.25
Qwen2.5-7B
38.81
48.18
57.94
Qwen3-8B-Base
38.39
47.89
57.39
Model average
39.17
48.63
58.32
Appendix
Table 14: Measured Transformer MAC sparsity (%) of TEAL, averaged over 11 benchmarks. Column headers indicate the target MAC sparsity.
Figure 8: PPL changes versus relative RMS tolerance for Llama-3.1-8B at 0% (left) and 50% (right) sparsity, evaluated on WikiText ( Merity et al., 2016 ) and Mixed Calibration. Δ PPL is the difference from the 0%-tolerance result in each setting.
Figure 9: End-to-end accuracy–speedup trade-offs for TEAL and RoShift with dense and rotated sparse LM-heads on an RTX 5090 at batch size 1. Accuracy is averaged over 11 benchmarks; labels indicate target uniform sparsity.
Figure 10: (a) Decoding speedup over Dense and (b) mean correction fallback rate of Llama-3.1-8B across generation lengths. Measurements used an RTX 5090, FP16, batch size 1, and 50% uniform sparsity for RoShift. The 16 prompts comprise the four original prompts plus three additional prompts from each of the four generation benchmarks. Decoding speed was measured over three repetitions per configuration, excluding dense prefill time. Fallback rates were obtained from separate replay diagnostics.
Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at much lower bandwidth than local GPU-memory access. Activation sparsity reduces these transfers by skipping weights associated with zero or near-zero activations. However, as more activation contributions are omitted, model quality eventually degrades rapidly, indicating that weights associated with small-magnitude activations collectively influence model quality sharply. In this work, we improve the trade-off between model quality and decoding performance when exploiting activation sparsity. Our key idea is to replace the binary choice of whether or not to read a weight with three options: fully retain it, approximate it using a compressed weight representation, or omit it entirely. SpAx skips weights associated with activations closest to zero, reads approximate weights for smaller-magnitude activations, and reads original weights for the largest-magnitude activations. Smaller-magnitude activations attenuate the errors introduced by approximate weights, while compressed weight representations require fewer bytes to be transferred. With weights offloaded to CPU memory, SpAx speeds up decoding by 3.86X on average (up to 5.57X) with 16-bit weights and 2.06X (up to 2.74X) with 4-bit weights, at a WikiText-2 perplexity increase of at most 10%. With weights offloaded to flash storage, the speedups are 3.31X on average (up to 4.81X) and 1.54X (up to 2.03X).
JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar
Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present Prox, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a 1.99× end-to-end decoding speedup at 70% FFN sparsity, and is compatible with quantization and sparse attention.
Jinyi Liu, Wei Chen, Pengyu Chen +4
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications
Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.
Wenwen Hou, Xinyuan Song, Wenqian Weng +1
The Hong Kong University of Science and Technology(Guangzhou) · Emory University · ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems