Activation sparsity accelerates LLM decoding by skipping unnecessary weight reads and computation. However, SiLU-based LLMs exhibit little natural sparsity, and imposing high sparsity can substantially degrade accuracy. We propose RoShift, a training-free method that transforms activations into sparsity-friendly representations. RoShift introduces lazy shift, which recenters SiLU activation distributions around zero and defers correction to the end of the FFN to obtain additional sparsity. It also reduces sparsification-induced output error through rotation that jointly accounts for activation and weight information. By fusing the corrections required for shifting and rotation, RoShift translates these gains into decoding speedup. At 50% and 60% sparsity, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, across five models. It achieves up to 1.72× decoding speedup over the dense model on an RTX 5090 at 50% sparsity and yields a better accuracy-speedup Pareto frontier than TEAL. Code is available at https://github.com/hulahula3247/RoShift
Figures & tables
Figure 1: Activation distributions of post-SiLU h1 before and after recentering. Light shading denotes the 50% of components removed by magnitude in each representation; dark shading denotes retained components. Dashed lines indicate zero.
Figure 2: Overview of RoShift, which introduces lazy shift and weight-aware rotation, tightly integrating both with efficient correction fusion for activation sparsification.
Diagnostic
No Rotation
PCA
AW-Rot
Leading 50% activation variance
49.51%
83.69%
80.49%
Leading 50% weight energy
49.97%
58.44%
78.92%
Leading 50% output-contribution variance
49.59%
90.77%
94.85%
Removed contribution proxy
4.41%
1.62%
0.89%
Output NMSE
2.11%
1.33%
0.70%
Table 1: Statistics and sparsification errors for No Rotation, PCA, and AW-Rot on Llama-3.1-8B at 50% sparsity. Values are means over the Q/K/V/Gate projections across 32 layers (128 projections). “Leading 50%” denotes the first half of coordinates in descending eigenvalue order (original order for No Rotation), not the Top-K support. Removed contribution proxy is the token-average fraction of total squared activations, weighted by squared weight-row norms, removed by the actual B128 mask.
Figure 3: Layer-wise RMS distributions of FFN inputs before RMSNorm in Llama-3.1-8B, measured over 32×4096 tokens.
Scope
LaRoSA
TEAL
WAS
RoShift
Projection-wise sparsity
△
✓
✓
✓
Layer-wise sparsity
×
×
△
✓
Table 2: Sparsity allocation flexibility.
Method / Model
Llama-3.1-8B
Llama-2-7B
Mistral-7B
Qwen2.5-7B
Qwen3-8B
Avg
Dense (0%)
63.87
47.14
59.67
73.28
76.43
64.08
TEAL 40%
61.83 (-2.05)
45.89(-1.26)
58.56(-1.11)
71.97(-1.32)
74.54 (-1.89)
62.56 (-1.52)
LaRoSA 40%
61.21(-2.66)
44.99(-2.15)
57.83(-1.84)
71.04(-2.25)
74.02(-2.40)
61.82(-2.26)
WINA 40%
61.71(-2.16)
46.39(-0.75)
58.60 (-1.07)
71.26(-2.02)
73.97(-2.45)
62.39(-1.69)
RoShift 40%
62.26(-1.62)
46.03 (-1.12)
59.24(-0.43)
71.86 (-1.42)
75.16(-1.27)
62.91(-1.17)
TEAL 50%
59.21 (-4.67)
44.34(-2.81)
56.95 (-2.72)
70.46 (-2.82)
73.49 (-2.94)
60.89 (-3.19)
Table 3: Accuracy under non-uniform sparsity at matched Transformer MAC-reduction targets, excluding the LM-head. Each model entry reports mean accuracy over 11 benchmarks. Full results are in Appendix A.9 .
GPU
Sparsity
Llama-3.1 8B
Llama-2 7B
Mistral 7B
Qwen2.5 7B
Qwen3 8B
RTX 5090
Dense
100.58(1.00 × )
104.62(1.00 × )
94.66(1.00 × )
96.72(1.00 × )
97.43(1.00 × )
25%
120.03(1.19 × )
121.64(1.16 × )
121.63(1.28 × )
129.81( 1.34 × )
113.52(1.17 × )
50%
159.86(1.59 × )
159.20(1.52 × )
162.66( 1.72 × )
159.37(1.65 × )
146.86(1.51 × )
75%
238.18(2.37 × )
235.79(2.25 × )
246.03( 2.60 × )
239.96(2.48 × )
217.42(2.23 × )
Jetson AGX Orin 64GB
Dense
11.49(1.00 × )
12.58(1.00 × )
12.12(1.00 × )
12.08(1.00 × )
11.34(1.00 × )
25%
14.09( 1.23 × )
14.78(1.17 × )
14.25(1.18 × )
14.62(1.21 × )
13.45(1.19 × )
Table 4: Batch-size-1 decoding throughput (tokens/s; speedup over the dense baseline in parentheses) of RoShift with uniform sparsity across Transformer layers and projections. Measurement details are in Appendix A.5 .
Figure 4: Accuracy-speedup trade-offs for three models on the RTX 5090. Accuracy is averaged over 11 benchmarks at uniform sparsity; speedup is the end-to-end decoding speedup over each model’s Dense baseline.
Figure 5: Decoding latency versus fused-matrix cache size for Llama-3.1-8B at 50% sparsity. The x-axis shows the average number of fused matrices per layer. The GiB annotations exclude the fixed 2 GiB for transition and correction matrices.
Method
40%
50%
60%
Dense
63.87
63.87
63.87
Top-K
58.81
52.44
41.28
WA-Top-K
58.75
52.97
42.57
A-Rot. + Top-K
58.55
53.21
39.40
A-Rot. + WA-Top-K
58.69
52.63
37.60
AW-Rot. + WA-Top-K
59.47
54.34
44.92
Table 5: Accuracy ablation on Llama-3.1-8B at uniform sparsity. Top-K uses activation magnitudes; WA-Top-K also uses weight norms. A-Rot. and AW-Rot. denote activation-only and activation- and weight-aware rotations.
Figure 6: Cumulative decoding speedup over Dense at 50% uniform sparsity.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Three-stage LM-head candidate selection. Only shaded regions are computed. Subsequent computations for pruned candidates are skipped, and their logits are set to −∞ .
Model
Sparsity allocation
Rotation search
Llama-3.1-8B
5.2
0.8
Llama-2-7B
6.3
0.6
Mistral-7B
4.0
0.7
Qwen2.5-7B
4.8
0.7
Qwen3-8B
2.6
0.8
Model average
4.6
0.7
Appendix
Table 13: Approximate search time (hours) per sparsity setting on a single RTX 5090, averaged over 40%, 50%, and 60% sparsity using combined local and server run records.
Model
40%
50%
60%
Llama-3.1-8B
38.78
48.24
57.89
Llama-2-7B
40.00
49.43
59.14
Mistral-7B-v0.1
39.89
49.42
59.25
Qwen2.5-7B
38.81
48.18
57.94
Qwen3-8B-Base
38.39
47.89
57.39
Model average
39.17
48.63
58.32
Appendix
Table 14: Measured Transformer MAC sparsity (%) of TEAL, averaged over 11 benchmarks. Column headers indicate the target MAC sparsity.
Figure 8: PPL changes versus relative RMS tolerance for Llama-3.1-8B at 0% (left) and 50% (right) sparsity, evaluated on WikiText ( Merity et al., 2016 ) and Mixed Calibration. Δ PPL is the difference from the 0%-tolerance result in each setting.
Figure 9: End-to-end accuracy–speedup trade-offs for TEAL and RoShift with dense and rotated sparse LM-heads on an RTX 5090 at batch size 1. Accuracy is averaged over 11 benchmarks; labels indicate target uniform sparsity.
Figure 10: (a) Decoding speedup over Dense and (b) mean correction fallback rate of Llama-3.1-8B across generation lengths. Measurements used an RTX 5090, FP16, batch size 1, and 50% uniform sparsity for RoShift. The 16 prompts comprise the four original prompts plus three additional prompts from each of the four generation benchmarks. Decoding speed was measured over three repetitions per configuration, excluding dense prefill time. Fallback rates were obtained from separate replay diagnostics.
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications
The Hong Kong University of Science and Technology(Guangzhou) · Emory University · ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems