cs.LGNov 16, 2026

RoShift: Lazy Shift and Weight-aware Rotation for Activation Sparsity in LLM Inference

Authors: Geunsoo Song, Jiwon Seo, Hoeseok Yang, Youngmin Yi

Organizations: 1Sogang University · 2Santa Clara University

Abstract

Activation sparsity accelerates LLM decoding by skipping unnecessary weight reads and computation. However, SiLU-based LLMs exhibit little natural sparsity, and imposing high sparsity can substantially degrade accuracy. We propose RoShift, a training-free method that transforms activations into sparsity-friendly representations. RoShift introduces lazy shift, which recenters SiLU activation distributions around zero and defers correction to the end of the FFN to obtain additional sparsity. It also reduces sparsification-induced output error through rotation that jointly accounts for activation and weight information. By fusing the corrections required for shifting and rotation, RoShift translates these gains into decoding speedup. At 50% and 60% sparsity, RoShift improves average accuracy over TEAL by 1.60 and 3.29 percentage points, respectively, across five models. It achieves up to 1.72×1.72\times decoding speedup over the dense model on an RTX 5090 at 50% sparsity and yields a better accuracy-speedup Pareto frontier than TEAL. Code is available at https://github.com/hulahula3247/RoShift

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

    Oct 1, 2026JuneHyung Kim, Sankeerth Durvasula, Nandita VijaykumarLLM InferenceLLM Inference Acceleration

  2. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

    Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation SparsityStructured Sparsity

  3. ActTail: Global Activation Sparsity in Large Language Models

    Feb 18, 2026Wenwen Hou, Xinyuan Song, Wenqian Weng +1Activation SparsityLLM Inference Acceleration