Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

JuneHyung Kim; Sankeerth Durvasula; Nandita Vijaykumar

cs.LGOct 1, 2026

Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

Authors: JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar

Organizations: LG Electronics · University of Toronto

Abstract

Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at much lower bandwidth than local GPU-memory access. Activation sparsity reduces these transfers by skipping weights associated with zero or near-zero activations. However, as more activation contributions are omitted, model quality eventually degrades rapidly, indicating that weights associated with small-magnitude activations collectively influence model quality sharply. In this work, we improve the trade-off between model quality and decoding performance when exploiting activation sparsity. Our key idea is to replace the binary choice of whether or not to read a weight with three options: fully retain it, approximate it using a compressed weight representation, or omit it entirely. SpAx skips weights associated with activations closest to zero, reads approximate weights for smaller-magnitude activations, and reads original weights for the largest-magnitude activations. Smaller-magnitude activations attenuate the errors introduced by approximate weights, while compressed weight representations require fewer bytes to be transferred. With weights offloaded to CPU memory, SpAx speeds up decoding by 3.86X on average (up to 5.57X) with 16-bit weights and 2.06X (up to 2.74X) with 4-bit weights, at a WikiText-2 perplexity increase of at most 10%. With weights offloaded to flash storage, the speedups are 3.31X on average (up to 4.81X) and 1.54X (up to 2.03X).

Figures & tables

Figure 1: Llama-3.1-8B (BF16) with all weights offloaded, at different sparsity levels. (a), (b): Time per decoded token with the weights offloaded to CPU memory and to flash storage. (c): Perplexity increase relative to dense on the WikiText-2 test set. (See Section 6 for the experiment setup.)

Figure 2: Overview of SpAx. Offline encoding and calibration prepare compressed weight codes and determine tier sizes. During decoding, activation magnitudes determine which columns to read in the original format, reconstruct from compressed codes, or skip.

Figure 3: Reconstructing a middle-tier column from its own

b

-bit codes.

Figure 4: Weight columns of

W_{q}

W_{k}

and

W_{v}

arranged contiguously in flash storage.

	Identifier	Name
MODEL	LLAMA	Llama-3.1-8B ( Grattafiori and others, 2024 )
	GEMMA	Gemma-4-31B ( Gemma Team, 2026 )
	QWEN	Qwen3.6-27B ( Qwen Team, 2026 )
CONFIG	ALL	FFN and attention projection weights offloaded
	CA	FFN weights offloaded, attention projection weights cached in GPU memory
FORMAT	BF16	16-bit

Table 1: The models, configurations and weight formats we evaluate, and their identifiers.

Panel of Figure 5 — Figure 5: Perplexity increase over dense against the budget on WikiText-2 test. The shaded bands mark the two tolerances , a perplexity increase of $10\%$ and $20\%$ .

	Budget	PPL	Arc-C	HS	PIQA	WG	MMLU	GSM8K
LLAMA-BF16 (Dense)	100%	6.24	52.1	60.8	79.1	75.1	65.4	50.2
TEAL-ALL $+$ 10%	57%	6.89 $(+10.4\%)$	48.4	58.5	78.8	72.3	62.3	42.2
SpAx-ALL $+$ 10%	27%	6.87 $(+10.1\%)$	49.1	58.5	78.6	73.4	62.1	43.9
TEAL-ALL $+$ 20%	50%	7.44 $(+19.2\%)$	47.4	56.7	78.0	71.0	59.1	36.0
SpAx-ALL $+$ 20%	22%	7.47 $(+19.7\%)$	48.1	56.9	77.4	70.6	60.1	39.2

Table 2: Downstream accuracy of LLAMA-ALL-BF16 for SpAx and TEAL, the closest baseline in Figure 5 , at the two tolerances. Appendix D gives all models and settings.

Panel of Figure 6 — Figure 6: Decoding speed with the weights offloaded to CPU memory (a, b) and to flash storage (c, d), in the ALL and CA configurations of Table 1 . Each method runs at its budget for each tolerance.

			CPU memory (ms)			Flash (ms)
	needed (GB)	GPU (ms)	transfer	other	total	moved (GB)	transfer	other	total
BF16 (Dense)	13.96	26	1,118	12	1,156	13.96	2,388	60	2,474
TEAL $+$ 10%	7.95	14	637	29	680	7.99	1,382	142	1,538
SpAx-trellis $+$ 10%	3.76	64	302	31	397	4.11	834	134	1,032
SpAx-scalar $+$ 10%	4.83	27	388	34	449	5.25	1,016	146	1,190
Q4KM (Dense)	4.18	18	336	12	366	4.18	825	53	896

Table 3: Breakdown of decoding time per token of LLAMA-ALL in both formats with CPU memory and flash offloading. Appendix C gives all models and settings.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4:

k_{\mathrm{top}}

and

k_{\mathrm{mid}}

of each group at layer 15 of Llama-3.1-8B.

				CPU memory			Flash
		needed	GPU	transfer	other	total	moved	transfer	other	total
		(GB)	(ms)	(ms)	(ms)	(ms)	(GB)	(ms)	(ms)	(ms)
LLAMA-ALL-BF16	Dense	13.96	26	1,118	12	1,156	13.96	2,388	60	2,474
	TEAL $+$ 10%	7.95	14	637	29	680	7.99	1,382	142	1,538
	SpAx-trellis $+$ 10%	3.76	64	302	31	397	4.11	834	134	1,032
	SpAx-scalar $+$ 10%	4.83	27	388	34	449	5.25	1,016	146	1,190

Appendix

Table 5: Breakdown of the decoding time per token of Llama-3.1-8B with CPU memory and flash offloading.

				CPU memory			Flash
		needed	GPU	transfer	other	total	moved	transfer	other	total
		(GB)	(ms)	(ms)	(ms)	(ms)	(GB)	(ms)	(ms)	(ms)
GEMMA-ALL-BF16	Dense	58.57	103	4,690	34	4,827	58.57	9,389	170	9,662
	TEAL $+$ 10%	29.20	44	2,339	74	2,456	30.55	4,868	343	5,255
	SpAx-trellis $+$ 10%	13.30	226	1,067	84	1,377	15.18	2,631	374	3,231
	SpAx-scalar $+$ 10%	21.40	76	1,716	88	1,880	23.77	3,913	396	4,385

Appendix

Table 6: Breakdown of the decoding time per token of Gemma-4-31B with CPU memory and flash offloading.

				CPU memory			Flash
		needed	GPU	transfer	other	total	moved	transfer	other	total
		(GB)	(ms)	(ms)	(ms)	(ms)	(GB)	(ms)	(ms)	(ms)
QWEN-CA-BF16	Dense	34.23	58	2,739	35	2,832	34.23	5,485	109	5,652
	TEAL $+$ 10%	14.41	25	1,156	56	1,236	14.90	2,452	205	2,682
	SpAx-trellis $+$ 10%	3.78	145	303	60	508	4.62	836	193	1,174
	SpAx-scalar $+$ 10%	6.87	44	551	65	660	7.79	1,374	230	1,648

Appendix

Table 7: Breakdown of decoding time per token of Qwen3.6-27B with CPU memory and flash offloading.

	Budget	PPL	Arc-C	HS	PIQA	WG	MMLU	GSM8K
LLAMA-BF16 (Dense)	100%	6.24	52.1	60.8	79.1	75.1	65.4	50.2
TEAL-ALL $+$ 10%	57%	6.89 $(+10.4\%)$	48.4	58.5	78.8	72.3	62.3	42.2
SpAx-ALL $+$ 10%	27%	6.87 $(+10.1\%)$	49.1	58.5	78.6	73.4	62.1	43.9
TEAL-ALL $+$ 20%	50%	7.44 $(+19.2\%)$	47.4	56.7	78.0	71.0	59.1	36.0
SpAx-ALL $+$ 20%	22%	7.47 $(+19.7\%)$	48.1	56.9	77.4	70.6	60.1	39.2
TEAL-CA $+$ 10%	46%	6.86 $(+9.9\%)$	49.8	59.5	78.2	71.0	63.3	44.0

Appendix

Table 8: Downstream accuracy of Llama-3.1-8B for SpAx and TEAL at the two tolerances.

	Budget	PPL	Arc-C	HS	PIQA	WG	MMLU
GEMMA-BF16 (Dense)	100%	6.39	64.8	65.8	82.9	80.1	84.5
TEAL-ALL $+$ 10%	50%	7.07 $(+10.7\%)$	63.3	62.8	82.2	77.7	82.4
SpAx-ALL $+$ 10%	23%	7.12 $(+11.5\%)$	63.0	64.0	81.2	79.1	82.8
TEAL-ALL $+$ 20%	44%	7.65 $(+19.8\%)$	59.2	60.8	81.6	76.5	80.6
SpAx-ALL $+$ 20%	18%	7.63 $(+19.4\%)$	61.2	63.2	81.5	76.2	82.0
GEMMA-Q4KM (Dense)	100%	6.54	63.7	65.7	82.8	79.6	83.6

Appendix

Table 9: Downstream accuracy of Gemma-4-31B for SpAx and TEAL at the two tolerances.

	Budget	PPL	Arc-C	HS	PIQA	WG
QWEN-BF16 (Dense)	100%	7.40	60.3	64.0	81.6	77.2
TEAL-CA $+$ 10%	30%	8.18 $(+10.6\%)$	58.7	62.2	80.9	75.3
SpAx-CA $+$ 10%	8%	8.16 $(+10.2\%)$	57.6	62.9	80.5	74.3
TEAL-CA $+$ 20%	25%	8.86 $(+19.7\%)$	56.3	60.2	79.3	73.5
SpAx-CA $+$ 20%	6%	8.69 $(+17.4\%)$	56.4	62.2	80.3	73.2
QWEN-Q4KM (Dense)	100%	7.47	59.0	63.9	81.4	77.0

Appendix

Table 10: Downstream accuracy of Qwen3.6-27B for SpAx and TEAL at the two tolerances.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 285 papers

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
Feb 16, 2026Lei Chen, Yuan Meng, Xiaoyu Zhan +2Activation Sparsity LLM Inference Acceleration
Tsinghua University
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Oct 8, 2026Qitong Wang, Xinwei Niu, Mingluo Su +3LLM Pruning LLM Inference Acceleration
Westlake University · Ant Group
SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM Pruning High-Performance Computing
Snowflake AI Research* · Seoul National University†

Feb 16, 2026J/K move · Enter open · S save

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity

Lei Chen, Yuan Meng, Xiaoyu Zhan, Zhi Wang+1

Tsinghua University

Activation Sparsity LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) offer strong capabilities but incur high inference costs due to dense computation and memory access. Training-free activation sparsity is a promising approach for efficient LLM inference, yet existing methods often rely solely on activation information and uniform sparsity ratios. This overlooks the critical interplay with weights and inter-block sensitivity variation, leading to suboptimal performance. We identify two key phenomena in modern LLMs: 1) less significant activations may align with highly important weights, and 2) sparsity sensitivity varies non-monotonically across model blocks. We propose Weight-aware Mixed-Granularity Training-free Activation Sparsity (WiSparse), which leverages both activation and weight information for adaptive sparsity allocation. Specifically, we introduce a weight-aware mechanism integrating activation magnitudes with precomputed weight norms to accurately identify salient channels. This is combined with a mixed-granularity allocation scheme: a global budget is distributed across blocks via evolutionary search to protect sensitive regions, then refined within blocks to minimize reconstruction error. We improve sparse kernels and demonstrate effectiveness on three representative models. Notably, at 50% sparsity, WiSparse preserves 97% of Llama3.1's dense performance, surpassing the strongest baseline by 2.23 percentage points while achieving a 21.4% acceleration in end-to-end inference speed. Our research advances the limits of training-free approaches for efficient LLM inference, pushing the boundaries of achievable speedup without training.

Abstract

LLM Inference

Explore similar work

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference