Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

Jungseob Lee; Seungyoon Lee; Seongtae Hong; Sugyeong Eo; Heuiseok Lim

cs.LGSep 27, 2026

Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

Authors: Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, Heuiseok Lim

Organizations: Korea University · Yonsei University Mirae Campus

Abstract

At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with context. Activation sparsity trims the first term and KV-cache sparsity the second, yet their reported speedups are hard to compare because each depends on context length and on the dense attention kernel it is measured against. We derive a byte crossover, the context length at which the two savings are equal, together with ideal speedup bounds for each branch and for their composition, from model dimensions and keep ratios alone. We then time both branches and their composition from 2K to 128K tokens on two GPUs after a dense prefill of real text, with dense and sparse modes reading the cache through the same split-K attention kernel. The projection branch leads at short context and the KV branch at long context, with speedups that follow their byte bounds up to fixed kernel costs. Adding these costs, measured in separate sweeps, lets the byte account predict the measured crossings of three keep-ratio pairs, a second model, and a second GPU to within 4.1K tokens. Timing the dense baseline with masked instead of split-K attention inflates the apparent speedup of the same KV policy about fivefold. An attention-scored KV selection answers the same passkey and multi-key placements as dense decoding up to 127K tokens, whereas a KV window misses most of them. Under matched perplexity budgets, activation sparsity composed with this selection decodes 14 to 26% faster than the best single branch on both GPUs. Code is available at https://github.com/js-lee-AI/ByteCross.

Figures & tables

Model	$d$	$d_{\text{ff}}$	$n^{*}$	$n^{*}_{\text{FF}}$	Window
Qwen3-8B	4096	12288	65.7K	51.4K	40K
Llama-3.1-8B	4096	14336	74.3K	60.0K	128K
Mistral-7B	4096	14336	74.3K	60.0K	32K
Llama-3.1-70B	8192	28672	291.4K	240.0K	128K

Table 1: Byte crossover lengths at 50% projection keep and 30% KV keep, for all seven projections and for the feed-forward projections alone. Window is the released context length.

Model	$r_{\text{P}}\,/\,r_{\text{KV}}$	$n^{*}$	Pred.	Measured
A100-SXM4
8B	0.7 / 0.3	44.6	28.6	25.1
8B	0.5 / 0.3	74.3	51.9	50.4
8B	0.5 / 0.5	104.0	72.4	68.3
3B	0.5 / 0.3	34.3	13.9	16.5
RTX A6000

Table 2: Byte crossover

n^{*}

, cost-aware prediction, and measured crossing, in K tokens. Each crossing is measured in fresh-process blocks, five on A100-SXM4 and four on RTX A6000.

		Passkey			Multi-key
Configuration	$\Delta$ PPL	32K	64K	127K	32K	64K	127K
Dense	0	40/40	40/40	40/40	40/40	40/40	36/40
KV window, 30%	0.06	16/40	16/40	16/40	16/40	16/40	14/40
Selection, 30%	$<$ 0.01	40/40	40/40	40/40	40/40	40/40	36/40
Selection, 20%	$<$ 0.01	40/40	40/40	40/40	40/40	40/40	36/40
Proj. 50% + sel. 20%	0.96	40/40	40/40	40/40	40/40	40/40	36/40

Table 3: Perplexity increase

\Delta

PPL over dense decoding at 32K and retrieval accuracy on Llama-3.1-8B-Instruct. Dense perplexity is 6.09.

Budget	$r_{\text{P}}\,/\,r_{\text{KV}}$	$\Delta$ PPL	dense	best single
			Speedup over
0.1	1.0 / 0.2	0.004	1.20 $\times$	1.00 $\times$
0.2	0.7 / 0.2	0.146	1.53 $\times$	1.25 $\times$
0.5	0.6 / 0.2	0.402	1.69 $\times$	1.26 $\times$
1.0	0.5 / 0.2	0.963	1.86 $\times$	1.25 $\times$

Table 4: Fastest configuration within each perplexity budget at 32K on RTX A6000, timed in five independent blocks. The KV branch is the attention-scored selection, a keep ratio of 1 disables a branch, and shaded rows select a composition.

Policy	A100-SXM4	RTX A6000
Always projection	1.270	1.518
Always KV selection	1.088	1.101
Always both	1.401	1.694
Calibrated gate	1.404	1.716

Table 5: Decode speedup over dense decoding of each dispatch policy on the request mix. Requests that use the selection also pay for its scoring pass.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Decode only
Setting	Dense	Sparse	Speedup
Llama-3.1-8B-Instruct, A100	98.3	134.5	1.37 $\times$
Llama-3.1-8B-Instruct, RTX A6000	44.3	75.3	1.70 $\times$
Mistral-7B-v0.3, RTX A6000	48.0	82.2	1.71 $\times$
Prefill and decode, Llama-3.1-8B-Instruct on A100
Prompt of about 100 words	70.37	87.89	1.25 $\times$

Appendix

Table 6: Projection-branch throughput in tok/s at 50% keep (FP16, batch size 1, compiled decoding). Llama decode-only rows average five launches with standard deviation at most 2.3% of the mean, and other rows are single runs.

Metric	Dense	FF-only	Full
MMLU, 5-shot accuracy (%)	68.4	67.7	67.7
WikiText-2 chunked perplexity	7.20	7.59	7.68

Appendix

Table 7: Task quality of the projection branch at 50% keep on Llama-3.1-8B-Instruct. FF-only sparsifies the feed-forward projections, and Full also sparsifies the attention projections, as the timed branch does.

	RTX A6000		A100		Perplexity
Keep	tok/s	Speedup	tok/s	Speedup	FF-only	Full
100%	43.8	1.00 $\times$	98.1	1.00 $\times$	7.20	7.20
70%	56.7	1.29 $\times$	116.5	1.19 $\times$	7.25	7.27
60%	63.1	1.44 $\times$	126.2	1.29 $\times$	7.36	7.40
50%	71.8	1.64 $\times$	135.1	1.38 $\times$	7.59	7.68
40%	82.7	1.89 $\times$	142.6	1.45 $\times$	8.10	8.30

Appendix

Table 8: Keep-ratio sweep of the full projection branch. Speed is decode only, from a single launch of each cell, and perplexity uses the chunked protocol for both variants.

	Share of step reads (%)				Saved (MB)
Context	MLP	Attn	KV	Other	MLP	KV
512	71.5	19.9	0.5	8.2	5436	53
1K	71.1	19.8	1.0	8.1	5436	106
2K	70.4	19.6	2.0	8.1	5436	211
4K	69.1	19.2	3.8	7.9	5436	423
8K	66.5	18.5	7.4	7.6	5436	846

Appendix

Table 9: Qwen3-8B decode-step accounting in BF16. The last two columns give the megabytes that the FF-only branch at 50% keep and the KV branch at 30% keep remove from each step.

	A100-SXM4				RTX A6000
Context	Proj.	Sel.	Window	Both	Proj.	Sel.	Window	Both
2K	2.05	$-$ 0.13	0.98	2.17	1.29	0.02	0.66	1.82
4K	1.95	$-$ 0.16	1.02	2.17	1.01	$-$ 0.08	0.70	2.06
8K	2.13	$-$ 0.03	1.53	2.29	1.53	$-$ 0.03	0.55	1.89
16K	2.25	0.14	2.01	2.36	1.18	0.17	0.66	1.92
32K	2.11	0.34	2.08	2.35	1.10	0.20	0.81	2.11

Appendix

Table 10: Kernel cost beyond the byte time in ms, the measured step time minus the dense step time scaled by the byte ratio, averaged over the three blocks of each context sweep of Llama-3.1-8B after a real-text prefill.

	Dense throughput (tok/s)			KV-window speedup
Context	Masked	Fused	Split-K	Masked	Fused	Split-K	Bound
2K	67.7	83.4	94.0	1.37	1.11	0.90	1.01
4K	52.3	72.3	92.8	1.69	1.22	0.92	1.02
8K	36.1	60.0	89.7	2.25	1.35	0.92	1.05
16K	22.0	45.0	84.8	3.21	1.57	0.95	1.10
32K	12.1	28.1	76.3	4.66	2.00	1.02	1.18

Appendix

Table 11: Dense throughput in tok/s and KV-window speedup over dense decoding under three compiled attention paths on A100-SXM4, with the byte bound of the window.

Context	Projection	Selection	Difference	95% interval	Faster
Llama-3.1-8B, keep 0.7 / 0.3, byte crossover 44.6K, crossing 25.1K [23.6K, 28.7K]
20K	10.65	10.98	$-$ 0.328	[ $-$ 0.409, $-$ 0.255]	0/5
24K	11.13	11.14	$-$ 0.009	[ $-$ 0.062, 0.032]	3/5
28K	11.53	11.50	0.025	[ $-$ 0.063, 0.112]	3/5
32K	11.87	11.66	0.212	[0.126, 0.337]	5/5
36K	12.24	11.63	0.608	[0.508, 0.687]	5/5

Appendix

Table 12: Fresh-process step latency in ms on A100-SXM4, averaged over five blocks, with paired differences. Positive differences favor the KV selection, intervals enumerate all block resamples, and the last column counts the blocks in which the selection is faster.

Context	Projection	Selection	Difference	95% interval	Faster
Llama-3.1-8B, keep 0.7 / 0.3, byte crossover 44.6K, crossing 39.3K [38.9K, 40.3K]
28K	22.56	24.09	$-$ 1.529	[ $-$ 1.587, $-$ 1.451]	0/4
32K	23.22	24.35	$-$ 1.131	[ $-$ 1.232, $-$ 1.029]	0/4
36K	23.94	24.50	$-$ 0.562	[ $-$ 0.718, $-$ 0.402]	0/4
40K	24.92	24.80	0.115	[ $-$ 0.048, 0.277]	3/4
44K	25.60	25.04	0.562	[0.524, 0.600]	4/4

Appendix

Table 13: Fresh-process step latency in ms on RTX A6000, averaged over four blocks, with paired differences as in Table 12 .

Batch size $B$	1	2	4	8	16	64
Union at 50% keep	0.52	0.75	0.92	0.985	0.997	0.999
Union at 70% keep	0.71	0.91	0.986	0.998	0.999	0.999
$n^{*}_{B}$ at 50% keep	71.0K	18.5K	3.0K	0.27K	0.02K	$<$ 0.01K
$n^{*}_{1}/B$ at 50% keep	71.0K	35.5K	17.8K	8.9K	4.4K	1.1K

Appendix

Table 14: Fraction of projection weights read by a batched sparse projection, the union of the channels its sequences activate, and the batched crossover at 30% KV keep for Llama-3.1-8B-Instruct.

	RTX A6000 (48 GB)			A100 (80 GB)			H100 NVL (94 GB)
$B$	tok/s	ms	Scale	tok/s	ms	Scale	tok/s	ms	Scale
1	35.9	27.8	1.0 $\times$	58.8	17.0	1.0 $\times$	39.9	25.0	1.0 $\times$
4	133.5	30.0	3.7 $\times$	208.7	19.2	3.6 $\times$	147.3	27.2	3.7 $\times$
8	242.5	33.0	6.8 $\times$	372.8	21.5	6.3 $\times$	263.0	30.4	6.6 $\times$
16	409.6	39.1	11.4 $\times$	621.6	25.7	10.6 $\times$	455.6	35.1	11.4 $\times$
32	632.8	50.6	17.6 $\times$	958.1	33.4	16.3 $\times$	722.9	44.3	18.1 $\times$

Appendix

Table 15: Dense decode throughput, token latency, and scaling relative to

B{=}1

for Llama-3.1-8B-Instruct in the Hugging Face eager loop, with a 128-token prompt and 64 generated tokens.

$r_{\text{P}}$	$r_{\text{KV}}$	Perplexity	Increase	95% interval	Speedup	Passkey
1.0	1.0	6.091	0.000	[0.000, 0.000]	1.000	40/40
0.7	1.0	6.271	0.180	[0.136, 0.229]	1.197	40/40
0.6	1.0	6.492	0.402	[0.308, 0.491]	1.298	40/40
0.5	1.0	7.086	0.995	[0.799, 1.209]	1.465	40/40
1.0	0.5	6.084	$-$ 0.007	[ $-$ 0.023, 0.006]	1.108	40/40
1.0	0.3	6.094	0.004	[ $-$ 0.009, 0.018]	1.151	40/40

Appendix

Table 16: Complete 32K composition grid with the attention-scored selection as the KV branch. Speedup is the screening ratio to dense decoding on RTX A6000, and a keep ratio of 1 disables the corresponding branch.

		RTX A6000		A100-SXM4
$r_{\text{P}}\,/\,r_{\text{KV}}$	Bound	tok/s	Speedup [95% interval]	tok/s	Speedup [95% interval]
1.0 / 1.0	1.000	34.95	1.000 [1.000, 1.000]	74.43	1.000 [1.000, 1.000]
0.7 / 1.0	1.277	42.65	1.220 [1.215, 1.225]	84.15	1.131 [1.118, 1.145]
0.6 / 1.0	1.407	46.63	1.334 [1.328, 1.341]	88.81	1.193 [1.189, 1.198]
0.5 / 1.0	1.566	51.84	1.483 [1.478, 1.493]	94.36	1.268 [1.254, 1.282]
1.0 / 0.2	1.217	42.11	1.205 [1.204, 1.206]	89.00	1.196 [1.189, 1.205]

Appendix

Table 17: Five-block timing of the seven selected sparse settings and dense decoding at 32K, with the byte bound of each setting. Throughput averages across blocks, and speedup averages paired ratios to the dense cell of each block.

Explore similar work

Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Dynamic Sparse Attention LLM Inference Optimization
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoDynamic Sparse Attention Large Language Model Decoding
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2Dynamic Sparse Attention LLM Inference Optimization
1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1

Feb 9, 2026, cs.LGJ/K move · Enter open · S save

Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

Yifei Gao, Lei Wang, Rong-Cheng Tu, Qixin Zhang+2

University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong+1

Dynamic Sparse Attention LLM Inference Optimization Kv-Cache Management Sparsity Bottlenecks Oracles

Open paper AskPDF arXiv

A core bottleneck in large language model (LLM) inference is the cost of attending over the ever-growing key-value (KV) cache. Although near-oracle top-k KV selection can preserve the quality of dense attention while sharply reducing computation and bandwidth, existing sparse methods generally rely on posterior heuristics, i.e., selectors conditioned on observed attention or proxy scores. Such conditioning introduces posterior bias: it tends to distort true token importance and miss salient tokens, thereby impairing long-range reasoning. To tackle this problem, we propose Pre-hoc Sparsity (PrHS), which selects KV entries before attention scoring and provides explicit accuracy control. Let the attention mass of discarded entries be delta (the dropped mass). Through a marginal-to-mutual-information analysis, we derive an upper bound on the mutual-information loss that depends only on the dropped mass. This relation explains failure modes of posterior heuristics and enables verifiable guarantees by controlling the dropped mass in advance. Within PrHS, we instantiate three orthogonal pre-hoc selectors along the axes of time, depth, and layer. Extensive experiments on LLaMA and Mistral families validate PrHS. Across GSM8K and CoQA, PrHS reduces retrieval overhead by over 90%, achieving 3x higher retrieval sparsity than HShare at matched or better accuracy. It incurs under 1% average degradation on LongBench, lowers attention FLOPs by about 15% versus prior sparse baselines, and yields a 9.9x speedup in attention-operator latency and 2.8x higher throughput on NVIDIA A100-80GB GPUs than the dense baseline.

Abstract

Explore similar work

Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

EntmaxKV: Support-Aware Decoding for Entmax Attention

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference