More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

Noam Elata; Itay Lamprecht; Mikey Shechter; Daniel Ohayon; Itay Hubara; Daniel Soudry

cs.CLOct 3, 2026

More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

Authors: Noam Elata, Itay Lamprecht, Mikey Shechter, Daniel Ohayon, Itay Hubara, Daniel Soudry

Organizations: Technion – Haifa, Israel · Intel, Israel · Corma · Stealth Startup

Abstract

Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Sparse attention methods mitigate this cost by selecting only high-probability entries of the attention matrix. We observe that in many such methods, this renders the probability-value multiplication negligible, shifting the bottleneck to the query-key step. Key heads can therefore be reduced to accelerate inference, while retaining more value heads preserves capacity with limited additional decoding cost. We introduce Sparse Asymmetric Group-Query Attention (SAGA), which decouples key and value head counts to exploit this principle, and pair it with approximate top-N (Atop-N) attention, a simple sparse attention method designed to study the interaction between sparsity and head-count asymmetry. We formalize the benefits of this asymmetry theoretically and validate them empirically through latency measurements and quality evaluations on models up to 1.5B parameters. Together, SAGA and Atop-N achieve end-to-end decoding speedups exceeding $2\times$ over our full-attention GQA baseline at long contexts. Models trained from scratch with SAGA nearly match the quality of comparable GQA variants on the evaluated benchmarks. To facilitate adoption, we introduce an efficient fine-tuning method that converts pretrained models to the SAGA architecture, enabling practitioners to benefit from our approach without costly retraining.

Figures & tables

Figure 1: Diagram of SAGA. There are two significant decoding bottlenecks in attention mechanisms: key-query multiplications and probability-value multiplications (top). SAGA addresses the query-key bottleneck by reducing the number of key heads (left). The probability-value bottleneck is mitigated by utilizing Atop-N to select important values (right).

Figure 2: End-to-End speedup over baseline full attention as a function of sequence length, for Llama 3.2-1B (

h_{k}:8\to 4

, H200) and Qwen2.5-1.5B (

h_{k}:2\to 1

, B200) at various batch sizes. Merging alone (SAGA full) gives a consistent speedup across all configurations. Combining merging with Atop-N attention adds overhead at short sequences but provides substantial gains at long contexts.

Figure 3: Training from scratch on SmolLM2-360M with

h_{q}=16

query heads. Left: validation perplexity (

\downarrow

). Right: validation accuracy (

\uparrow

). SAGA

(h_{k},h_{v})=(1,8)

closely tracks the high-capacity baseline

(8,8)

while outperforming both the matched-key-budget

(1,1)

baseline, and the parameter-matched baseline

(4,4)

. Solid curves are means of 3 seeds; dotted lines indicate

\pm 1

std.

Figure 4: RULER accuracy for full attention and Atop-N attention at

N\in\{512,1024,2048\}

, across context lengths 8K, 16K, and 32K on Qwen2.5-1.5B, for different task categories. SAGA with Atop-N remains near baseline quality, while alternative eviction-based methods degrade substantially at longer contexts.

	Qwen2.5-1.5B ( $h_{k}:2\to 1$ )		Llama 3.2-1B ( $h_{k}:8\to 4$ )
	Baseline	SAGA	Baseline	SAGA
HellaSwag (acc_norm $\uparrow$ )	0.678	0.644	0.643	0.583
WikiText (word ppl $\downarrow$ )	12.10	12.43	11.98	13.09

Table 1: General-capability benchmarks for the teacher and SAGA student models.

$N$	GPQA Diamond @16K ( $\uparrow$ )	GPQA Main @8K ( $\uparrow$ )
Baseline	0.4343	0.4241
2048	0.3939	0.4129
1024	0.3838	0.4330
512	0.3586	0.4241

Table 2: Llama3.1-70B-Instruct accuracy on GPQA with AtopN for varying

N

. GPQA Main is limited to 8K output tokens and GPQA Diamond to 16K.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Reduction	Method	CE	KL
$8\to 4$	Sequential pairing	7.231	4.754
$8\to 4$	Key matching	7.727	5.254
$8\to 4$	Logit matching (ours)	6.748	4.271
$8\to 2$	Sequential pairing	10.306	7.819
$8\to 2$	Key matching	9.077	6.596
$8\to 2$	Logit matching (ours)	7.597	5.125

Appendix

Table 3: Initialization quality before distillation (Llama 3.2-1B). Lower is better for both metrics.

Figure 5: Speedup over baseline full attention on Qwen2.5-1.5B (

h_{k}:2\to 1

) measured on H200 at various batch sizes and sequence lengths.

Configuration	HellaSwag (acc_norm $\uparrow$ )	WikiText (word ppl $\downarrow$ )	Validation accuracy ( $\uparrow$ )
$(h_{k},h_{v})=(1,1)$	$0.3310\pm 0.0047$	34.9129	0.440
$(h_{k},h_{v})=(1,8)$	$0.3343\pm 0.0047$	33.8604	0.443
$(h_{k},h_{v})=(8,8)$	$0.3368\pm 0.0047$	33.3382	0.444

Appendix

Table 4: From-scratch results for a Qwen3-0.6B-based architecture trained on 50B tokens.

Figure 6: Per-task RULER accuracy for Llama 3.2-1B (

h_{k}:8\to 4

) with full attention and Atop-N attention at

N\in\{512,1024,2048\}

, across context lengths 8K, 16K, and 32K.

Kernel	GQA (full)	SAGA (full)	SAGA + Atop-N 1024
Query-key multiplication	870.1	466.0	446.0
Probability-value multiplication	1090.9	1091.1	87.5
Softmax / threshold	249.8	257.3	107.7
Gather	1.2	1.2	88.8
Q/K/V/O projections	22.8	23.3	22.8
MLP	42.7	43.0	43.3

Appendix

Table 5: Decode-step profile of Llama 3.2-1B (

h_{k}:8\to 4

) at 64K context, batch size 16, on H100. Kernel times are

\mu

s per non-exempt layer per decode step; the last rows give end-to-end per-token latency.

	Batch 1	Batch 4	Batch 16
GQA + full attention	46.90	84.99	263.06
SAGA + Atop-N 1024	39.43	68.93	175.06
Speedup	$1.19\times$	$1.23\times$	$1.50\times$

Appendix

Table 6: Per-token decode latency (ms) of the Qwen2.5-32B architecture (

h_{k}:8\to 4

) at 128K context on H100.

Batch	Heads	Attention	8K	16K	32K	64K	128K
1	GQA	full	1.00	1.00	1.00	1.00	1.00
1	GQA	Atop-N	0.93	1.00	1.17	1.43	1.88
1	GQA	exact top-N	0.68	0.63	0.89	1.08	1.39
1	SAGA	full	1.00	1.02	1.05	1.05	1.04
1	SAGA	Atop-N	0.95	1.02	1.23	1.53	2.06
1	SAGA	exact top-N	0.68	0.64	0.92	1.14	1.48

Appendix

Table 7: Decode speedup of Qwen2.5-1.5B relative to GQA with full attention at the same batch size (H100),

N=1024

Explore similar work

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2Dynamic Sparse Attention LLM Inference Optimization
1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Dynamic Sparse Attention Spatial Transcriptomics
The Hong Kong University of Science and Technology · UC Berkeley
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3Dynamic Sparse Attention Alternative
1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University

Jun 3, 2026, cs.CLJ/K move · Enter open · S save

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han+1

1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed+1

Dynamic Sparse Attention LLM Inference Optimization Preemption

Open paper AskPDF arXiv

Sparse attention reduces compute and memory bandwidth for long-context LLM inference. However, two key challenges remain: (1) KV cache capacity still grows with sequence length, and offloading to CPU memory introduces a PCIe transfer bottleneck; (2) the sparse selection step itself retains

O(T^2)

complexity and can dominate attention cost at long contexts. We propose SparDA, a decoupled sparse attention architecture that introduces a fourth per-layer projection, the Forecast, alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution. Because Forecast is decoupled from the attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi-head selector. SparDA adds

<

0.5% parameters and trains only the Forecast projections by matching the original selector's attention distribution. On two sparse-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1.25

\times

prefill speedup and 1.7

\times

decode speedup over the sparse-attention offload baseline. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5.3

\times

higher decode throughput than the non-offload sparse baseline. Our source code is available at https://github.com/NVlabs/SparDA.

Abstract

Explore similar work

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

STS: Efficient Sparse Attention with Speculative Token Sparsity

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel