CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

Chuxu Song; Zhencan Peng; Jiuqi Wei; Chuanhui Yang

cs.LGMar 30, 2026

CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

Authors: Chuxu Song, Zhencan Peng, Jiuqi Wei, Chuanhui Yang

Organizations: Department of Computer Science, Rutgers University · Oceanbase, Ant Group.

Abstract

Long-context LLMs increasingly rely on extended, reusable prefill prompts for agents and domain Q&A, pushing attention and KV-cache to become the dominant decode-time bottlenecks. While sparse attention reduces computation and transfer costs, it often struggles to maintain accuracy at high sparsity levels due to the inherent distribution shift between Queries and Keys. We propose Centroid-Scoring Attention (CSAttention), a training-free sparse attention method optimized for high-throughput serving of reusable contexts. CSAttention adopts a storage-for-computation strategy tailored to the offline-prefill/online-decode setting: it front-loads computation into a one-time offline prefill phase that can be amortized across multiple queries, while aggressively optimizing per-step decoding latency. Specifically, CSAttention constructs query-centric lookup tables during offline prefill, whose size remains fixed during decoding, and enables online decoding to replace full-context scans with efficient table lookups and GPU-friendly score accumulation. Extensive experiments demonstrate that CSAttention achieves near-identical accuracy to full attention. Under high sparsity (95%) and long-context settings (32K-128K), CSAttention consistently outperforms state-of-the-art sparse attention methods in both model accuracy and inference speed, achieving up to 4.6x inference speedup over the most accurate baseline at a context length of 128K.

Figures & tables

Panel of Figure 1 — Figure 1: Observations. (a/b) Accuracy vs. sparsity/recall on four LongBench tasks (Llama-3.1-8B-Instruct) (c) Heterogeneous subspace: rank-share of accumulated $q\!\cdot\!k$ contributions across $m=8$ subspaces, grouped over Layers 8–10. (d) PCA of queries $Q$ and keys $K$ from Llama-3.1-8B-Instruct (Layer 10, Head 0) shows a distribution shift between $Q$ and $K$ .

Figure 2: CSAttention overview. Prefill (top): build query-centric centroids per subspace, score all keys per centroid, and store fixed Top-

L

(\text{idx},\text{score})

lists. Decode (bottom): pick nearest centroid in each subspace, fetch

m

lists, reduce-by-key, then take Top-K and run standard KV gather/attention.

	LongBench Evaluation Tasks
	MQA-E	MQA-Z	NarQA	M-News	Musiq	Trec	Samsum	TrivQA	P-Ret	Hotpot	G-Rep	LCC	LSHT	VCSum	Avg
Llama-3.1-8B-Instruct
Full	55.54	62.87	29.91	27.16	30.89	72.50	43.75	91.65	100.0	56.16	35.26	64.89	46.00	17.16	52.41
CSAttention	56.02	62.01	30.46	26.38	31.11	71.50	44.16	91.95	99.50	55.94	33.60	63.33	45.00	17.63	52.04
PQCache	52.96	57.57	30.14	16.67	28.69	71.00	40.01	91.82	99.00	55.22	34.01	60.52	43.00	16.38	49.79
H 2 O	40.17	40.01	29.21	23.94	28.08	62.00	41.10	90.32	97.00	53.36	28.32	57.96	23.50	16.50	45.11

Table 1: LongBench accuracy of sparse methods across three models. Abbreviations: MQA-E (multifieldqa_en), MQA-Z (multifieldqa_zh), NarQA (narrativeqa), M-News (multi_news), Musiq (musique), TrivQA (triviaqa), P-Ret (passage_retrieval_en), Hotpot (hotpotqa), G-Rep (gov_report).

Method	Overall	Easy	Hard	Short	Medium	Long
Full	31.0	35.4	28.3	37.2	26.0	30.6
CSAttention	31.2	34.4	29.3	37.8	25.1	32.4
PQCache	29.8	33.3	27.7	37.8	22.3	31.5
H 2 O	29.9	32.9	28.0	33.8	27.9	31.5
SparQ	26.2	27.6	25.4	30.0	22.3	27.8
MagicPig	29.2	29.5	29.0	31.8	26.9	29.4

Table 2: LongBench v2 evaluation results on Llama-3.1-8B.

	MQA-E	MQA-Z	NarQA	M-News	Musique	Trec	Samsum	TrivQA	P-Ret	Hotpot	G-Rep	LCC	LSHT	VCSum	Avg
Full	55.54	62.87	29.91	27.16	30.89	72.50	43.75	91.65	100.00	56.16	35.26	64.89	46.00	17.16	52.41
0.05 step 1	56.02	62.01	30.46	26.38	31.11	71.50	44.16	91.95	99.50	55.94	33.60	63.33	45.00	17.63	52.04
0.05 step 2	53.98	60.91	28.99	26.05	30.09	70.50	42.18	90.05	97.50	54.03	31.56	63.21	45.00	17.32	50.81
0.15 step 4	54.97	62.04	30.28	26.95	30.72	71.00	44.43	91.89	99.00	55.64	33.84	63.12	44.50	16.78	51.80
0.20 step 8	54.99	62.50	29.84	26.41	31.43	72.00	43.98	91.87	98.50	55.51	33.57	62.88	45.50	17.23	51.87

Table 3: Task-level accuracy on LongBench for CSAttention schedules (Llama3 8B). Schedules are denoted as “keep ratio + search period”: 0.05-step-1 keeps 5% tokens and searches every step; 0.15-step-4 keeps 15% and searches every 4 steps; etc. All schedules are accuracy-stable relative to Full; macro-average gaps are

\leq 0.6

points.

Figure 3: Long-context decode efficiency.

Schedule	P50	P90	P99
step 8	0.900	0.920	1.013
step 4	0.947	0.970	1.007
step 1	0.948	0.969	1.006

Table 4: Normalized per-step overhead latency (mean of append+search normalized to 1.00).

Model	Overall	Easy	Hard	Short	Med.	Long
Llama-70B (Full)	36.5	38.6	35.2	46.0	33.0	27.6
Llama-70B (our)	36.4	39.1	34.8	44.0	35.4	25.9
Qwen3-32B (Full)	49.2	53.1	46.8	60.0	41.1	47.2
Qwen3-32B (our)	48.1	51.1	46.2	57.0	40.0	49.4

Table 5: LongBench-v2 with CoT (max gen 2048): accuracy buckets under prolonged decode. ”our” = CSAttention (default schedule).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: All-GPU mode decode efficiency.

Figure 5: Long-context prefill latency. Left: CPU

\leftrightarrow

GPU mode; Right: All-GPU mode.

Parameter	Default / Guidance
Subspaces $m$	$8$ (use $6$ if memory-bound; $12$ for ultra-long contexts)
Centroids $C$	$64$ ( $32$ for compact tables; $128$ for very long $N$ )
Top- $L$	$L{=}\alpha N_{\text{pre}}$ , $\alpha{=}0.2$ (tune $0.10{\sim}0.40$ )
Keep $K$	$K{=}\rho N$ , $\rho{=}0.05$ (95% sparsity)
Recent window $R$	$32$ (increase to $128$ for stronger recency bias)
Subspace weights $w_{b}$	$1$ (learned/confidence weights optional)

Appendix

Table 6: Recommended defaults (per KV head) and tuning guidance.

Explore similar work

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache Offloading Long-Context Language Model Inference
1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group

Jun 3, 2026J/K move · Enter open · S save

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han+1

1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed+1

KV-Cache Offloading Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

Sparse attention reduces compute and memory bandwidth for long-context LLM inference. However, two key challenges remain: (1) KV cache capacity still grows with sequence length, and offloading to CPU memory introduces a PCIe transfer bottleneck; (2) the sparse selection step itself retains

O(T^2)

complexity and can dominate attention cost at long contexts. We propose SparDA, a decoupled sparse attention architecture that introduces a fourth per-layer projection, the Forecast, alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution. Because Forecast is decoupled from the attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi-head selector. SparDA adds

<

0.5% parameters and trains only the Forecast projections by matching the original selector's attention distribution. On two sparse-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1.25

\times

prefill speedup and 1.7

\times

decode speedup over the sparse-attention offload baseline. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5.3

\times

higher decode throughput than the non-offload sparse baseline. Our source code is available at https://github.com/NVlabs/SparDA.

Abstract

Explore similar work

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps