cs.LGNov 3, 2026

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention

Authors: Sosuke Kobayashi, Shoichiro Yamaguchi, Seiya Tokui

Organizations: Preferred Networks

Abstract

We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from noncontiguous positions. Each query position selects blocks by scoring their centroids and attends to the original keys and values within them. Queries that select the same block are processed together, so prefill stays fast even though each query chooses its own blocks. Large blocks make centroid scoring and this grouped processing cheap, so each query can attend to more tokens in the same time. Larger blocks with a larger token budget can thus be both faster and more accurate on long-context tasks than smaller blocks with a smaller budget. We evaluate two hybrid LLMs, Qwen3.5-4B and 35B-A3B, on one NVIDIA H200. Including the cost of building and using the index, whole-model wall-clock speedups over dense attention are 2.3-2.4x for prefill and 1.7-2.0x for decoding at 512K tokens, and 2.9-3.0x for prefill at 1M. The 35B-A3B model stays close to dense attention on five RULER tasks through 1M, on LongBench-v2 overall, and on MRCR through 256K. The 4B model loses accuracy mainly on multi-key retrieval, yet outperforms dense Qwen3.5-2B, whose latency is similar at 512K, on all three benchmarks.

Figures & tables

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

    Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference AccelerationLong-Context Language Model Inference

  2. RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

    Sep 17, 2026Chuxu Song, Jiuqi Wei, Zhencan PengLong-Context Language Model InferenceLLM Inference Acceleration