cs.CLOct 21, 2025

Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

Authors: Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang, Xiaoxing Ma, Ran Zhu, Chun Cao, Jingwei Xu

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · rednote hilab, China

Abstract

Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce Adamas, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top-kk candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to 6.5×6.5\times self-attention speedup and 1.7×1.7\times end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

    Oct 30, 2026Ales Kubicek, Andrea Caciolai, Liangzu Peng +7Sparse AttentionLong-Context Language Modeling

  2. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

    Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache OffloadingLong-Context Language Model Inference