Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · rednote hilab, China
Abstract
Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce Adamas, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top- candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to self-attention speedup and end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.
Figures & tables
| Method | Llama-3.1-8B-Instruct (GQA) | Qwen3-8B (GQA) | LongChat-7b-v1.5-32k (MHA) | Average |
| Quest | 86.96 | 76.92 | 94.97 | 86.28 |
| SnapKV | 84.86 | 83.64 | 86.44 | 84.98 |
| StreamingLLM | 82.32 | 81.22 | 48.61 | 70.72 |
| Adamas (Ours) | 97.65 | 98.04 | 100.65 | 98.78 |
| Benchmark | 1024 | 2048 | 4096 | Full Attn. |
| AIME 2024 | 23 | 21 | 20 | 20 |
| AIME 2025 | 17 | 19 | 21 | 19 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Mass Recall (%) | Top- Set Recall (%) | ||||
| Budget | Adamas | Quest | Adamas | Quest | |
| 256 | 88.97 | 83.88 | 76.49 | 69.68 | |
| 512 | 92.03 | 88.54 | 83.80 | 78.59 | |
| 1024 | 94.70 | 92.27 | 90.18 | 86.22 | |
| 2048 | 96.93 | 95.35 | 95.11 | 92.45 | |
| 4096 | 98.58 | 97.77 | 98.18 | 96.84 | |
| Model | Methods | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 |
| Llama-3.1-8B-Instruct (GQA) | StreamingLLM | 1% | 1% | 1% | 1% | 1% | 2% | 4% |
| Quest | 48% | 68% | 91% | 100% | 100% | 100% | 100% | |
| SnapKV | 21% | 34% | 49% | 53% | 72% | 86% | 95% | |
| Adamas (Ours) | 45% | 91% | 100% | 100% | 100% | 100% | 100% | |
| Qwen3-8B (GQA) | StreamingLLM | 1% | 1% | 1% | 1% | 1% | 2% | 4% |
| Quest | 9% | 19% | 76% | 76% | 76% | 86% | 85% |
| Symbol | Description . |
| Batch size | |
| Sequence length | |
| Hidden dimension | |
| Hidden dimension of Key and Value | |
| Head dimension | |
| Number of buckets |
| Component | FLOPs | |
| Full attention | QKV projection | |
| Output projection | ||
| Quest | QKV projection | |
| Reduce keys |
| Component | Read access | Write access | |
| Full attention | QKV projection | ||
| Output projection | |||
| Quest | QKV projection | ||
| Reduce keys | |||
| Criticality estimation |
| Datasets | Methods / Budget | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 |
| GovReport (34.51) | StreamingLLM | 17.20 | 18.65 | 20.32 | 23.26 | 25.20 | 28.25 | 30.86 |
| SnapKV | 16.79 | 19.54 | 21.93 | 24.31 | 26.51 | 29.34 | 31.93 | |
| Quest | 13.73 | 20.67 | 28.01 | 32.38 | 33.84 | 34.23 | 34.90 | |
| Adamas (Ours) | 25.01 | 28.47 | 30.97 | 32.84 | 33.29 | 34.24 | 34.42 | |
| HotpotQA (55.65) | StreamingLLM | 45.78 | 46.84 | 45.79 | 45.66 | 47.77 | 48.36 | 51.28 |
| SnapKV | 31.77 | 38.34 | 43.72 | 46.04 | 50.77 | 55.12 | 55.10 |
| Datasets | Methods / Budget | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 |
| GovReport (33.38) | StreamingLLM | 14.27 | 15.62 | 18.86 | 22.39 | 25.81 | 28.22 | 31.37 |
| SnapKV | 13.83 | 16.79 | 20.42 | 23.96 | 26.65 | 29.76 | 31.75 | |
| Quest | 8.47 | 13.69 | 21.79 | 28.60 | 32.24 | 32.75 | 32.81 | |
| Adamas (Ours) | 18.88 | 29.38 | 35.07 | 34.43 | 33.98 | 33.57 | 33.69 | |
| HotpotQA (53.65) | StreamingLLM | 36.72 | 37.77 | 38.00 | 38.97 | 40.60 | 44.17 | 49.09 |
| SnapKV | 27.37 | 37.90 | 36.91 | 41.84 | 47.79 | 50.28 | 54.26 |
| Datasets | Methods / Budget | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 |
| GovReport (31.08) | StreamingLLM | 1.24 | 1.28 | 1.18 | 1.21 | 6.18 | 24.80 | 27.77 |
| SnapKV | 1.55 | 1.21 | 3.66 | 21.11 | 23.62 | 26.31 | 28.47 | |
| Quest | 3.30 | 11.93 | 22.84 | 27.27 | 29.89 | 31.19 | 31.23 | |
| Adamas (Ours) | 30.41 | 30.44 | 30.32 | 30.25 | 31.03 | 31.10 | 30.86 | |
| HotpotQA (31.01) | StreamingLLM | 10.72 | 12.02 | 13.90 | 10.20 | 14.56 | 24.80 | 27.04 |
| SnapKV | 10.55 | 12.16 | 20.28 | 35.65 | 33.90 | 35.23 | 33.94 |
| Context length | Model | Method | 256 | 512 | 1024 | 2048 | 4096 | Full |
| 32k | LongChat-7b-v1.5-32k | Quest | 25.95 | 26.17 | 26.55 | 27.48 | 28.65 | 47.28 |
| SnapKV | 32.85 | 33.29 | 34.25 | 36.07 | 39.90 | |||
| Adamas (Ours) | 27.91 | 27.96 | 28.42 | 29.24 | 30.82 | |||
| 32k | Llama-3.1-8B-Instruct | Quest | – | – | – | – | – | 32.35 |
| SnapKV | 19.33 | 19.44 | 20.10 | 21.29 | 23.57 | |||
| Adamas (Ours) | 28.88 | 28.86 | 29.06 | 29.25 | 29.96 |
| Context length | Model | Method | 256 | 512 | 1024 | 2048 | 4096 | Full |
| 48k | LongChat-7b-v1.5-32k | Quest | 14.67 | 14.66 | 14.93 | 15.21 | 15.73 | 24.37 |
| SnapKV | 17.75 | 18.17 | 18.95 | 20.17 | 22.60 | |||
| Adamas (Ours) | 14.23 | 14.27 | 14.51 | 14.95 | 15.77 | |||
| 48k | Llama-3.1-8B-Instruct | Quest | 16.22 | 16.13 | 16.53 | 16.67 | 17.45 | 15.68 |
| SnapKV | 19.06 | 19.28 | 19.81 | 21.25 | – | |||
| Adamas (Ours) | 14.70 | 14.68 | 14.87 | 15.06 | 15.27 |
| Prefill (s) | Decode (s/token) | |||||
| Setting | Adamas | +MInference | Change | Adamas | +MInference | Change |
| 100K | 16.656 | 11.895 | 0.016692 | 0.016768 | ||
| 128K | 25.794 | 17.269 | 0.018559 | 0.018661 | ||
| Average | 21.225 | 14.582 | 0.017625 | 0.017714 | ||
| Token | Context | Adamas | +MInference | Prefill | Adamas | +MInference | Decode |
| Budget | Length | Prefill | Prefill | Decode | Decode | ||
| 256 | 100K | 16.544 | 11.755 | 0.016001 | 0.016073 | ||
| 256 | 128K | 25.769 | 17.316 | 0.017860 | 0.017976 | ||
| 512 | 100K | 16.654 | 11.888 | 0.016053 | 0.016153 | ||
| 512 | 128K | 25.783 | 17.297 | 0.017784 | 0.017936 | ||
| 1024 | 100K | 16.687 | 11.890 | 0.016207 | 0.016331 |
| Datasets | Methods / Budget | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 |
| GovReport | Adamas (Ours) | 22.08 | 28.31 | 30.41 | 30.44 | 30.37 | 31.18 | 30.77 | 31.00 | 31.07 |
| Adamas w/o Hadamard | 0.60 | 0.59 | 0.59 | 0.64 | 2.06 | 6.32 | 14.12 | 23.53 | 29.14 | |
| Adamas-1bit | 17.17 | 23.67 | 27.88 | 30.09 | 30.72 | 31.16 | 31.38 | 30.84 | 30.70 | |
| (31.12) | Adamas-3bit | 24.69 | 29.01 | 30.05 | 30.59 | 30.83 | 31.01 | 30.95 | 30.71 | 31.33 |
| Adamas w/ L2 distance | 18.27 | 27.19 | 30.09 | 30.45 | 30.86 | 30.93 | 30.96 | 30.50 | 30.56 | |
| HotpotQA | Adamas (Ours) | 24.05 | 31.65 | 32.18 | 32.89 | 32.30 | 31.45 | 33.06 | 32.56 | 31.41 |
Long-Context Language Model Inference
Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.
See all 243 papers