Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce Adamas, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top-k candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to 6.5× self-attention speedup and 1.7× end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.
Figures & tables
Figure 1: Overview of Adamas. Queries Q and keys K are processed by the Hadamard transform, followed by bucketization and 2-bit compression. The compressed query HQ and keys HK are compared using a lightweight distance-based estimator to select the top- k candidate positions. The corresponding original keys and values, KI and VI , are then gathered for sparse attention with the original query Q . All boxed components are implemented as dedicated GPU kernels for efficiency.
Figure 2: Evaluation results on RULER. Adamas exhibits the smallest performance drop compared to full attention while maintaining high sparsity.
Method
Llama-3.1-8B-Instruct (GQA)
Qwen3-8B (GQA)
LongChat-7b-v1.5-32k (MHA)
Average
Quest
86.96
76.92
94.97
86.28
SnapKV
84.86
83.64
86.44
84.98
StreamingLLM
82.32
81.22
48.61
70.72
Adamas (Ours)
97.65
98.04
100.65
98.78
Table 1: Performance comparison on LongBench across models with different attention mechanisms (GQA and MHA). Scores denote normalized score (%) over budgets ranging from 256 to 4096, normalized by full-attention performance. The best results are shown in bold .
Benchmark
1024
2048
4096
Full Attn.
AIME 2024
23
21
20
20
AIME 2025
17
19
21
19
Table 2: Long-reasoning evaluation on AIME 2024 and AIME 2025 with Qwen3-8B in reasoning mode. Scores denote the number of correctly solved problems out of 30.
Figure 3: Kernel-level breakdown of Adamas self-attention compared to full attention (FlashInfer).
Figure 5: Ablation studies on the Hadamard transform, bucketization, and distance metrics.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Empirical distributions of the Hadamard-transformed queries HQ and keys HK on the calibration subset. Both distributions are approximately zero-centered, with most values concentrated around the origin. The dashed lines indicate ±σX , which we use as a simple scale-normalized reference for defining the bucketization thresholds.
Table 3: Sensitivity analysis of σ -based bucketization thresholds on LongBench. We vary the threshold multiplier α∈{0.5,1,1.5,2} and report results on six LongBench tasks. Best scores for each dataset are in bold . The macro-average results indicate that performance is robust to the threshold scale, with 1σ providing a strong default across both models.
Mass Recall (%)
Top- p Set Recall (%)
Budget
Adamas
Quest
Adamas
Quest
256
88.97
83.88
76.49
69.68
512
92.03
88.54
83.80
78.59
1024
94.70
92.27
90.18
86.22
2048
96.93
95.35
95.11
92.45
4096
98.58
97.77
98.18
96.84
Appendix
Table 4: Candidate-selection quality on LongBench. Attention mass recall and top- p set recall ( p=0.9 ) are averaged equally across six datasets. Both Adamas and Quest are evaluated on the same full-attention decoding trajectories. Higher is better.
Model
Methods
64
128
256
512
1024
2048
4096
Llama-3.1-8B-Instruct (GQA)
StreamingLLM
1%
1%
1%
1%
1%
2%
4%
Quest
48%
68%
91%
100%
100%
100%
100%
SnapKV
21%
34%
49%
53%
72%
86%
95%
Adamas (Ours)
45%
91%
100%
100%
100%
100%
100%
Qwen3-8B (GQA)
StreamingLLM
1%
1%
1%
1%
1%
2%
4%
Quest
9%
19%
76%
76%
76%
86%
85%
Appendix
Table 5: Passkey retrieval accuracy ( % ) of Adamas, Quest, SnapKV and StreamingLLM under different token budgets. The best results are in bold . The results highlight the limitations of StreamingLLM’s sliding window design, the budget sensitivity of Quest and SnapKV, and the robustness of Adamas across most budgets. Due to incompatibility between SnapKV and Yarn-Llama-2-7b-128k, we omit the corresponding results.
Figure 7: Llama-3.1-8B-Instruct (GQA): perplexity results of StreamingLLM, Quest, Adamas (Ours) and full attention evaluated on PG19 under varying token budgets. Adamas consistently matches full attention and even shows lower perplexity at larger token budget.
Figure 8: Qwen3-8B (GQA): perplexity results of StreamingLLM, Quest, Adamas (Ours) and full attention evaluated on PG19 under varying token budgets.
Figure 9: LongChat-7b-v1.5-32k (MHA): perplexity results of StreamingLLM, Quest, Adamas (Ours) and full attention evaluated on PG19 under varying token budgets.
Symbol
Description .
b
Batch size
s
Sequence length
h
Hidden dimension
hkv
Hidden dimension of Key and Value
d
Head dimension
n
Number of buckets
Appendix
Table 6: Mathematical notations.
Component
FLOPs
Full attention
QKV projection
2bh(h+2hkv)
A=softmax(QK⊤)
2bsh
AV
2bsh
Output projection
2bh2
Quest
QKV projection
2bh(h+2hkv)
Reduce keys
2Ph
Appendix
Table 7: Computational workload
Component
Read access
Write access
Full attention
QKV projection
bh+h(h+2hkv)+(h+2hkv)
b(h+2hkv)
A=softmax(QK⊤)V
bh+2bshkv
bh
Output projection
bh+h2
bh
Quest
QKV projection
bh+h(h+2hkv)+(h+2hkv)
b(h+2hkv)
Reduce keys
3bh
2Ph
Criticality estimation
P2bsh+bh
Pdbsh
Appendix
Table 8: Memory access
Datasets
Methods / Budget
64
128
256
512
1024
2048
4096
GovReport (34.51)
StreamingLLM
17.20
18.65
20.32
23.26
25.20
28.25
30.86
SnapKV
16.79
19.54
21.93
24.31
26.51
29.34
31.93
Quest
13.73
20.67
28.01
32.38
33.84
34.23
34.90
Adamas (Ours)
25.01
28.47
30.97
32.84
33.29
34.24
34.42
HotpotQA (55.65)
StreamingLLM
45.78
46.84
45.79
45.66
47.77
48.36
51.28
SnapKV
31.77
38.34
43.72
46.04
50.77
55.12
55.10
Appendix
Table 9: Llama-3.1-8B-Instruct: detailed LongBench evaluation results of StreamingLLM, SnapKV, Quest and Adamas (Ours). The best results are in bold , and the second-best results are underlined .
Datasets
Methods / Budget
64
128
256
512
1024
2048
4096
GovReport (33.38)
StreamingLLM
14.27
15.62
18.86
22.39
25.81
28.22
31.37
SnapKV
13.83
16.79
20.42
23.96
26.65
29.76
31.75
Quest
8.47
13.69
21.79
28.60
32.24
32.75
32.81
Adamas (Ours)
18.88
29.38
35.07
34.43
33.98
33.57
33.69
HotpotQA (53.65)
StreamingLLM
36.72
37.77
38.00
38.97
40.60
44.17
49.09
SnapKV
27.37
37.90
36.91
41.84
47.79
50.28
54.26
Appendix
Table 10: Qwen3-8B: detailed LongBench evaluation results of StreamingLLM, SnapKV, Quest and Adamas (Ours). The best results are in bold , and the second-best results are underlined .
Datasets
Methods / Budget
64
128
256
512
1024
2048
4096
GovReport (31.08)
StreamingLLM
1.24
1.28
1.18
1.21
6.18
24.80
27.77
SnapKV
1.55
1.21
3.66
21.11
23.62
26.31
28.47
Quest
3.30
11.93
22.84
27.27
29.89
31.19
31.23
Adamas (Ours)
30.41
30.44
30.32
30.25
31.03
31.10
30.86
HotpotQA (31.01)
StreamingLLM
10.72
12.02
13.90
10.20
14.56
24.80
27.04
SnapKV
10.55
12.16
20.28
35.65
33.90
35.23
33.94
Appendix
Table 11: LongChat-7b-v1.5-32k: detailed LongBench evaluation results of StreamingLLM, SnapKV, Quest and Adamas (Ours). The best results are in bold , and the second-best results are underlined .
Table 14: Latency of Adamas combined with MInference. MInference is applied during prefilling, while Adamas is used during decoding. Results are averaged across all evaluated settings. The relative change is measured against Adamas without MInference.
Token
Context
Adamas
+MInference
Δ Prefill
Adamas
+MInference
Δ Decode
Budget
Length
Prefill
Prefill
Decode
Decode
256
100K
16.544
11.755
−28.9%
0.016001
0.016073
+0.45%
256
128K
25.769
17.316
−32.8%
0.017860
0.017976
+0.65%
512
100K
16.654
11.888
−28.6%
0.016053
0.016153
+0.62%
512
128K
25.783
17.297
−32.9%
0.017784
0.017936
+0.86%
1024
100K
16.687
11.890
−28.7%
0.016207
0.016331
+0.76%
Appendix
Table 15: Detailed latency (s for prfill and s/token for decode) results for combining Adamas with MInference. MInference accelerates the prefill stage, while the decoding stage continues to use Adamas. Δ reports the relative latency change after enabling MInference.
Datasets
Methods / Budget
16
32
64
128
256
512
1024
2048
4096
GovReport
Adamas (Ours)
22.08
28.31
30.41
30.44
30.37
31.18
30.77
31.00
31.07
Adamas w/o Hadamard
0.60
0.59
0.59
0.64
2.06
6.32
14.12
23.53
29.14
Adamas-1bit
17.17
23.67
27.88
30.09
30.72
31.16
31.38
30.84
30.70
(31.12)
Adamas-3bit
24.69
29.01
30.05
30.59
30.83
31.01
30.95
30.71
31.33
Adamas w/ L2 distance
18.27
27.19
30.09
30.45
30.86
30.93
30.96
30.50
30.56
HotpotQA
Adamas (Ours)
24.05
31.65
32.18
32.89
32.30
31.45
33.06
32.56
31.41
Appendix
Table 16: Ablation results evaluated on LongBench with LongChat-7b-v1.5-32k.
Figure 10: NIAH stress test on Adamas (ours), Quest and full attention, using Yarn-Llama-2-7b-128k. Token budgets range from 256 to 2048. Adamas consistently outperforms Quest in all budgets.