DeepSeek Sparse Attention (DSA) reduces the computational cost of attention through token-level selection. However, its indexer still scores the entire causal prefix for each query, making indexing a dominant source of latency in long-context inference. Notably, the indexer alone can account for more than 50% of total prefill GPU time at long context lengths. To address this bottleneck, we propose ADIndex, an Adaptive and Distillation-driven Indexing method for token-level sparse attention. ADIndex integrates three key components: (i) a layer-wise calibration strategy that adaptively accommodates cross-query retrieval overlap, motivated by our observation of pronounced inter-layer heterogeneity; (ii) a group-level distillation scheme that leverages this heterogeneity to efficiently train adaptive layer-wise group indexers, capturing salient retrieval signals from individual queries; and (iii) a low-latency inference strategy that reduces repeated full-prefix scans through token bank sharing within query groups while retaining query-specific refinement. Experiments on DeepSeek-V3.2 and GLM-5.1 demonstrate that ADIndex preserves near-lossless task performance on LongBench, LongBench v2, and RULER while delivering substantial efficiency gains over standard DSA, including prefill and decoding kernel speedups of up to 6.22x and 3.40x, respectively, at a context length of 256K across the layer-wise reuse configurations, as well as an end-to-end serving speedup of 1.54x at a context length of 128K.
Figures & tables
Figure 1: Motivating observations for ADIndex. (a) Indexer overhead. The plot shows indexer time proportion during prefill and decoding at different context lengths. The indexer can alone account for 54.2% of total prefill GPU time at 128k context length. (b) Layer-wise overlap heterogeneity. The plot shows per-layer maximum union sizes of the original indexer’s Top- K selections ( K=2048 ) for query group sizes G∈{4,8,16} , computed over all complete prefill query groups from 100 sequences of approximately 128K tokens each. Colored vertical dashed lines indicate the corresponding no-overlap upper bounds GK . (c) Distillation-driven coverage gains. The plot shows absolute gains in Top-2048 coverage of the distilled group indexer over the frozen mean-pooling baseline. Coverage measures the fraction of the original DSA indexer’s per-query Top-2048 selections included in the shared bank. The group size G=16 and bank budget P=8K are fixed, while the number of indexing heads in the group indexer varies. Gains are positive in almost all layers and remain robust across head counts. Selected layers from shallow (7, 12), mid (34, 38), and deep (50, 59) parts of the model are highlighted in warm colors, while the remaining layers are shown in gray. Pale warm and cool backgrounds indicate positive and negative gains, respectively; the horizontal dashed line marks zero gain.
Figure 2: Overview of ADIndex . During training, a lightweight group indexer learns the group-level retrieval requirements of neighboring queries through distillation from the frozen DSA indexer. During inference, the trained group indexer constructs a shared token bank through a single full-prefix scan per group. The original DSA indexer then performs query-specific Top- K refinement over the shared bank augmented with newly visible post-anchor tokens.
LongBench
LongBench v2
RULER
Method
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
DSA
51.38
53.89
22.48
67.95
67.33
73.40
56.07
55.21
49.84
55.56
47.91
53.70
51.89
96.72
95.78
95.28
93.69
95.37
HISA
51.74
53.81
22.23
67.82
68.83
74.38
56.47
51.04
48.55
57.22
41.86
51.85
49.50
94.34
85.45
69.03
47.91
74.18
MISA
51.30
53.27
22.56
68.17
58.50
74.32
54.69
45.31
45.34
52.22
40.93
42.59
45.33
96.29
95.08
93.95
84.93
92.56
IndexCache
51.78
52.87
22.57
68.03
63.17
72.84
55.21
52.60
47.91
58.33
44.19
46.30
49.70
96.45
95.76
93.33
90.13
93.92
Table 1: Results (%) on LongBench, LongBench v2, and RULER. We report results for DeepSeek-V3.2 and GLM-5.1. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. The best and second-best results are highlighted in bold and underlined, respectively. Orange-shaded rows indicate ADIndex .
Figure 3: Inference efficiency across different context lengths. We report kernel speedup for prefill and decoding, together with end-to-end inference speedup. All speedups are measured relative to the original DSA baseline.
Context Length
Overall
Training
16K
32K
64K
128K
AVG
DeepSeek-V3.2
w/o training
96.41
95.16
94.63
91.66
94.47
w/ training
96.81 ↑ 0.40
95.81 ↑ 0.65
95.16 ↑ 0.53
93.34 ↑ 1.68
95.28 ↑ 0.81
GLM-5.1
w/o training
95.48
95.30
95.41
93.39
94.90
Table 3: Ablation on group indexer training (RULER, %). Arrows indicate percentage-point changes relative to the variant without training.
Context Length
Overall
Num. Heads
16K
32K
64K
128K
Avg
DeepSeek-V3.2
64
96.58
95.82
95.13
93.28
95.20
32
96.58
96.09
95.56
93.07
95.33
16
96.35
95.97
94.75
92.86
94.98
8
96.81
95.81
95.16
93.34
95.28
Table 4: Ablation on the number of group indexer heads (RULER, %). The best score in each column is highlighted in bold.
Table 5: Indexer scoring speedups ( × ) across reuse configurations: (a) prefill, (b) one MTP target verification round ( R=1 , m=16 ), and (c) the joint prefill and MTP workload ( R=32 , m=16 , a=8 , D=256 ). G is the query group size and P is the token bank budget; 1K=1024 tokens. Column headings give the prefill context length in (a,c) and the keys visible to the first query of the MTP round in (b). All calculations include post-anchor scoring and short-context fallback, with the same verification workload for the baseline and our method.
LongBench
LongBench v2
RULER
Applied to
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
DSA
51.38
53.89
22.48
67.95
67.33
73.40
56.07
55.21
49.84
55.56
47.91
53.70
51.89
96.72
95.78
95.28
93.69
95.37
Prefill only
52.08
53.80
22.55
68.04
67.69
72.96
56.19
58.85
50.16
55.56
49.77
57.41
53.48
96.40
95.54
95.18
93.35
95.12
MTP only
51.39
53.98
22.68
68.08
67.33
72.31
55.96
56.77
49.20
54.44
49.30
53.70
52.09
96.58
95.76
94.72
93.58
95.16
Both
51.84
54.33
22.54
68.24
67.50
72.90
56.23
59.38
48.55
58.89
46.51
54.63
52.68
96.81
95.81
95.16
93.34
95.28
Table 6: Effect of the deployment phase on LongBench, LongBench v2, and RULER performance (%). MTP refers to MTP target verification. DSA uses the original indexer in both stages. Prefill only and MTP only apply our method exclusively during prefill and MTP target verification, respectively, while Both applies it in both stages. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. Within each model, the best score in each column is highlighted in bold.
LongBench
LongBench v2
RULER
Criterion
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
Mean
51.80
54.10
22.65
68.53
66.83
73.70
56.27
57.81
47.59
55.56
48.37
50.93
51.49
96.53
95.91
94.23
92.25
94.73
P90
51.81
53.82
22.55
68.20
68.00
73.92
56.38
57.81
50.16
56.11
51.16
51.85
53.08
96.44
95.98
94.71
93.07
95.05
Max
51.84
54.33
22.54
68.24
67.50
72.90
56.23
59.38
48.55
58.89
46.51
54.63
52.68
96.81
95.81
95.16
93.34
95.28
GLM-5.1
Table 7: Effect of the layer-wise calibration criterion on LongBench, LongBench v2, and RULER performance (%). Mean, P90 , and Max use the mean, 90th percentile, and maximum of the observed reference-index union sizes to determine configuration feasibility, respectively. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. Within each model, the best score in each column is highlighted in bold, including ties.
Context Length
Overall
Representation
16K
32K
64K
128K
Avg
DeepSeek-V3.2
First
92.23
90.16
89.58
85.74
89.43
Last
91.94
89.35
90.12
84.41
88.96
Mean Pooling
96.56
95.47
95.01
92.77
94.95
Max–Min–Mean
96.81
95.81
95.16
93.34
95.28
Table 8: Effect of group representation construction on RULER performance (%). First and Last use the representation of the first and last query in each group, respectively. Avg denotes the mean score across the four context lengths. Within each model, the best score in each column is highlighted in bold.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.