DeepSeek Sparse Attention (DSA) reduces the computational cost of attention through token-level selection. However, its indexer still scores the entire causal prefix for each query, making indexing a dominant source of latency in long-context inference. Notably, the indexer alone can account for more than 50% of total prefill GPU time at long context lengths. To address this bottleneck, we propose ADIndex, an Adaptive and Distillation-driven Indexing method for token-level sparse attention. ADIndex integrates three key components: (i) a layer-wise calibration strategy that adaptively accommodates cross-query retrieval overlap, motivated by our observation of pronounced inter-layer heterogeneity; (ii) a group-level distillation scheme that leverages this heterogeneity to efficiently train adaptive layer-wise group indexers, capturing salient retrieval signals from individual queries; and (iii) a low-latency inference strategy that reduces repeated full-prefix scans through token bank sharing within query groups while retaining query-specific refinement. Experiments on DeepSeek-V3.2 and GLM-5.1 demonstrate that ADIndex preserves near-lossless task performance on LongBench, LongBench v2, and RULER while delivering substantial efficiency gains over standard DSA, including prefill and decoding kernel speedups of up to 6.22x and 3.40x, respectively, at a context length of 256K across the layer-wise reuse configurations, as well as an end-to-end serving speedup of 1.54x at a context length of 128K.
Figures & tables
Figure 1: Motivating observations for ADIndex. (a) Indexer overhead. The plot shows indexer time proportion during prefill and decoding at different context lengths. The indexer can alone account for 54.2% of total prefill GPU time at 128k context length. (b) Layer-wise overlap heterogeneity. The plot shows per-layer maximum union sizes of the original indexer’s Top- K selections ( K=2048 ) for query group sizes G∈{4,8,16} , computed over all complete prefill query groups from 100 sequences of approximately 128K tokens each. Colored vertical dashed lines indicate the corresponding no-overlap upper bounds GK . (c) Distillation-driven coverage gains. The plot shows absolute gains in Top-2048 coverage of the distilled group indexer over the frozen mean-pooling baseline. Coverage measures the fraction of the original DSA indexer’s per-query Top-2048 selections included in the shared bank. The group size G=16 and bank budget P=8K are fixed, while the number of indexing heads in the group indexer varies. Gains are positive in almost all layers and remain robust across head counts. Selected layers from shallow (7, 12), mid (34, 38), and deep (50, 59) parts of the model are highlighted in warm colors, while the remaining layers are shown in gray. Pale warm and cool backgrounds indicate positive and negative gains, respectively; the horizontal dashed line marks zero gain.
Figure 2: Overview of ADIndex . During training, a lightweight group indexer learns the group-level retrieval requirements of neighboring queries through distillation from the frozen DSA indexer. During inference, the trained group indexer constructs a shared token bank through a single full-prefix scan per group. The original DSA indexer then performs query-specific Top- K refinement over the shared bank augmented with newly visible post-anchor tokens.
LongBench
LongBench v2
RULER
Method
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
DSA
51.38
53.89
22.48
67.95
67.33
73.40
56.07
55.21
49.84
55.56
47.91
53.70
51.89
96.72
95.78
95.28
93.69
95.37
HISA
51.74
53.81
22.23
67.82
68.83
74.38
56.47
51.04
48.55
57.22
41.86
51.85
49.50
94.34
85.45
69.03
47.91
74.18
MISA
51.30
53.27
22.56
68.17
58.50
74.32
54.69
45.31
45.34
52.22
40.93
42.59
45.33
96.29
95.08
93.95
84.93
92.56
IndexCache
51.78
52.87
22.57
68.03
63.17
72.84
55.21
52.60
47.91
58.33
44.19
46.30
49.70
96.45
95.76
93.33
90.13
93.92
Table 1: Results (%) on LongBench, LongBench v2, and RULER. We report results for DeepSeek-V3.2 and GLM-5.1. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. The best and second-best results are highlighted in bold and underlined, respectively. Orange-shaded rows indicate ADIndex .
Figure 3: Inference efficiency across different context lengths. We report kernel speedup for prefill and decoding, together with end-to-end inference speedup. All speedups are measured relative to the original DSA baseline.
Context Length
Overall
Training
16K
32K
64K
128K
AVG
DeepSeek-V3.2
w/o training
96.41
95.16
94.63
91.66
94.47
w/ training
96.81 ↑ 0.40
95.81 ↑ 0.65
95.16 ↑ 0.53
93.34 ↑ 1.68
95.28 ↑ 0.81
GLM-5.1
w/o training
95.48
95.30
95.41
93.39
94.90
Table 3: Ablation on group indexer training (RULER, %). Arrows indicate percentage-point changes relative to the variant without training.
Context Length
Overall
Num. Heads
16K
32K
64K
128K
Avg
DeepSeek-V3.2
64
96.58
95.82
95.13
93.28
95.20
32
96.58
96.09
95.56
93.07
95.33
16
96.35
95.97
94.75
92.86
94.98
8
96.81
95.81
95.16
93.34
95.28
Table 4: Ablation on the number of group indexer heads (RULER, %). The best score in each column is highlighted in bold.
Table 5: Indexer scoring speedups ( × ) across reuse configurations: (a) prefill, (b) one MTP target verification round ( R=1 , m=16 ), and (c) the joint prefill and MTP workload ( R=32 , m=16 , a=8 , D=256 ). G is the query group size and P is the token bank budget; 1K=1024 tokens. Column headings give the prefill context length in (a,c) and the keys visible to the first query of the MTP round in (b). All calculations include post-anchor scoring and short-context fallback, with the same verification workload for the baseline and our method.
LongBench
LongBench v2
RULER
Applied to
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
DSA
51.38
53.89
22.48
67.95
67.33
73.40
56.07
55.21
49.84
55.56
47.91
53.70
51.89
96.72
95.78
95.28
93.69
95.37
Prefill only
52.08
53.80
22.55
68.04
67.69
72.96
56.19
58.85
50.16
55.56
49.77
57.41
53.48
96.40
95.54
95.18
93.35
95.12
MTP only
51.39
53.98
22.68
68.08
67.33
72.31
55.96
56.77
49.20
54.44
49.30
53.70
52.09
96.58
95.76
94.72
93.58
95.16
Both
51.84
54.33
22.54
68.24
67.50
72.90
56.23
59.38
48.55
58.89
46.51
54.63
52.68
96.81
95.81
95.16
93.34
95.28
Table 6: Effect of the deployment phase on LongBench, LongBench v2, and RULER performance (%). MTP refers to MTP target verification. DSA uses the original indexer in both stages. Prefill only and MTP only apply our method exclusively during prefill and MTP target verification, respectively, while Both applies it in both stages. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. Within each model, the best score in each column is highlighted in bold.
LongBench
LongBench v2
RULER
Criterion
SQA
MQA
Sum
FS
Syn
Code
Avg
Easy
Hard
Short
Medium
Long
Avg
16K
32K
64K
128K
Avg
DeepSeek-V3.2
Mean
51.80
54.10
22.65
68.53
66.83
73.70
56.27
57.81
47.59
55.56
48.37
50.93
51.49
96.53
95.91
94.23
92.25
94.73
P90
51.81
53.82
22.55
68.20
68.00
73.92
56.38
57.81
50.16
56.11
51.16
51.85
53.08
96.44
95.98
94.71
93.07
95.05
Max
51.84
54.33
22.54
68.24
67.50
72.90
56.23
59.38
48.55
58.89
46.51
54.63
52.68
96.81
95.81
95.16
93.34
95.28
GLM-5.1
Table 7: Effect of the layer-wise calibration criterion on LongBench, LongBench v2, and RULER performance (%). Mean, P90 , and Max use the mean, 90th percentile, and maximum of the observed reference-index union sizes to determine configuration feasibility, respectively. LongBench categories include Single-Document QA (SQA), Multi-Document QA (MQA), Summarization (Sum), Few-Shot Learning (FS), Synthetic (Syn), and Code Completion (Code). LongBench v2 results are reported for the Easy and Hard difficulty subsets and the Short, Medium, and Long context-length subsets. RULER results are reported at context lengths of 16K, 32K, 64K, and 128K. Avg denotes the mean score across the six task categories for LongBench, the overall accuracy for LongBench v2, and the mean score across the four context lengths for RULER. Within each model, the best score in each column is highlighted in bold, including ties.
Context Length
Overall
Representation
16K
32K
64K
128K
Avg
DeepSeek-V3.2
First
92.23
90.16
89.58
85.74
89.43
Last
91.94
89.35
90.12
84.41
88.96
Mean Pooling
96.56
95.47
95.01
92.77
94.95
Max–Min–Mean
96.81
95.81
95.16
93.34
95.28
Table 8: Effect of group representation construction on RULER performance (%). First and Last use the representation of the first and last query in each group, respectively. Avg denotes the mean score across the four context lengths. Within each model, the best score in each column is highlighted in bold.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from O(L2) to O(Lk). However, the indexer itself retains O(L2) complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82× prefill speedup and 1.48× decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
Sparse attention cuts the cost of long-context inference by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or, in recent production LLMs, learned per query. In DeepSeek Sparse Attention (DSA), the attention of DeepSeek-V3.2 and the GLM-5 family, every layer is preceded by a lightning indexer that scores the whole cache and keeps the k=2048 highest-scoring tokens for the layer to attend. At long context this indexer scan, not the sparse attention it feeds, is where the decode time goes. But adjacent layers select nearly the same tokens, so a layer can attend the selection of the previous indexer layer instead of its own. This removes the dependency: each indexer then runs concurrently with the attention of its layer. The resulting Multi-layer Indexer (MLI) needs no retraining and no new weights. Its default realization changes no kernel, running inside SGLang's production decode graph; an optional fused decode kernel accelerates it further on DeepSeek-V3.2. MLI composes with IndexCache (the cross-layer reuse GLM-5.2 ships as IndexShare): IndexCache drops the indexer from some layers, and MLI hides the scan on the rest. We serve DeepSeek-V3.2 and GLM-5 under IndexCache's protocol at retentions 1/2 to 1/32 (the fraction of layers that keep an indexer) and contexts 10K to 195K. At equal retention the composition decodes faster than IndexCache: up to 14.5% faster on DeepSeek-V3.2 and 9.8% on GLM-5 at 1/2, and 7.9% and 5.1% at 1/4, largest where IndexCache keeps the most indexers and growing with context. At retention 1/r it is faster still than IndexCache at 1/(2r) while keeping twice the fresh selections: at 128K it reaches 1.32x the unmodified model's decode throughput at 1/4, above IndexCache's 1.28x at 1/8. Its LongBench, RULER and LongBench v2 accuracy stays within a few percentage points of IndexCache's.
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses many query heads (for example, 64 on DeepSeek-V3.2) that share the same selected token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of Indexer Sparse Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts. A lightweight router uses cheap block-level statistics to pick a query-dependent subset of only a few active heads, and only those heads run the heavy token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix token with every head to scoring it with only a handful of routed heads, plus a negligible router term computed on a small set of pooled keys. We further introduce a hierarchical variant of MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully green Needle-in-a-Haystack heatmaps up to a 128K-token context and recovers more than 92% of the tokens selected by the DSA indexer per layer. Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.