Sparse attention cuts the cost of long-context inference by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or, in recent production LLMs, learned per query. In DeepSeek Sparse Attention (DSA), the attention of DeepSeek-V3.2 and the GLM-5 family, every layer is preceded by a lightning indexer that scores the whole cache and keeps the k=2048 highest-scoring tokens for the layer to attend. At long context this indexer scan, not the sparse attention it feeds, is where the decode time goes. But adjacent layers select nearly the same tokens, so a layer can attend the selection of the previous indexer layer instead of its own. This removes the dependency: each indexer then runs concurrently with the attention of its layer. The resulting Multi-layer Indexer (MLI) needs no retraining and no new weights. Its default realization changes no kernel, running inside SGLang's production decode graph; an optional fused decode kernel accelerates it further on DeepSeek-V3.2. MLI composes with IndexCache (the cross-layer reuse GLM-5.2 ships as IndexShare): IndexCache drops the indexer from some layers, and MLI hides the scan on the rest. We serve DeepSeek-V3.2 and GLM-5 under IndexCache's protocol at retentions 1/2 to 1/32 (the fraction of layers that keep an indexer) and contexts 10K to 195K. At equal retention the composition decodes faster than IndexCache: up to 14.5% faster on DeepSeek-V3.2 and 9.8% on GLM-5 at 1/2, and 7.9% and 5.1% at 1/4, largest where IndexCache keeps the most indexers and growing with context. At retention 1/r it is faster still than IndexCache at 1/(2r) while keeping twice the fresh selections: at 128K it reaches 1.32x the unmodified model's decode throughput at 1/4, above IndexCache's 1.28x at 1/8. Its LongBench, RULER and LongBench v2 accuracy stays within a few percentage points of IndexCache's.
Figures & tables
Figure 1: Per-layer decode latency of the DSA attention path (DeepSeek-V3.2, H100, one request; kernels timed in isolation). (a) Standard DSA runs the indexer and attention in sequence, so their latencies add; the prefix scan grows with context and comes to dominate. (b) MLI overlaps them on a second stream, so the per-layer time is the bottleneck, not the sum; the full-width bar is the bottleneck and the striped bar the hidden component, the indexer split into scan and top- k as in (a). Idealized ceiling; the realized gain is smaller and bandwidth-bound ( Section 5.2 ).
Figure 2: Two-stream realization, depth 1 , one decode step across two full layers. The indexer of layer ℓ forks onto the side stream once hℓ exists and runs under the attention, output projection and feed-forward block of layer ℓ ; its selection is joined at the prepare step of layer ℓ+1 , which attends it. The whole pattern is captured once into the decode CUDA graph.
DeepSeek-V3.2
10K
60K
120K
128K
retention
IndexCache
+MLI
IndexCache
+MLI
IndexCache
+MLI
IndexCache
+MLI
none (base, tok/s)
89.1
88.0
85.7
85.5
all (MLI alone)
+21.5
+19.8
+26.1
+25.1
\nicefrac12
+11.5
+25.3
+12.3
+25.2
+13.6
+30.1
+13.7
+29.8
\nicefrac14
+19.1
+27.3
+20.5
+28.3
+22.8
+32.5
+22.7
+32.3
\nicefrac18
+23.4
+28.5
+25.2
+29.8
+27.9
+33.7
+28.0
+33.7
Table 1: Per-request decode throughput at single concurrency. Base row: tokens/s of the unmodified model; all other cells: percentage change over base in the same cell, positive = faster, for IndexCache alone and for IndexCache + MLI at the same retention. All arms of a cell run in one job on one node. \nicefrac116 and \nicefrac132 extend IndexCache’s published sweep ( \nicefrac12 – \nicefrac18 ). DeepSeek-V3.2 + MLI cells use the fused-kernel realization ( Section 5.3 ); GLM-5 uses the deferred-stream realization, which is faster there; both are selection-identical to the production top- k .
DeepSeek-V3.2
LongBench
RULER
LongBench v2
retention
IndexCache
+MLI
IndexCache
+MLI
IndexCache
+MLI
none (base, score)
0.449
0.975
0.501
all (MLI alone)
-0.0
-2.1
-0.4
all (MLI alone), distilled
-0.4
-4.7
-4.4
\nicefrac12
-0.2
-0.3
-2.7
-4.0
+0.8
+0.4
\nicefrac12 distilled
-0.0
-0.8
-1.1
-3.7
-0.8
+0.0
Table 2: Quality at the retentions IndexCache reports, IndexCache and IndexCache + MLI side by side. Base row: absolute score of the unmodified model; other cells: percentage change over base, positive = better, with the better arm of each IndexCache/ + MLI pair in bold. LongBench: macro-F1 over 11 tasks; RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set (one question is 0.2 points, so differences under a point are within its resolution). Distilled rows: indexer-only distillation, same recipe and budget for both methods. Needle retrieval is 1.00 for every arm shown (it collapses only at \nicefrac116 – \nicefrac132 , outside this table; see Table 8 ) and decode perplexity is within ±6% ( ±11% on GLM-5) of base; Table 8 has every arm, metric and retention.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Layer schedule with depth k=1 , drawn for a dense layer followed by two DSA layers. Left: standard DSA. Attention at layer ℓ starts only after the indexer of layer ℓ has produced Sℓ . Right: Multi-layer Indexer. The indexer of layer ℓ+1 and the attention of layer ℓ read the same hidden state and run concurrently (green step); a selection is attended one step after it is produced.
DeepSeek-V3.2
GLM-5
10K
128K
10K
195K
base (tok/s)
86.1
82.9
69.2
65.8
MLI depth 1
+22.0
+21.4
+13.6
+15.8
MLI depth 1, high-priority stream
+22.0
+21.0
+13.5
+15.6
MLI depth 2
+21.9
+21.8
+14.2
+16.4
MLI depth 3
+21.9
+21.8
+14.8
+16.8
Appendix
Table 3: MLI scheduling ablation with every indexer retained ( r=1 ), the deferred-stream realization on the same overlay and recipe as Table 1 ( HH_LAG_NO_ALT ). Per-request decode throughput relative to the unmodified model on the same node, drift-gated (base within 2% of a closing base), at the two ends of the context range. Deepening the lag window to k=2 or 3 full layers and giving the indexer a high-priority CUDA stream both leave the gain unchanged, so scheduling is not the limit. The DeepSeek cells here ran on nodes with a lower absolute base than Table 1 ( 82.9 – 86.1 vs ∼85.5 tok/s), so their percentages run a few percentage points high; the deferred arm reaches the same absolute throughput ( ∼100.7 tok/s at 128 K, matching Section 5.3 ).
DeepSeek-V3.2
GLM-5
10K
128K
10K
128K
stock
ours
stock
ours
stock
ours
stock
ours
base
89.4
89.4
85.8
85.6
68.4
68.2
66.3
66.1
IndexCache \nicefrac12
100.6
99.6
98.2
97.4
74.8
74.0
73.5
72.7
IndexCache \nicefrac14
107.3
106.2
106.1
105.0
78.0
77.7
77.2
76.9
Appendix
Table 4: Unmodified SGLang release (stock) vs our build (ours), same job and node, per-request decode tok/s; IndexCache arms use the release’s own index_topk_freq . Closing base sentinels: 0.0 to 0.2% .
DeepSeek-V3.2
10K
60K
120K
128K
retention
IndexCache
+MLI
IndexCache
+MLI
IndexCache
+MLI
IndexCache
+MLI
none (base, tok/s)
1760.5
494.7
316.3
314.2
requests in flight
64
13
6
6
all (MLI alone)
-2.2
+4.3
+5.7
+9.8
\nicefrac12
+3.1
+1.9
+8.1
+10.5
+12.3
+16.3
+12.1
+18.4
\nicefrac14
+6.1
+2.7
+13.2
+12.6
+20.7
+20.1
+21.4
+23.6
Appendix
Table 5: Full-KV-cache decode throughput. Base row: aggregate tokens/s of the unmodified model with the request set sized to the KV cache (the number of requests decoding together is given per context); other cells: percentage change over base, positive = faster. Reference is the closing base run of the cell where the opening run was cold by more than 2% .
DeepSeek-V3.2
128K
retention
IndexCache
+MLI
none (base, tok/s)
56.6
all (MLI alone)
+3.9
\nicefrac12
+8.9
+10.5
\nicefrac14
+13.3
+14.3
Appendix
Table 6: Replication under attention data-parallelism ( --enable-dp-attention --dp-size 8 , IndexCache’s parallelism): per-request decode throughput (tokens/s, single concurrency) relative to the unmodified model in the same cell; positive = faster. Decode throughput is the client-side output tokens / (latency − TTFT), which agrees with the server-side metric within 0.5% on the TP cells.
DeepSeek-V3.2
128K
retention
IndexCache
+MLI
none (base, tok/s)
65.0
requests in flight
4
all (MLI alone)
+2.3
\nicefrac12
+13.6
+12.6
\nicefrac14
+21.2
+21.4
Appendix
Table 7: Replication under attention data-parallelism: full-KV-cache aggregate decode throughput relative to the unmodified model; positive = faster.
DeepSeek-V3.2
LongBench
NIAH
PPL
RULER
LongBench v2
base
0.449
1.000
1.167
0.975
0.501
MLI
0.448 (-0.0%)
1.00
1.176 (+0.8%)
0.954 (-2.1%)
0.499 (-0.4%)
MLI (distilled)
0.447 (-0.4%)
1.00
1.203 (+3.1%)
0.929 (-4.7%)
0.479 (-4.4%)
IC 1/2
0.448 (-0.2%)
1.00
1.166 (-0.1%)
0.949 (-2.7%)
0.505 (+0.8%)
IC 1/2 (distilled)
0.448 (-0.0%)
1.00
1.160 (-0.6%)
0.965 (-1.1%)
0.497 (-0.8%)
IC 1/2 +MLI
0.447 (-0.3%)
1.00
1.132 (-3.0%)
0.936 (-4.0%)
0.503 (+0.4%)
Appendix
Table 8: Quality vs the unmodified model (same job, same endpoints, greedy). LongBench: macro-F1 over 11 tasks; NIAH: worst score over haystacks up to 520K characters and both needle modes; PPL: perplexity of the model’s own greedy continuation, geometric mean over 4K–64K inputs and five seeds (a deviation in either direction is drift from the unmodified model); RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set. Distilled rows use indexer-only span-aware distillation with the same recipe for both methods. A missing entry (–) is an evaluation whose server ran out of memory during prefill in two attempts.
undistilled
distilled
arm
decode
LongBench
RULER
LBv2
LongBench
RULER
LBv2
DeepSeek-V3.2 , 128K
IndexCache \nicefrac14
+22.7
-0.3
-0.9
-1.6
+0.2
-2.0
-0.8
IndexCache + MLI \nicefrac12
+29.8
-0.3
-4.0
+0.4
-0.8
-3.7
+0.0
IndexCache \nicefrac18
+28.0
-0.1
-0.7
-0.4
-0.6
-0.9
+0.4
IndexCache + MLI \nicefrac14
+32.3
+0.5
-0.4
-0.4
-0.4
-4.0
-0.8
Appendix
Table 9: The halving pair: IndexCache at \nicefrac12r and IndexCache + MLI at \nicefrac1r , the latter with twice the fresh selections. Decode: per-request throughput change over base at the longest context of each model (IndexCache + MLI at \nicefrac1r is faster than IndexCache at \nicefrac12r , not merely equal); quality: percentage change over base, positive = better.
Figure 4: What adding MLI to IndexCache does at each retention, at the longest measured context of each model. Left: per-request decode speedup over the unmodified model; right: LongBench macro-F1 change (symmetric-log axis so the \nicefrac132 collapse fits). Light: IndexCache alone; dark: IndexCache + MLI at the same retention; hollow: distilled arms.
DeepSeek-V3.2
LongBench
RULER
LongBench v2
IC 1/16 (distilled) − base
-0.4 [-2.2, +1.3]
-1.1 [-3.3, +0.3]
-2.4 [-6.3, +1.6]
IC 1/2 (distilled) − base
-0.0 [-1.8, +1.6]
-1.1 [-3.9, +1.6]
-0.8 [-4.4, +2.8]
IC 1/32 (distilled) − base
-1.1 [-3.0, +0.7]
-6.4 [-9.9, -3.1] ∗
-5.2 [-9.5, -0.8] ∗
IC 1/4 (distilled) − base
+0.2 [-1.7, +2.1]
-2.0 [-4.5, +0.4]
-0.8 [-4.0, +2.4]
IC 1/8 (distilled) − base
-0.6 [-2.4, +1.2]
-0.9 [-2.5, +0.4]
+0.4 [-3.6, +4.4]
IC 1/16 − base
-0.7 [-2.4, +0.9]
-1.4 [-3.6, +0.2]
-5.6 [-10.7, -0.4] ∗
Appendix
Table 10: DeepSeek-V3.2. Paired bootstrap 95% intervals for the contrasts in the text (percentage change over the unmodified model’s score; 10,000 resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task × length cells, both macro-averaged as in Table 8 ). ∗ : interval excludes zero. Examples per arm: LongBench 1,100 , RULER 234 , LongBench v2 as stated in Table 8 .
GLM-5
LongBench
RULER
LongBench v2
IC 1/16 (distilled) − base
-0.1 [-2.2, +2.0]
-7.4 [-10.2, -4.7] ∗
+1.7 [-2.9, +6.2]
IC 1/2 (distilled) − base
-0.2 [-2.2, +1.8]
+2.0 [+0.4, +3.6] ∗
+2.9 [-1.7, +7.9]
IC 1/32 (distilled) − base
-0.8 [-3.0, +1.5]
-14.4 [-18.6, -10.2] ∗
-3.3 [-9.6, +2.5]
IC 1/4 (distilled) − base
-0.8 [-2.9, +1.1]
+0.9 [-2.4, +4.1]
-0.8 [-5.0, +3.3]
IC 1/8 (distilled) − base
-0.4 [-2.4, +1.5]
+1.2 [-1.4, +3.7]
+0.0 [-4.6, +4.6]
IC 1/16 − base
-11.5 [-14.4, -8.5] ∗
-61.3 [-64.9, -57.7] ∗
-20.8 [-27.9, -13.8] ∗
Appendix
Table 11: GLM-5. Paired bootstrap 95% intervals for the contrasts in the text (percentage change over the unmodified model’s score; 10,000 resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task × length cells, both macro-averaged as in Table 8 ). ∗ : interval excludes zero. Examples per arm: LongBench 1,100 , RULER 234 , LongBench v2 as stated in Table 8 .
Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from O(L2) to O(Lk). However, the indexer itself retains O(L2) complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82× prefill speedup and 1.48× decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer's hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30-2.72 times decode indexer speedups over DSA across 8K-128K contexts, retaining most of LatentIndex's recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.
Zhaohui Wang, Zhixin Pan, Fanxu Meng +1
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses many query heads (for example, 64 on DeepSeek-V3.2) that share the same selected token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of Indexer Sparse Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts. A lightweight router uses cheap block-level statistics to pick a query-dependent subset of only a few active heads, and only those heads run the heavy token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix token with every head to scoring it with only a handful of routed heads, plus a negligible router term computed on a small set of pooled keys. We further introduce a hierarchical variant of MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully green Needle-in-a-Haystack heatmaps up to a 128K-token context and recovers more than 92% of the tokens selected by the DSA indexer per layer. Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.