Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention

Serdarcan Dilbaz; Arjun Krishna; Kiran Kamble; Daniel M. Bikel

cs.LGNov 2, 2026

Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention

Authors: Serdarcan Dilbaz, Arjun Krishna, Kiran Kamble, Daniel M. Bikel

Organizations: Writer AI Research, Writer, Inc.

Abstract

Sparse attention cuts the cost of long-context inference by making the attention matrix sparse: each query attends only a subset of the sequence, chosen by a fixed pattern or, in recent production LLMs, learned per query. In DeepSeek Sparse Attention (DSA), the attention of DeepSeek-V3.2 and the GLM-5 family, every layer is preceded by a lightning indexer that scores the whole cache and keeps the k=2048 highest-scoring tokens for the layer to attend. At long context this indexer scan, not the sparse attention it feeds, is where the decode time goes. But adjacent layers select nearly the same tokens, so a layer can attend the selection of the previous indexer layer instead of its own. This removes the dependency: each indexer then runs concurrently with the attention of its layer. The resulting Multi-layer Indexer (MLI) needs no retraining and no new weights. Its default realization changes no kernel, running inside SGLang's production decode graph; an optional fused decode kernel accelerates it further on DeepSeek-V3.2. MLI composes with IndexCache (the cross-layer reuse GLM-5.2 ships as IndexShare): IndexCache drops the indexer from some layers, and MLI hides the scan on the rest. We serve DeepSeek-V3.2 and GLM-5 under IndexCache's protocol at retentions 1/2 to 1/32 (the fraction of layers that keep an indexer) and contexts 10K to 195K. At equal retention the composition decodes faster than IndexCache: up to 14.5% faster on DeepSeek-V3.2 and 9.8% on GLM-5 at 1/2, and 7.9% and 5.1% at 1/4, largest where IndexCache keeps the most indexers and growing with context. At retention 1/r it is faster still than IndexCache at 1/(2r) while keeping twice the fresh selections: at 128K it reaches 1.32x the unmodified model's decode throughput at 1/4, above IndexCache's 1.28x at 1/8. Its LongBench, RULER and LongBench v2 accuracy stays within a few percentage points of IndexCache's.

Figures & tables

Figure 1: Per-layer decode latency of the DSA attention path (DeepSeek-V3.2, H100, one request; kernels timed in isolation). (a) Standard DSA runs the indexer and attention in sequence, so their latencies add; the prefix scan grows with context and comes to dominate. (b) MLI overlaps them on a second stream, so the per-layer time is the bottleneck, not the sum; the full-width bar is the bottleneck and the striped bar the hidden component, the indexer split into scan and top-

k

as in (a). Idealized ceiling; the realized gain is smaller and bandwidth-bound ( Section 5.2 ).

Panel of Figure 2 — Figure 2: Two-stream realization, depth $1$ , one decode step across two full layers. The indexer of layer $\ell$ forks onto the side stream once $\mathbf{h}_{\ell}$ exists and runs under the attention, output projection and feed-forward block of layer $\ell$ ; its selection is joined at the prepare step of layer $\ell{+}1$ , which attends it. The whole pattern is captured once into the decode CUDA graph.

DeepSeek-V3.2	10K		60K		120K		128K
retention	IndexCache	+MLI	IndexCache	+MLI	IndexCache	+MLI	IndexCache	+MLI
none (base, tok/s)	89.1		88.0		85.7		85.5
all (MLI alone)	+21.5		+19.8		+26.1		+25.1
$\nicefrac{{1}}{{2}}$	+11.5	+25.3	+12.3	+25.2	+13.6	+30.1	+13.7	+29.8
$\nicefrac{{1}}{{4}}$	+19.1	+27.3	+20.5	+28.3	+22.8	+32.5	+22.7	+32.3
$\nicefrac{{1}}{{8}}$	+23.4	+28.5	+25.2	+29.8	+27.9	+33.7	+28.0	+33.7

Table 1: Per-request decode throughput at single concurrency. Base row: tokens/s of the unmodified model; all other cells: percentage change over base in the same cell, positive = faster, for IndexCache alone and for IndexCache

+

MLI at the same retention. All arms of a cell run in one job on one node.

\nicefrac{{1}}{{16}}

and

\nicefrac{{1}}{{32}}

extend IndexCache’s published sweep (

\nicefrac{{1}}{{2}}

–

\nicefrac{{1}}{{8}}

). DeepSeek-V3.2

+

MLI cells use the fused-kernel realization ( Section 5.3 ); GLM-5 uses the deferred-stream realization, which is faster there; both are selection-identical to the production top-

k

DeepSeek-V3.2	LongBench		RULER		LongBench v2
retention	IndexCache	+MLI	IndexCache	+MLI	IndexCache	+MLI
none (base, score)	0.449		0.975		0.501
all (MLI alone)	-0.0		-2.1		-0.4
all (MLI alone), distilled	-0.4		-4.7		-4.4
$\nicefrac{{1}}{{2}}$	-0.2	-0.3	-2.7	-4.0	+0.8	+0.4
$\nicefrac{{1}}{{2}}$ distilled	-0.0	-0.8	-1.1	-3.7	-0.8	+0.0

Table 2: Quality at the retentions IndexCache reports, IndexCache and IndexCache

+

MLI side by side. Base row: absolute score of the unmodified model; other cells: percentage change over base, positive = better, with the better arm of each IndexCache/

+

MLI pair in bold. LongBench: macro-F1 over 11 tasks; RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set (one question is

0.2

points, so differences under a point are within its resolution). Distilled rows: indexer-only distillation, same recipe and budget for both methods. Needle retrieval is

1.00

for every arm shown (it collapses only at

\nicefrac{{1}}{{16}}

–

\nicefrac{{1}}{{32}}

, outside this table; see Table 8 ) and decode perplexity is within

\pm 6\%

(

\pm 11\%

on GLM-5) of base; Table 8 has every arm, metric and retention.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Layer schedule with depth $k{=}1$ , drawn for a dense layer followed by two DSA layers. Left: standard DSA. Attention at layer $\ell$ starts only after the indexer of layer $\ell$ has produced $\mathcal{S}_{\ell}$ . Right: Multi-layer Indexer. The indexer of layer $\ell{+}1$ and the attention of layer $\ell$ read the same hidden state and run concurrently (green step); a selection is attended one step after it is produced.

	DeepSeek-V3.2		GLM-5
	10K	128K	10K	195K
base (tok/s)	86.1	82.9	69.2	65.8
MLI depth 1	$+22.0$	$+21.4$	$+13.6$	$+15.8$
MLI depth 1, high-priority stream	$+22.0$	$+21.0$	$+13.5$	$+15.6$
MLI depth 2	$+21.9$	$+21.8$	$+14.2$	$+16.4$
MLI depth 3	$+21.9$	$+21.8$	$+14.8$	$+16.8$

Appendix

Table 3: MLI scheduling ablation with every indexer retained (

r{=}1

), the deferred-stream realization on the same overlay and recipe as Table 1 ( HH_LAG_NO_ALT ). Per-request decode throughput relative to the unmodified model on the same node, drift-gated (base within

2\%

of a closing base), at the two ends of the context range. Deepening the lag window to

k{=}2

3

full layers and giving the indexer a high-priority CUDA stream both leave the gain unchanged, so scheduling is not the limit. The DeepSeek cells here ran on nodes with a lower absolute base than Table 1 (

82.9

–

86.1

\sim 85.5

tok/s), so their percentages run a few percentage points high; the deferred arm reaches the same absolute throughput (

\sim 100.7

tok/s at

128

K, matching Section 5.3 ).

	DeepSeek-V3.2				GLM-5
	10K		128K		10K		128K
	stock	ours	stock	ours	stock	ours	stock	ours
base	89.4	89.4	85.8	85.6	68.4	68.2	66.3	66.1
IndexCache $\nicefrac{{1}}{{2}}$	100.6	99.6	98.2	97.4	74.8	74.0	73.5	72.7
IndexCache $\nicefrac{{1}}{{4}}$	107.3	106.2	106.1	105.0	78.0	77.7	77.2	76.9

Appendix

Table 4: Unmodified SGLang release (stock) vs our build (ours), same job and node, per-request decode tok/s; IndexCache arms use the release’s own index_topk_freq . Closing base sentinels:

0.0

0.2\%

DeepSeek-V3.2	10K		60K		120K		128K
retention	IndexCache	+MLI	IndexCache	+MLI	IndexCache	+MLI	IndexCache	+MLI
none (base, tok/s)	1760.5		494.7		316.3		314.2
requests in flight	64		13		6		6
all (MLI alone)	-2.2		+4.3		+5.7		+9.8
$\nicefrac{{1}}{{2}}$	+3.1	+1.9	+8.1	+10.5	+12.3	+16.3	+12.1	+18.4
$\nicefrac{{1}}{{4}}$	+6.1	+2.7	+13.2	+12.6	+20.7	+20.1	+21.4	+23.6

Appendix

Table 5: Full-KV-cache decode throughput. Base row: aggregate tokens/s of the unmodified model with the request set sized to the KV cache (the number of requests decoding together is given per context); other cells: percentage change over base, positive = faster. Reference is the closing base run of the cell where the opening run was cold by more than

2\%

DeepSeek-V3.2	128K
retention	IndexCache	+MLI
none (base, tok/s)	56.6
all (MLI alone)	+3.9
$\nicefrac{{1}}{{2}}$	+8.9	+10.5
$\nicefrac{{1}}{{4}}$	+13.3	+14.3

Appendix

Table 6: Replication under attention data-parallelism ( --enable-dp-attention --dp-size 8 , IndexCache’s parallelism): per-request decode throughput (tokens/s, single concurrency) relative to the unmodified model in the same cell; positive = faster. Decode throughput is the client-side output tokens / (latency

-

TTFT), which agrees with the server-side metric within

0.5\%

on the TP cells.

DeepSeek-V3.2	128K
retention	IndexCache	+MLI
none (base, tok/s)	65.0
requests in flight	4
all (MLI alone)	+2.3
$\nicefrac{{1}}{{2}}$	+13.6	+12.6
$\nicefrac{{1}}{{4}}$	+21.2	+21.4

Appendix

Table 7: Replication under attention data-parallelism: full-KV-cache aggregate decode throughput relative to the unmodified model; positive = faster.

DeepSeek-V3.2	LongBench	NIAH	PPL	RULER	LongBench v2
base	0.449	1.000	1.167	0.975	0.501
MLI	0.448 (-0.0%)	1.00	1.176 (+0.8%)	0.954 (-2.1%)	0.499 (-0.4%)
MLI (distilled)	0.447 (-0.4%)	1.00	1.203 (+3.1%)	0.929 (-4.7%)	0.479 (-4.4%)
IC 1/2	0.448 (-0.2%)	1.00	1.166 (-0.1%)	0.949 (-2.7%)	0.505 (+0.8%)
IC 1/2 (distilled)	0.448 (-0.0%)	1.00	1.160 (-0.6%)	0.965 (-1.1%)	0.497 (-0.8%)
IC 1/2 +MLI	0.447 (-0.3%)	1.00	1.132 (-3.0%)	0.936 (-4.0%)	0.503 (+0.4%)

Appendix

Table 8: Quality vs the unmodified model (same job, same endpoints, greedy). LongBench: macro-F1 over 11 tasks; NIAH: worst score over haystacks up to 520K characters and both needle modes; PPL: perplexity of the model’s own greedy continuation, geometric mean over 4K–64K inputs and five seeds (a deviation in either direction is drift from the unmodified model); RULER: mean over tasks and lengths 4K–128K; LongBench v2: accuracy on the full 503-question set. Distilled rows use indexer-only span-aware distillation with the same recipe for both methods. A missing entry (–) is an evaluation whose server ran out of memory during prefill in two attempts.


		undistilled			distilled
arm	decode	LongBench	RULER	LBv2	LongBench	RULER	LBv2
IndexCache $\nicefrac{{1}}{{4}}$	+22.7	-0.3	-0.9	-1.6	+0.2	-2.0	-0.8
IndexCache $+$ MLI $\nicefrac{{1}}{{2}}$	+29.8	-0.3	-4.0	+0.4	-0.8	-3.7	+0.0
IndexCache $\nicefrac{{1}}{{8}}$	+28.0	-0.1	-0.7	-0.4	-0.6	-0.9	+0.4
IndexCache $+$ MLI $\nicefrac{{1}}{{4}}$	+32.3	+0.5	-0.4	-0.4	-0.4	-4.0	-0.8

Appendix

Table 9: The halving pair: IndexCache at

\nicefrac{{1}}{{2r}}

and IndexCache

+

MLI at

\nicefrac{{1}}{{r}}

, the latter with twice the fresh selections. Decode: per-request throughput change over base at the longest context of each model (IndexCache

+

MLI at

\nicefrac{{1}}{{r}}

is faster than IndexCache at

\nicefrac{{1}}{{2r}}

, not merely equal); quality: percentage change over base, positive = better.

Figure 4: What adding MLI to IndexCache does at each retention, at the longest measured context of each model. Left: per-request decode speedup over the unmodified model; right: LongBench macro-F1 change (symmetric-log axis so the

\nicefrac{{1}}{{32}}

collapse fits). Light: IndexCache alone; dark: IndexCache

+

MLI at the same retention; hollow: distilled arms.

DeepSeek-V3.2	LongBench	RULER	LongBench v2
IC 1/16 (distilled) $-$ base	-0.4 [-2.2, +1.3]	-1.1 [-3.3, +0.3]	-2.4 [-6.3, +1.6]
IC 1/2 (distilled) $-$ base	-0.0 [-1.8, +1.6]	-1.1 [-3.9, +1.6]	-0.8 [-4.4, +2.8]
IC 1/32 (distilled) $-$ base	-1.1 [-3.0, +0.7]	-6.4 [-9.9, -3.1] ∗	-5.2 [-9.5, -0.8] ∗
IC 1/4 (distilled) $-$ base	+0.2 [-1.7, +2.1]	-2.0 [-4.5, +0.4]	-0.8 [-4.0, +2.4]
IC 1/8 (distilled) $-$ base	-0.6 [-2.4, +1.2]	-0.9 [-2.5, +0.4]	+0.4 [-3.6, +4.4]
IC 1/16 $-$ base	-0.7 [-2.4, +0.9]	-1.4 [-3.6, +0.2]	-5.6 [-10.7, -0.4] ∗

Appendix

Table 10: DeepSeek-V3.2. Paired bootstrap

95\%

intervals for the contrasts in the text (percentage change over the unmodified model’s score;

10{,}000

resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task

\times

length cells, both macro-averaged as in Table 8 ). ∗ : interval excludes zero. Examples per arm: LongBench

1{,}100

, RULER

234

, LongBench v2 as stated in Table 8 .

GLM-5	LongBench	RULER	LongBench v2
IC 1/16 (distilled) $-$ base	-0.1 [-2.2, +2.0]	-7.4 [-10.2, -4.7] ∗	+1.7 [-2.9, +6.2]
IC 1/2 (distilled) $-$ base	-0.2 [-2.2, +1.8]	+2.0 [+0.4, +3.6] ∗	+2.9 [-1.7, +7.9]
IC 1/32 (distilled) $-$ base	-0.8 [-3.0, +1.5]	-14.4 [-18.6, -10.2] ∗	-3.3 [-9.6, +2.5]
IC 1/4 (distilled) $-$ base	-0.8 [-2.9, +1.1]	+0.9 [-2.4, +4.1]	-0.8 [-5.0, +3.3]
IC 1/8 (distilled) $-$ base	-0.4 [-2.4, +1.5]	+1.2 [-1.4, +3.7]	+0.0 [-4.6, +4.6]
IC 1/16 $-$ base	-11.5 [-14.4, -8.5] ∗	-61.3 [-64.9, -57.7] ∗	-20.8 [-27.9, -13.8] ∗

Appendix

Table 11: GLM-5. Paired bootstrap

95\%

intervals for the contrasts in the text (percentage change over the unmodified model’s score;

10{,}000

resamples of examples, the same resample applied to both arms; LongBench resampled within each of its 11 tasks and RULER within each of its 30 task

\times

length cells, both macro-averaged as in Table 8 ). ∗ : interval excludes zero. Examples per arm: LongBench

1{,}100

, RULER

234

, LongBench v2 as stated in Table 8 .

Long-Context Language Model Inference

Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.

See all 237 papers

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Mar 12, 2026Yushi Bai, Qian Dong, Ting Jiang +5LLM Inference Acceleration Sparse Attention
Tsinghua University
LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Oct 3, 2026Zhaohui Wang, Zhixin Pan, Fanxu Meng +1Cross-Layer Attention LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference

Mar 12, 2026J/K move · Enter open · S save

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Yushi Bai, Qian Dong, Ting Jiang, Xin Lv+4

Tsinghua University

LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from

O(L^2)

O(Lk)

. However, the indexer itself retains

O(L^2)

complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82

\times

prefill speedup and 1.48

\times

decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).

Abstract

Long-Context Language Model Inference

Explore similar work

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference