Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from O(L2) to O(Lk). However, the indexer itself retains O(L2) complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82× prefill speedup and 1.48× decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
Figures & tables
Figure 1: Benchmark comparison between GLM-5 and GLM-5 + IndexCache. IndexCache removes 50% of indexer computations while maintaining comparable performance across both long-context and reasoning tasks, delivering ∼1.2× end-to-end speedup.
Figure 2
Figure 2: Side-by-side comparison of inference loops. (a) Standard DSA runs the lightning indexer at every layer. (b) IndexCache adds a single conditional branch (red lines): F layers compute and cache fresh indices; S layers reuse the cached indices. Note that Tcache is a temporary buffer holding only the current index tensor; it is overwritten at each F layer and requires no additional GPU memory beyond what standard DSA already allocates.
Figure 4
10K
60K
120K
200K
Prefill time (s) ↓
DSA
0.57
3.38
8.57
19.5
+ IndexCache (1/2)
0.47
2.86
6.57
13.7
+ IndexCache (1/4)
0.45
2.59
5.66
10.7
Decode throughput, per request (tok/s) ↑
DSA
73.5
67.0
63.0
58.0
Table 1: End-to-end inference performance of the 30B DSA model with IndexCache at two retention ratios. Prefill time : seconds (lower is better). Decode per request : tokens/s under single concurrency (higher is better). Decode full : total tokens/s (higher is better). Decode throughput is reported per GPU.
Figure 3: Relative speedup of IndexCache over the DSA baseline across three inference settings on the 30B model. DSA baseline is normalized to 100%.
Averages
Long-Context
General & Reasoning
Config
Long
G&R
MRCR
GW
LB2
RULER
LCR
AIME
GPQA
LCB
IFB
Original DSA
50.2
74.6
24.5
49.6
45.5
87.9
43.6
91.0
77.6
71.4
58.4
1/2 Unif. IndexCache
47.4
74.3
22.0
46.6
46.0
83.6
38.6
92.2
76.4
69.7
59.0
+Search pattern
50.3
74.4
24.7
49.5
46.3
87.8
43.2
91.9
76.3
71.3
58.2
1/4 Unif. IndexCache
43.0
73.8
17.7
37.2
43.1
79.2
37.8
91.3
75.7
69.4
58.9
+Search pattern
49.9
74.9
25.1
47.4
45.7
87.6
43.8
92.6
78.6
70.0
58.3
Table 2: Training-free IndexCache at 1/2, 1/4, and 1/8 indexer retention. ‘Long’ and ‘G&R’ aggregate benchmark scores. We compare uniform interleaving against searched patterns.
Averages
Long-Context
General & Reasoning
Config
Long
G&R
MRCR
GW
LB2
RULER
LCR
AIME
GPQA
LCB
IFB
Original DSA
51.0
74.2
24.7
49.1
46.9
87.3
47.0
88.8
79.4
70.5
57.9
1/2 Unif. IndexCache
51.6
74.5
23.8
50.2
47.2
87.0
49.8
89.3
76.7
72.2
59.9
w/ searched pattern
50.6
73.6
23.9
48.1
47.1
87.5
46.6
89.6
78.6
68.5
57.7
w/o cross-layer loss
49.8
74.5
24.6
48.3
45.0
87.1
44.0
88.8
79.4
71.7
58.0
1/4 Unif. IndexCache
50.6
74.1
23.7
48.1
46.9
86.1
48.4
89.3
78.0
70.5
58.7
Table 3: Training-aware IndexCache at 1/2 and 1/4 indexer retention with uniform interleaving. w/ searched pattern : the greedy-searched pattern replaces uniform interleaving. w/o cross-layer loss : each indexer is distilled only against its own layer.
Long Avg
MRCR v2
GraphWalks
LongBench v2
RULER
AA-LCR
Original DSA
78.4
71.1
92.7
64.5
97.7
66.2
1/2 Unif. IndexCache
78.1
72.8
90.2
65.1
97.6
64.6
+Searched pattern
78.7
72.3
90.8
66.0
97.3
67.2
1/4 Unif. IndexCache
72.7
65.8
74.9
62.2
96.2
64.6
+Searched pattern
78.0
70.8
90.3
63.7
97.6
67.6
Table 4: Preliminary results on GLM-5 (744B) with training-free IndexCache.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Pairwise top- k index overlap ratio between all layer pairs of the 30B DSA model. Shared blocks according to the greedy-searched 1/4 IndexCache pattern are marked .
Avg
MRCR v2
GraphWalks
RULER
Original DSA
54.0
24.5
49.6
87.9
1/2 Unif. IndexCache
50.7
22.0
46.6
83.6
+Searched pattern
49.8
22.9
43.5
82.9
Appendix
Table 5: Evaluation results of training-free similarity-based searched pattern.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.