Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from O(L2) to O(Lk). However, the indexer itself retains O(L2) complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82× prefill speedup and 1.48× decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
Figures & tables
Figure 1: Benchmark comparison between GLM-5 and GLM-5 + IndexCache. IndexCache removes 50% of indexer computations while maintaining comparable performance across both long-context and reasoning tasks, delivering ∼1.2× end-to-end speedup.
Figure 2
Figure 2: Side-by-side comparison of inference loops. (a) Standard DSA runs the lightning indexer at every layer. (b) IndexCache adds a single conditional branch (red lines): F layers compute and cache fresh indices; S layers reuse the cached indices. Note that Tcache is a temporary buffer holding only the current index tensor; it is overwritten at each F layer and requires no additional GPU memory beyond what standard DSA already allocates.
Figure 4
10K
60K
120K
200K
Prefill time (s) ↓
DSA
0.57
3.38
8.57
19.5
+ IndexCache (1/2)
0.47
2.86
6.57
13.7
+ IndexCache (1/4)
0.45
2.59
5.66
10.7
Decode throughput, per request (tok/s) ↑
DSA
73.5
67.0
63.0
58.0
Table 1: End-to-end inference performance of the 30B DSA model with IndexCache at two retention ratios. Prefill time : seconds (lower is better). Decode per request : tokens/s under single concurrency (higher is better). Decode full : total tokens/s (higher is better). Decode throughput is reported per GPU.
Figure 3: Relative speedup of IndexCache over the DSA baseline across three inference settings on the 30B model. DSA baseline is normalized to 100%.
Averages
Long-Context
General & Reasoning
Config
Long
G&R
MRCR
GW
LB2
RULER
LCR
AIME
GPQA
LCB
IFB
Original DSA
50.2
74.6
24.5
49.6
45.5
87.9
43.6
91.0
77.6
71.4
58.4
1/2 Unif. IndexCache
47.4
74.3
22.0
46.6
46.0
83.6
38.6
92.2
76.4
69.7
59.0
+Search pattern
50.3
74.4
24.7
49.5
46.3
87.8
43.2
91.9
76.3
71.3
58.2
1/4 Unif. IndexCache
43.0
73.8
17.7
37.2
43.1
79.2
37.8
91.3
75.7
69.4
58.9
+Search pattern
49.9
74.9
25.1
47.4
45.7
87.6
43.8
92.6
78.6
70.0
58.3
Table 2: Training-free IndexCache at 1/2, 1/4, and 1/8 indexer retention. ‘Long’ and ‘G&R’ aggregate benchmark scores. We compare uniform interleaving against searched patterns.
Averages
Long-Context
General & Reasoning
Config
Long
G&R
MRCR
GW
LB2
RULER
LCR
AIME
GPQA
LCB
IFB
Original DSA
51.0
74.2
24.7
49.1
46.9
87.3
47.0
88.8
79.4
70.5
57.9
1/2 Unif. IndexCache
51.6
74.5
23.8
50.2
47.2
87.0
49.8
89.3
76.7
72.2
59.9
w/ searched pattern
50.6
73.6
23.9
48.1
47.1
87.5
46.6
89.6
78.6
68.5
57.7
w/o cross-layer loss
49.8
74.5
24.6
48.3
45.0
87.1
44.0
88.8
79.4
71.7
58.0
1/4 Unif. IndexCache
50.6
74.1
23.7
48.1
46.9
86.1
48.4
89.3
78.0
70.5
58.7
Table 3: Training-aware IndexCache at 1/2 and 1/4 indexer retention with uniform interleaving. w/ searched pattern : the greedy-searched pattern replaces uniform interleaving. w/o cross-layer loss : each indexer is distilled only against its own layer.
Long Avg
MRCR v2
GraphWalks
LongBench v2
RULER
AA-LCR
Original DSA
78.4
71.1
92.7
64.5
97.7
66.2
1/2 Unif. IndexCache
78.1
72.8
90.2
65.1
97.6
64.6
+Searched pattern
78.7
72.3
90.8
66.0
97.3
67.2
1/4 Unif. IndexCache
72.7
65.8
74.9
62.2
96.2
64.6
+Searched pattern
78.0
70.8
90.3
63.7
97.6
67.6
Table 4: Preliminary results on GLM-5 (744B) with training-free IndexCache.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Pairwise top- k index overlap ratio between all layer pairs of the 30B DSA model. Shared blocks according to the greedy-searched 1/4 IndexCache pattern are marked .
Avg
MRCR v2
GraphWalks
RULER
Original DSA
54.0
24.5
49.6
87.9
1/2 Unif. IndexCache
50.7
22.0
46.6
83.6
+Searched pattern
49.8
22.9
43.5
82.9
Appendix
Table 5: Evaluation results of training-free similarity-based searched pattern.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer's hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30-2.72 times decode indexer speedups over DSA across 8K-128K contexts, retaining most of LatentIndex's recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.
Zhaohui Wang, Zhixin Pan, Fanxu Meng +1
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies for the two stages, preventing one retrieval representation from being reused throughout inference. We propose Self-Indexing Attention, a training-free framework built on a shared transform-domain sign-magnitude representation. The key signs provide a reusable token-level index for grouped prefill selection and decode retrieval, while the same representation remains compatible with external KV-cache compression without separate indexer metadata. This 1-bit index enables efficient retrieval through bitwise operations widely supported by modern accelerators. At 5% attention density, Self-Indexing Attention remains close to dense attention on LongBench and RULER and achieves up to 6.1x prefill and 10.3x decode attention-operator speedups. Experiments with TurboQuant and DeepSeekV4-Flash further demonstrate compatibility with low-bit KV-cache compression and pretrained sparse-attention indexers.
Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. In this work, we propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of token sparse attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly, including pre-filling, KV-cache storage, and long-context decoding. Experiments across short-context and long-context benchmarks show that CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context. These results suggest a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.