LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

Zhaohui Wang; Zhixin Pan; Fanxu Meng; Muhan Zhang

cs.AIOct 3, 2026

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

Authors: Zhaohui Wang, Zhixin Pan, Fanxu Meng, Muhan Zhang

Organizations: Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology

Abstract

Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer's hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30-2.72 times decode indexer speedups over DSA across 8K-128K contexts, retaining most of LatentIndex's recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.

Figures & tables

Figure 1: DSA uses per-layer indexer key caches and token selections; IndexCache reuses the anchor’s selections (orange dashed paths). LatentIndex shares one latent cache derived from anchor hidden states, with layer-specific decoders defining layer-specific keys. At inference, decoders are absorbed into queries for direct cache scoring and independent token selection, without reconstructing historical keys. Queries are layer-specific but are illustrated only once for visual clarity.

Method	DeepSeek-V3.2						GLM-5
Method	4K	8K	16K	32K	64K	128K	4K	8K	16K	32K	64K	128K
DSA	99.33	97.40	95.09	93.48	91.10	90.00	99.45	98.33	97.21	96.37	95.16	94.30
IndexCache	98.86	95.86	92.49	90.26	86.68	85.06	99.16	97.49	95.78	94.65	92.79	91.55
LatentIndex	99.21	96.97	94.34	92.40	89.61	88.34	99.32	97.96	96.65	95.67	94.22	93.12
$\Delta$ vs. IndexCache	+0.35	+1.11	+1.85	+2.14	+2.93	+3.28	+0.16	+0.47	+0.87	+1.02	+1.43	+1.57

Table 1: Head-wise attention-mass recall on a common native DSA trajectory for each model.

Model	Method	4K	8K	16K	32K	64K	128K	Avg.
DeepSeek-V3.2	DSA	97.21	96.84	97.51	97.46	95.89	94.30	96.54
	IndexCache	97.28	96.94	97.51	97.18	95.91	93.59	96.40
	LatentIndex	97.18	96.92	97.35	97.42	95.85	94.22	96.49
GLM-5	DSA	98.07	97.57	97.69	97.41	97.40	96.74	97.48
	IndexCache	97.73	97.42	97.51	96.92	95.70	90.15	95.91
	LatentIndex	97.89	97.57	97.64	97.36	97.42	96.81	97.45

Table 2: RULER performance across context lengths. All grouped methods use

G=4

Model	Method	SQA	MQA	Sum	FS	Syn	Code	Avg.
DeepSeek-V3.2	DSA	52.05	53.79	22.42	61.87	68.17	73.12	55.24
	IndexCache	51.37	53.25	22.24	59.78	67.91	73.11	54.61
	LatentIndex	52.00	53.74	22.27	61.06	68.78	74.18	55.34
GLM-5	DSA	53.09	56.02	24.40	68.45	76.50	72.09	58.42
	IndexCache	53.01	55.67	24.44	69.28	73.33	72.11	57.97
	LatentIndex	53.31	55.71	24.35	68.58	76.33	71.34	58.27

Table 3: LongBench performance across six task categories.

Panel of Figure 2 — Figure 2: Cross-layer sharing on DeepSeek-V3.2: (a) additional $R_{\mathrm{head}}$ loss relative to native DSA by group offset ( $G=4$ ); (b) $R_{\mathrm{head}}$ across context lengths for $G=4$ and $G=8$ , with native DSA as a reference.

	Quality			Decode indexer latency (ms) $\downarrow$
Method	$R_{\mathrm{head}}$ $\uparrow$	RULER $\uparrow$	LongBench $\uparrow$	8K	16K	32K	64K	128K
Native DSA	94.40	96.54	55.24	48.70	52.06	48.28	48.43	49.60
IndexCache	91.54	96.40	54.61	14.31	13.29	14.35	14.28	13.40
LatentIndex + HS	93.19	96.58	54.81	19.89 (2.45 $\times$ )	19.13 (2.72 $\times$ )	20.55 (2.35 $\times$ )	21.10 (2.30 $\times$ )	20.48 (2.42 $\times$ )

Table 4: Hierarchical selection on DeepSeek-V3.2. Latency is aggregated over all 61 indexers in ms per decode token; parentheses show speedup over native DSA. Recall and RULER are averaged over six context lengths, and LongBench over six task categories.

Figure 3: Training dynamics on DeepSeek-V2-Lite at 64K: (a) warmup Recall@2048 and (b) sparse-training LM loss. Values are method minus DSA at matched steps; the dashed line marks zero difference. Faint and bold curves show raw and EMA-smoothed differences, respectively.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	DeepSeek-V3.2	GLM-5
Indexed layers $N_{L}$	61	78
Hidden dimension $d_{\mathrm{model}}$	7168	6144
Indexer heads $H_{I}$	64	32
Native RoPE / NoPE $(d_{R},d_{N})$	64 / 64	64 / 64
Shared RoPE / NoPE $(r_{R},r_{N})$	64 / 128	128 / 128
Independent layer indices	0	0, 1

Appendix

Table 5: Training-free model and latent configurations. Layer indices are zero-based.

Figure 4: Additional $R_{\mathrm{mean}}$ loss relative to native DSA by group offset on DeepSeek-V3.2 ( $G=4$ ). We use the same trajectory, sampling, and aggregation as in Figure 2(a) .

Setting	Dense warmup	Sparse training
Sequence length	65,536	65,536
Completed optimizer steps	400	2,426
Global batch size	8	256
Microbatch size	1	1
Gradient accumulation steps	8	4
Learning rate	$10^{-3}$	$4\times 10^{-5}$

Appendix

Table 6: Training settings. All methods completed both schedules. Batch sizes count sequences; the global indexer-loss coefficient is distinct from the method-specific layer weights in Appendix D.2 .

Cross-Layer Attention

See all 13 papers

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Mar 12, 2026Yushi Bai, Qian Dong, Ting Jiang +5LLM Inference Acceleration Sparse Attention
Tsinghua University
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Long-Context Language Model Inference KV-Cache Management
Microsoft Research · Tsinghua University

Mar 12, 2026J/K move · Enter open · S save

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Yushi Bai, Qian Dong, Ting Jiang, Xin Lv+4

Tsinghua University

LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant tokens per query, reducing core attention from

O(L^2)

O(Lk)

. However, the indexer itself retains

O(L^2)

complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search algorithm that selects which layers to retain indexers by directly minimizing language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82

\times

prefill speedup and 1.48

\times

decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).

Abstract

Cross-Layer Attention

Explore similar work

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing