MLIPS: Multi-Layer Indexer Pool Sharing

Samkit Jain; Sai Kishan Pampana; Shlok Agarwal; Yashwant Marathe; Bhargava Gopireddy; Aradhana Mohan Parvathy; Sachin Idgunji

cs.LGOct 20, 2026

MLIPS: Multi-Layer Indexer Pool Sharing

Authors: Samkit Jain, Sai Kishan Pampana, Shlok Agarwal, Yashwant Marathe, Bhargava Gopireddy, Aradhana Mohan Parvathy, Sachin Idgunji

Organizations: NVIDIA Corporation

Abstract

Sparse attention limits the keys used by each query, but finding those keys can still require a full-prefix scan at every layer. We introduce $MLIPS:$ $Multi$ - $Layer$ $Indexer$ $Pool$ $Sharing$ , a training-free method that shares candidate search while retaining each consumer's own scoring and selection. A producer publishes a top- $M$ pool larger than the attention budget $K$ ; subsequent layers select their own top- $K$ from that pool. Measurements across three sparse-attention model families show substantially higher coverage from broad pools than overlap between final selections. We implement layerwise sharing, persistent decode reuse, and a coarse producer in vLLM for DeepSeek-V4-Flash. Across four long-context suites, three consumer layers per producer give $1.14$ - $1.40\times$ speedups and $14.8$ - $27.4\%$ lower GPU energy, with observed accuracy changes of $-1.49$ to $+0.29$ percentage points in single runs. More extensive sharing exposes task-dependent quality losses. Sampled prefill traces also reveal recurring selected KV positions across query groups and layers. We characterize this shared working-set structure and use it to motivate Top- $G$ , an exploratory bounded pool construction for prefill. Its deployment, task accuracy, and serving benefits remain open.

Figures & tables

Panel of Figure 1 — Figure 1: A wider pool increases available consumer choices, but coverage declines at later positions. DeepSeek-V4-Flash, $h=1$ , $K=512$ , $M=16{,}384$ ; values are rounded from the capture summaries. Bands refer to positions within prompts. Full cross-model results appear in Appendix C .

Panel of Figure 2 — Figure 2: MLIPS producer–consumer design. Orange cells build a pool; blue cells re-score its candidates. Dashed boxes show reuse at $h=2$ and, on the right, $qh=2$ . Consumers retain independent selections, which may overlap.

Suite	$h$	Score	Speedup	GPU energy saved
LongBench-v2	stock	$50.50$	$1.000\times$	$0.0\%$
	$1$	$49.11$	$1.066\times$	$8.9\%$
	$2$	$49.70$	$1.162\times$	$15.3\%$
	$3$	$50.10$	$1.197\times$	$17.1\%$
RULER-1M	stock	$80.41$	$1.000\times$	$0.0\%$
	$1$	$79.91$	$1.177\times$	$16.9\%$

Table 1: Layer sharing with

M=16{,}384

and

qh=0

. Scores, speedups, and GPU energy savings use each suite’s stock run. Every row is one serving run.

$qh$	Decode time (s)	Change
$0$ (stock)	$857.2$	—
$1$	$867.2$	$+1.2\%$
$2$	$850.5$	$-0.8\%$
$3$	$829.0$	$-3.3\%$

Table 2: Isolated decode at

h=0

. Changes are relative to stock. This synthetic timing experiment does not measure answer quality.

Panel of Figure 3 — Figure 3: Observed quality limits of more extensive sharing and coarse filtering. Each configuration is one serving run; changes use the corresponding campaign’s stock baseline. Curves connect measured settings and are not uncertainty estimates. Full timing, energy, and accuracy values are in Appendix D.2 .

Panel of Figure 4 — Figure 4: A shared pool covers many subsequent selections, with coverage decreasing as its query window grows. Results use sampled queries from LB2 sequences 0–6; window sizes describe original positions, not the number of measured queries. Mean averages distances 1–6.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	$qh=0$	$qh=1$	$qh=2$	$qh=3$
$h=0$	$1.00$	$1.88$	$2.67$	$3.37$
$h=1$	$1.81$	$3.25$	$4.42$	$5.40$
$h=2$	$2.67$	$4.57$	$6.00$	$7.11$
$h=3$	$3.03$	$5.09$	$6.59$	$7.72$

Appendix

Table 3: Modeled indexer scoring-work reduction for the exact 21-layer stack. The

qh=0

column gives layerwise reuse alone. Values are reduction factors.

Panel of Figure 5 — Figure 5: Coarse-to-fine search and shared candidate production. Both score representatives and gather the highest-scoring groups. HISA refines the gathered candidates for one layer; MLIPS-CP publishes them as a pool and retains a fine selection for each producer and consumer. Group sizes differ in practice; the diagram uses equal granularity for illustration.

$P=2$					$P=4$
	$qh=0$	$1$	$2$	$3$		$qh=0$	$1$	$2$	$3$
$h=0$	1.78	3.20	4.36	5.33	$h=0$	3.20	5.33	6.86	8.00
$h=1$	3.08	5.17	6.68	7.81	$h=1$	5.17	7.81	9.42	10.50
$h=2$	4.36	6.86	8.47	9.60	$h=2$	6.86	9.60	11.08	12.00
$h=3$	4.87	7.47	9.08	10.18	$h=3$	7.47	10.18	11.59	12.44
$P=8$					$P=16$

Appendix

Table 4: Modeled scoring-work reductions with a coarse producer,

b=M/P

. Values are factors relative to full indexing; representative construction and memory costs are excluded.

Figure 6: Layer-sharing recall below 200K query positions. Rows are model families, columns are suites, and each panel varies schedule $h$ and producer-pool width $M$ . The rightmost column is the native top- $K$ selection overlap; wider columns show pool coverage. Units differ across families as labeled.

Figure 7: Layer-sharing recall over all sampled positions in up-to-1M captures. Broad candidate pools retain more consumer choices than final-selection inheritance on these captures; neither measurement alone establishes downstream accuracy. Axes follow Figure 6 .

Figure 8: Joint layer/query candidate coverage below 200K query positions. Each panel varies exact sparse-layer distance (labeled $h$ in the original plot, $d$ in the manuscript) and query offset $qh$ at the configured broad pool. The zero-layer-distance row isolates cross-query reuse.

Figure 9: Joint layer/query candidate coverage over all sampled positions in up-to-1M captures. Exact layer distance is labeled $h$ in this original plot. The self pair is 1.00; later queries retain substantial but imperfect coverage. The captured query offsets end at three.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
LongBench-v2	stock	$50.50$	—	$2{,}970$	—	$3{,}326$	—	—
	$h{=}1$	$49.11$	$-1.39$	$2{,}787$	$1.066\times$	$3{,}029$	$8.9\%$	$-3.0\%$
	$h{=}2$	$49.70$	$-0.80$	$2{,}556$	$1.162\times$	$2{,}818$	$15.3\%$	$-1.5\%$
	$h{=}3$	$50.10$	$-0.40$	$2{,}481$	$1.197\times$	$2{,}756$	$17.1\%$	$-0.8\%$
	$h{=}5$	$50.30$	$-0.20$	$2{,}352$	$1.263\times$	$2{,}645$	$20.5\%$	$+0.4\%$
	$h{=}6$	$49.11$	$-1.39$	$2{,}267$	$1.310\times$	$2{,}588$	$22.2\%$	$+1.9\%$

Appendix

Table 5: Complete layer-sharing results (

qh=0

). Scores and changes use the stock row within each suite; energy is GPU-only and measured after warm-up. Every configuration is one serving run.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
LongBench-v2	stock	$50.50$	—	$2{,}970$	—	$3{,}326$	—	—
	$h{=}0,qh{=}1$	$50.30$	$-0.20$	$3{,}076$	$0.966\times$	$3{,}354$	$-0.8\%$	$-2.6\%$
	$h{=}0,qh{=}2$	$50.70$	$+0.20$	$3{,}017$	$0.984\times$	$3{,}339$	$-0.4\%$	$-1.2\%$
	$h{=}0,qh{=}3$	$48.91$	$-1.59$	$3{,}031$	$0.980\times$	$3{,}341$	$-0.4\%$	$-1.6\%$
	$h{=}1,qh{=}1$	$49.50$	$-1.00$	$2{,}832$	$1.049\times$	$3{,}039$	$8.6\%$	$-4.2\%$
	$h{=}1,qh{=}2$	$49.50$	$-1.00$	$2{,}848$	$1.043\times$	$3{,}045$	$8.4\%$	$-4.5\%$

Appendix

Table 6: Persistent decode reuse on LongBench-v2 (

qh\geq 1

). The

h=0

rows use query reuse alone;

qh=0

results appear in Table 5 . Ratios use the displayed stock row.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
RULER-1M	stock	$80.41$	—	$5{,}982$	—	$6{,}624$	—	—
	$h{=}0,qh{=}1$	$78.41$	$-2.00$	$6{,}081$	$0.984\times$	$6{,}653$	$-0.4\%$	$-1.2\%$
	$h{=}0,qh{=}2$	$82.51$	$+2.10$	$6{,}027$	$0.992\times$	$6{,}638$	$-0.2\%$	$-0.6\%$
	$h{=}0,qh{=}3$	$79.85$	$-0.56$	$6{,}032$	$0.992\times$	$6{,}638$	$-0.2\%$	$-0.6\%$
	$h{=}1,qh{=}1$	$80.46$	$+0.05$	$5{,}119$	$1.168\times$	$5{,}519$	$16.7\%$	$-2.7\%$
	$h{=}1,qh{=}2$	$78.36$	$-2.05$	$5{,}119$	$1.168\times$	$5{,}514$	$16.8\%$	$-2.8\%$

Appendix

Table 7: Persistent decode reuse on RULER-1M (

qh\geq 1

). The

h=0

rows use query reuse alone;

qh=0

results appear in Table 5 . Ratios use the displayed stock row.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
MRCR-v2	stock	$75.02$	—	$4{,}911$	—	$5{,}093$	—	—
	$h{=}0,qh{=}1$	$73.70$	$-1.32$	$5{,}097$	$0.964\times$	$5{,}148$	$-1.1\%$	$-2.6\%$
	$h{=}0,qh{=}2$	$70.48$	$-4.54$	$5{,}114$	$0.960\times$	$5{,}159$	$-1.3\%$	$-2.7\%$
	$h{=}0,qh{=}3$	$72.24$	$-2.78$	$5{,}083$	$0.966\times$	$5{,}154$	$-1.2\%$	$-2.2\%$
	$h{=}1,qh{=}1$	$73.88$	$-1.14$	$4{,}839$	$1.015\times$	$4{,}747$	$6.8\%$	$-5.4\%$
	$h{=}1,qh{=}2$	$71.12$	$-3.90$	$4{,}872$	$1.008\times$	$4{,}759$	$6.6\%$	$-5.8\%$

Appendix

Table 8: Persistent decode reuse on MRCR-v2 (

qh\geq 1

). The

h=0

rows use query reuse alone;

qh=0

results appear in Table 5 . Ratios use the displayed stock row.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
GraphWalks	stock	$53.35$	—	$12{,}264$	—	$12{,}976$	—	—
	$h{=}0,qh{=}1$	$53.02$	$-0.33$	$12{,}269$	$1.000\times$	$12{,}987$	$-0.1\%$	$+0.1\%$
	$h{=}0,qh{=}2$	$53.17$	$-0.18$	$12{,}262$	$1.000\times$	$13{,}004$	$-0.2\%$	$+0.2\%$
	$h{=}0,qh{=}3$	$51.83$	$-1.52$	$12{,}363$	$0.992\times$	$13{,}005$	$-0.2\%$	$-0.6\%$
	$h{=}1,qh{=}1$	$51.98$	$-1.37$	$10{,}700$	$1.146\times$	$11{,}111$	$14.4\%$	$-1.8\%$
	$h{=}1,qh{=}2$	$52.05$	$-1.30$	$10{,}868$	$1.129\times$	$11{,}149$	$14.1\%$	$-3.0\%$

Appendix

Table 9: Persistent decode reuse on GraphWalks (

qh\geq 1

). The

h=0

rows use query reuse alone;

qh=0

results appear in Table 5 . Ratios use the displayed stock row.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
LongBench-v2	stock	$50.70$	—	$2{,}990$	—	$3{,}339$	—	—
	$h{=}1$	$50.10$	$-0.60$	$2{,}910$	$1.027\times$	$2{,}998$	$10.2\%$	$-7.8\%$
	$h{=}2$	$50.70$	$+0.00$	$2{,}602$	$1.149\times$	$2{,}793$	$16.3\%$	$-3.9\%$
	$h{=}3$	$49.90$	$-0.80$	$2{,}590$	$1.154\times$	$2{,}736$	$18.1\%$	$-5.4\%$
	$h{=}1,qh{=}1$	$51.49$	$+0.79$	$2{,}956$	$1.012\times$	$2{,}988$	$10.5\%$	$-9.5\%$
	$h{=}2,qh{=}1$	$51.49$	$+0.79$	$2{,}646$	$1.130\times$	$2{,}780$	$16.7\%$	$-5.9\%$

Appendix

Table 10: Coarse producer with mean representatives,

P=2

and

b=8{,}192

. Ratios and score changes use this campaign’s stock rows. Every configuration is one serving run.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
LongBench-v2	stock	$50.70$	—	$2{,}990$	—	$3{,}339$	—	—
	$P{=}2$	$50.10$	$-0.60$	$2{,}910$	$1.027\times$	$2{,}998$	$10.2\%$	$-7.8\%$
	$P{=}4$	$52.49$	$+1.79$	$2{,}878$	$1.039\times$	$2{,}828$	$15.3\%$	$-12.0\%$
	$P{=}8$	$49.90$	$-0.80$	$2{,}786$	$1.073\times$	$2{,}722$	$18.5\%$	$-12.5\%$
	$P{=}16$	$51.09$	$+0.39$	$2{,}768$	$1.080\times$	$2{,}680$	$19.7\%$	$-13.3\%$
RULER-1M	stock	$79.13$	—	$5{,}990$	—	$6{,}644$	—	—

Appendix

Table 11: Coarse-producer group-size sweep at

h=1

qh=0

, and fixed

bP=M=16{,}384

. The

P=2

rows repeat Table 10 ; all ratios use the displayed stock rows.

suite	config	acc	$\Delta$ acc	wall (s)	speedup	energy (Wh)	saved	$\Delta$ power
LongBench-v2	stock	$50.50$	—	$2{,}970$	—	$3{,}326$	—	—
	max, $h{=}1$	$50.30$	$-0.20$	$2{,}956$	$1.005\times$	$2{,}979$	$10.4\%$	$-10.0\%$
	max, $h{=}2$	$51.29$	$+0.79$	$2{,}657$	$1.118\times$	$2{,}782$	$16.3\%$	$-6.5\%$
	max, $h{=}3$	$49.70$	$-0.80$	$2{,}616$	$1.136\times$	$2{,}738$	$17.7\%$	$-6.5\%$
RULER-1M	stock	$80.41$	—	$5{,}982$	—	$6{,}624$	—	—
	max, $h{=}1$	$78.82$	$-1.59$	$4{,}974$	$1.203\times$	$5{,}118$	$22.7\%$	$-7.1\%$

Appendix

Table 12: Coarse producer with elementwise-max representatives,

P=2

and

b=8{,}192

. Ratios and score changes use the displayed stock rows, also used in Table 5 .

Figure 10: Per-sequence Top- $G$ coverage at $G=32{,}000$ . Rows are LB2 sequences 0–6 and columns are subsequent CSA-layer distances; panels vary the nominal query window. Cells average eligible recorded queries over the 11 producers with a complete ten-layer horizon. The original plot uses hop for exact layer distance $d$ .

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,080 papers

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Oct 3, 2026Zhaohui Wang, Zhixin Pan, Fanxu Meng +1Cross-Layer Attention LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Mar 12, 2026Yushi Bai, Qian Dong, Ting Jiang +5LLM Inference Acceleration Sparse Attention
Tsinghua University
SPIN: Shadow Predictive Indexer for Sparse Attention
Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference Acceleration Sparse Attention

Oct 3, 2026J/K move · Enter open · S save

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

Zhaohui Wang, Zhixin Pan, Fanxu Meng, Muhan Zhang

Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology

Cross-Layer Attention LLM Inference Acceleration Efficient Attention Sparse Attention Multi-Head Latent Attention

Open paper AskPDF arXiv

Sparse attention reduces core-attention computation, but its indexers still incur repeated selection work and per-layer key-cache storage. Reusing selected indices across layers reduces this overhead but constrains multiple layers to the same token set. We introduce LatentIndex, which extends the latent-sharing principle of Multi-head Latent Attention across indexer layers. Each layer group constructs a shared latent cache from its first layer's hidden states, while layer-specific scoring enables independent token selection. Absorbing key decoders into queries enables direct scoring of the shared cache without reconstructing historical per-layer keys. We develop training-free calibration and investigate a training-aware instantiation of this principle. To balance quality and computation, a hierarchical selection (HS) variant lets followers independently refine a shared candidate set proposed by the anchor. With four-layer sharing, LatentIndex reduces logical indexer-cache storage by 61.1% on DeepSeek-V3.2. Across DeepSeek-V3.2 and GLM-5, training-free LatentIndex improves head-wise attention-mass recall over IndexCache by up to 3.28 percentage points while maintaining RULER and LongBench performance close to native DSA. HS further achieves 2.30-2.72 times decode indexer speedups over DSA across 8K-128K contexts, retaining most of LatentIndex's recall. LatentIndex offers a new perspective on cross-layer indexing: sharing continuous representations rather than discrete selections enables efficient reuse while preserving layer-specific token selection.

Abstract

LLM Inference Acceleration

Explore similar work

LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

SPIN: Shadow Predictive Indexer for Sparse Attention