cs.LGOct 6, 2026
SaveSPIN: Shadow Predictive Indexer for Sparse Attention
Abstract
Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it. However, the indexer must still score the entire KV cache at every decoding step. This scoring overhead becomes a major bottleneck as the context length grows. We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead. SPIN uses lightweight, history-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step. SPIN treats KV blocks and speculative decoding as first-class design and implementation considerations. Across extensive evaluations on long-context and agentic benchmarks, SPIN achieves 30-40% sparsity while preserving task quality. In end-to-end vLLM serving, SPIN improves output throughput by up to 14.9% and reduces median inter-token latency by up to 13.2%.
Figures & tables
Figure 1 : Indexer top- patterns from a single DSA layer on a prompt from summscreenfd . The y-axis represents decode iterations. The prompt is 9.7K tokens long, and 1,400 context KV slots are visualized along the x-axis. DeepSeek-V3.2 exhibits prominent diagonal bands and vertical lines, whereas DeepSeek-V4 exhibits mostly vertical patterns.
| DeepSeek-V3.2 | DeepSeek-V4 Flash | |||||||
|---|---|---|---|---|---|---|---|---|
| summscreenfd | AA-LCR | summscreenfd | AA-LCR | |||||
| Method | Pearson correlation | top- recall | Pearson correlation | top- recall | Pearson correlation | top- recall | Pearson correlation | top- recall |
| Random baseline | 0.000 | 0.203 | 0.000 | 0.022 | 0.000 | 0.203 | 0.000 | 0.022 |
| Prev-layer | 0.289 | 0.359 | 0.431 | 0.164 | 0.547 | 0.482 | 0.519 | 0.306 |
| Prev-layer (Bo5) | 0.500 | 0.461 | – | – | 0.626 | 0.525 | – | – |
| Prev-iter | 0.913 | 0.824 | 0.902 | 0.615 | 0.876 | 0.751 | 0.849 | 0.577 |
Table 1 : Comparison of indexer prediction methods — profiling results for one prompt each from summscreenfd (length: 9.7k) and AA-LCR (length: 94k). Values are averaged over layers and decode iterations. The best method is shown in bold, and the second-best method is underlined.
| Configuration | DeepSeek-V4 Flash | DeepSeek-V4 Pro |
|---|---|---|
| All-keep | 53.48 | 58.05 |
| Spin (20%) | 53.08 | 57.85 |
| Spin (30%) | 52.09 | 57.06 |
| Spin (40%) | 54.27 | 58.25 |
| Spin (50%) | 54.08 | 58.65 |
Table 2 : LongBench-v2 exact-match score (%) over all 503 examples.
| AA-LCR | RULER | |||||
|---|---|---|---|---|---|---|
| Configuration | Score | Mean miss | P99 miss | Score | Mean miss | P99 miss |
| All-keep | 64.40 | – | – | 90.20 | – | – |
| Spin (20%) | 62.60 | 0.88% | 7.81% | 90.13 | 5.72% | 19.92% |
| Spin (30%) | 64.60 | 1.92% | 13.48% | 90.00 | 10.07% | 29.49% |
| Spin (40%) | 64.00 | 3.46% | 20.31% | 89.69 | 14.87% | 38.67% |
| Spin (50%) | 62.00 | 5.78% | 28.52% | 88.96 | 20.68% | 48.83% |
Table 3 : AA-LCR and RULER results for DeepSeek-V4 Flash. AA-LCR scores are averaged over execution-level runs; RULER scores are macro-averaged over task–length cells.
| 256K | 512K | 1M | ||||
|---|---|---|---|---|---|---|
| Sparsity | No exploration | Random | No exploration | Random | No exploration | Random |
| All-keep | 57.15 | 41.13 | 37.44 | |||
| 20% | 58.94 | 56.82 | 43.76 | 45.67 | 36.74 | 35.16 |
| 30% | 54.22 | 59.71 | 39.73 | 45.96 | 32.09 | 33.54 |
| 40% | 53.42 | 57.84 | 37.47 | 47.56 | 32.70 | 37.44 |
| 50% | 46.28 | 54.45 | 35.00 | 44.96 | 29.76 | 33.86 |
Table 4 : MRCRv2 task scores without exploration and with 5% random exploration. Each score averages 40 repeats over 100 fixed prompts per context bin.
| Agent configuration | Score | Mean miss | P99 miss |
|---|---|---|---|
| All-keep | 72.5% | – | – |
| Spin (20%) | 70.5% | 5.9% | 21.1% |
| Spin (30%) | 73.5% | 10.9% | 30.9% |
| Spin (40%) | 70.0% | 17.3% | 41.4% |
| Spin (50%) | 69.5% | 25.7% | 53.1% |
Table 5 : tau2-airline results for DeepSeek-V4 Flash. Mean and p99 miss are selection-fidelity metrics over eligible sparse decode rows and are not applicable to the all-keep control.
| Target block sparsity | Output TPS | TPS gain | Median ITL (ms) | ITL reduction |
|---|---|---|---|---|
| 0% (all-keep) | 4720.29 | – | 17.837 | – |
| 40% | 5218.65 | +10.6% | 16.204 | 9.2% |
| 50% | 5421.45 | +14.9% | 15.484 | 13.2% |
Table 6 : End-to-end serving performance for DeepSeek-V4 Flash with 507K cached context tokens and 5K new input tokens. The batch size is 96, and each request generates 16K tokens. TPS gain and ITL reduction are computed relative to the matched all-keep setting.
Figure 2 : MTP policy ablation over 60 AA-LCR traces with a draft length of three. Each point reports p99 miss and realized block sparsity for one target sparsity under one mask-construction and EMA-update strategy.
| Condition | Average acceptance length | Draft-token acceptance rate | Realized sparsity |
|---|---|---|---|
| All-keep MTP | 2.4049 | 47.19% | – |
| Row-0 sharing + row-0 update | 2.4059 | 47.39% | 39.67% |
| Row-local union + prefix-valid update | 2.4097 | 47.28% | 39.51% |
Table 7 : Online MTP acceptance behavior on a common subset of 1,499 SPEED-Bench requests after excluding repetition-flagged responses and requests without complete mask statistics.
Figure 3 : Block-reduction ablation. Panel (a) compares P99 miss rates for A0, A1-Max, and A2-Max; panels (b) and (c) show relative changes in P99 miss for Mean and Top-4 compared with Max pooling in A1 and A2. Results average per-layer, per-trace P99 over all 60 curated AA-LCR traces and C4A layers.
Figure 4 : Exploration-policy comparison on MRCRv2 at 256K, 512K, and 1M context lengths. Scores are shown as percentage changes relative to all-keep at each target block sparsity. Each point averages 40 repeats over 100 fixed prompts. All exploration methods reserve 5% of the retained-block budget.