Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention
Organizations: Yale University · Huawei Canada
Abstract
Sparse attention models read a few blocks of keys for every query at a fixed budget and can lose much of their long-context retrieval accuracy at that budget. We ask where this loss is made. Using exact attention mass selection at the shipped budget as an oracle, we show that the selector under-ranks the needle rather than running out of budget. We find attention heads that miss the needle by running a calibration probe that teacher-forces the answer and reads two quantities in every head. The first is the needle attention that the selector forfeits, and the second is the share of the answer's log-probability that fixing the head's selection restores, integrated along the path from the shipped selector to full attention. Four checkpoints of the InfLLM-V2 family and NOSA-8B lose multi-key retrieval in the same three blind heads, which attend to the needle while a mean-pooled key summary dilutes its single decisive token. Full attention in those heads or a 4-bit scorer that keeps the shipped budget recovers most of the loss without training. On MiniCPM4.1-8B at 64K, either fix lifts multi-key retrieval accuracy from 54.1 to 97.5%, level with full attention in every head. In production serving engines, full attention in those heads keeps 84 to 89% of the time to first token (TTFT) that sparse attention saves at 32K to 128K context length. Beyond the InfLLM-V2 selector lineage, one unified rule flags attention heads above a threshold swept on each checkpoint until they close most of the gap to full attention on calibration prompts. A similar small set of blind attention heads carry the loss for the SeerAttention-R gates, NSA, and DashAttention, whereas many such heads exist in MoBA. On Qwen3-8B with the SeerAttention-R gates, full attention in 18 of 288 key-value heads lifts multi-key retrieval accuracy from 29.8 to 97.7%.
Figures & tables
| Selection | InfLLM-V2-8B, 32K | MiniCPM4.1-8B, 32K | MiniCPM4.1-8B, 64K |
|---|---|---|---|
| Sparse | 63.1 | 82.7 | 54.1 |
| Exact mass at decode | 85.4 | 89.2 | 80.0 |
| Exact mass at decode and window | 95.3 | 100.0 | 97.5 |
| Exact mass at every token of flagged heads | 94.7 | 100.0 | 96.9 |
| Exact mass at decode tokens of flagged heads | 85.5 | 89.3 | 80.2 |
| Dense at every token of flagged heads | 96.0 | 100.0 | 97.5 |
| 4-bit Scorer, Flagged Heads | |||||||
|---|---|---|---|---|---|---|---|
| Model | Context | Dense | Sparse | Full Attention, Flagged Heads | Decode | + Question | Every Token |
| InfLLM-V2-8B | 32K | 97.8 | 63.1 | 96.0 | 85.6 | 95.1 | 94.7 |
| MiniCPM4.1-8B | 32K | 100.0 | 82.7 | 100.0 | 89.1 | 99.9 | 100.0 |
| MiniCPM4.1-8B | 64K | 97.1 | 54.1 | 97.5 | 79.7 | 97.2 | 97.5 |
| Fix | MK3 | CWE | PopQA | NQ | HotpotQA | Bytes per Cached Token |
|---|---|---|---|---|---|---|
| Shipped selector | 63.1 | 45.5 | 41.2 | 44.7 | 41.5 | 1,024 |
| Full attention, flagged heads | 96.0 | 33.9 | 38.0 | 44.8 | 43.5 | 2,512 |
| 4x budget, flagged layers | 97.2 | 36.6 | 37.7 | 44.8 | 43.7 | 2,140 |
| Scorer, every token | 94.7 | 40.0 | 37.8 | 43.7 | 42.3 | 1,168 |
| Scorer, + question | 95.1 | 45.7 | 37.2 | 44.2 | 41.2 | 1,168 |
| Scorer, decode | 85.6 | 45.6 | 41.0 | 45.2 | 42.2 | 1,168 |
| Fix | Worst Change | Mean Change | Beats Shipped (%) |
|---|---|---|---|
| Full attention, flagged heads | (Common Words) | 97.3 / 98.8 | |
| 4x budget, flagged layers | (Common Words) | 98.5 / 99.0 | |
| Scorer, every token | (Common Words) | 98.9 / 98.3 | |
| Scorer, + question | (PopQA) | 99.8 / 98.0 | |
| Scorer, decode | (Multi-value) | 100.0 / 100.0 | |
| Scorer, decode, 16 CWE heads | (Frequent Words) | 99.8 / 92.0 |
| MiniCPM4.1-8B, 64K | NOSA-8B, 32K | Bytes per | |||
|---|---|---|---|---|---|
| MK3 | RULER | MK3 | RULER | Cached Token | |
| Dense | 97.0 | 86.9 | 94.0 | 89.4 | 32,768 |
| Sparse | 55.5 | 80.2 | 18.5 | 70.6 | 1,024 |
| Quest | 22.0 | 73.8 | 18.5 | 74.2 | 512 |
| Double Sparsity | 34.0 | 72.6 | 29.5 | 72.5 | 2,048 |
| Loki | 96.0 | 88.0 | 83.0 | 86.7 | 4,096 |
| Flagged Heads | Same Count | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Selector | Heads | Dense | Sparse | Full Attention | 4-bit Scorer | Echo | Random |
| Qwen3-8B | SeerAttention-R | 18 of 288 | 98.7 | 29.8 | 97.7 | 97.3 | 69.4 | 31.4 |
| Qwen3-4B | SeerAttention-R | 9 of 288 | 97.5 | 16.6 | 87.8 | 88.9 | 56.3 | 17.5 |
| Qwen3-14B | SeerAttention-R | 5 of 320 | 99.7 | 54.5 | 96.9 | 98.1 | 64.1 | 54.6 |
| R1-Distill-Qwen-14B | SeerAttention-R | 22 of 384 | 85.6 | 17.9 | 79.1 | 73.9 | 42.9 | 21.2 |
| Dash-8B | DashAttention | 13 of 64 | 99.4 | 66.6 | 98.0 | 92.0 | 80.9 | 91.0 |
| Flagged Heads | Same Count | |||||
|---|---|---|---|---|---|---|
| Model | Heads | Dense | Sparse | Full Attention | 4-bit Scorer | Random |
| Qwen3-8B | 18 of 288 | 36.0 | 29.2 | 33.2 | 33.2 | 29.3 |
| Qwen3-14B | 5 of 320 | 35.2 | 32.7 | 32.7 | 33.3 | 33.0 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Checkpoint | Local Window | Forced Blocks | Free Slots | Blocks per Token | Dense Below |
|---|---|---|---|---|---|
| InfLLM-V2-8B | 2,048 | 34 | 62 | 96 | 8,192 |
| MiniCPM4-8B Instruct | 2,048 | 34 | 62 | 96 | 8,192 |
| MiniCPM4.1-8B | 2,048 | 34 | 62 | 96 | 8,192 |
| InfLLMv2-8B Retraining | 1,024 | 18 | 46 | 64 | None |
| Model | Units | First Order | Complete | Integrated |
|---|---|---|---|---|
| SeerAttention-R, Qwen3-8B | 288 | 10 | 18 | 18 |
| SeerAttention-R, Qwen3-4B | 288 | 11 | 15 | 9 |
| SeerAttention-R, Qwen3-14B | 320 | 5 | 5 | 5 |
| SeerAttention-R, R1-Distill-Qwen-14B | 384 | 49 | 16 | 22 |
| Dash-8B | 64 | 19 | 23 | 13 |
| NSA-8B | 64 | 5 | 4 | 4 |
| Heads | MK3 | |
|---|---|---|
| Calibration source | ||
| MK3, the default | L9.h0, L16.h0, L22.h0 | 98.0 |
| HELMET’s JSON key-value data | L9.h0, L16.h0, L22.h0 | 98.0 |
| MK2 | L9.h0, L22.h0 | 95.0 |
| MK1, an essay haystack | L3.h0, L6.h0, L9.h0 | 82.0 |
| One needle among repeated filler sentences | None | 65.5 |
| Head | Gain over Sparse |
|---|---|
| L9.h0 | |
| L16.h0 | |
| L22.h0 | |
| Largest of the other 61 heads |
| Checkpoint | First Four Heads by the Fix Gain | Heads Kept | Share of the Gap |
|---|---|---|---|
| InfLLM-V2-8B | L22.h0, L9.h0, L16.h0, L29.h1 | 2 | 0.94 |
| MiniCPM4-8B Instruct | L16.h0, L9.h0, L22.h0, L29.h1 | 4 | 0.97 |
| MiniCPM4.1-8B | L22.h0, L16.h0, L9.h0, L23.h0 | 3 | 1.00 |
| InfLLMv2-8B Retraining | L9.h0, L16.h0, L22.h0, L8.h1 | 3 | 0.97 |
| Nats | Block Score | Top Query Heads | ||||||
|---|---|---|---|---|---|---|---|---|
| Head | Flagged | Sparse Recall | Needle Key | Rest of Block | Exact Keeps | Pooled Loses | Token Level | Pooled |
| L9.h0 | Yes | 0.19 | 21.4 | 0.89 | 0.65 | 0.74 | 0.18 | |
| L16.h0 | Yes | 0.24 | 21.6 | 0.86 | 0.65 | 0.71 | 0.25 | |
| L22.h0 | Yes | 0.73 | 28.2 | 3.8 | 1.00 | 0.25 | 0.97 | 0.75 |
| L23.h1 | No | 0.83 | 7.4 | 4.4 | 0.94 | 0.07 | 0.85 | 0.85 |
| L26.h0 | No | 0.98 | 18.4 | 11.7 | 1.00 | 0.02 | 1.00 | 0.99 |
| Bits | Predicted Noise | Measured Noise | Recall | MK3 |
|---|---|---|---|---|
| 1 | 0.60 | 0.60 | 81.0 | |
| 2 | 0.34 | 0.35 | 90.5 | |
| 3 | 0.19 | 0.20 | 90.0 | |
| 4 | 0.11 | 0.11 | 90.0 |
| Share, Step | Step, Share 0.9 | |||||
|---|---|---|---|---|---|---|
| Model | 0.80 | 0.85 | 0.90 | 0.95 | ||
| SeerAttention-R, Qwen3-8B | 11 | 11 | 18 | 18 | 29 | 14 |
| SeerAttention-R, Qwen3-4B | 9 | 9 | 9 | 22 | 15 | 8 |
| SeerAttention-R, Qwen3-14B | 5 | 5 | 5 | 5 | 7 | 5 |
| SeerAttention-R, R1-Distill-Qwen-14B | 12 | 22 | 22 | 33 | 22 | 22 |
| Dash-8B | 13 | 13 | 13 | 22 | 22 | 11 |
| Probe (GPU Hours) | Sweep Step | Gap | ||
|---|---|---|---|---|
| Model | Integrated | First Order | (GPU Minutes) | Recovered |
| SeerAttention-R, Qwen3-8B | 1.6 | 0.3 | 11 | 0.79 |
| SeerAttention-R, Qwen3-4B | 1.6 | 0.2 | 10 | 0.87 |
| SeerAttention-R, Qwen3-14B | 2.0 | 0.3 | 16 | 0.89 |
| SeerAttention-R, R1-Distill-Qwen-14B | 2.2 | 0.4 | 17 | 0.76 |
| Dash-8B | 6.0 | 0.4 | 14 | 0.61 |
| Exact Mass | Full Attention, | ||||
|---|---|---|---|---|---|
| Model | Sparse | Flagged Heads | Every Head | Flagged Heads | Dense |
| NOSA-8B, 32K | 18.5 | 90.0 | 94.0 | 94.0 | 94.0 |
| InfLLMv2-8B Retraining, 32K | 23.5 | 82.5 | 85.0 | 90.5 | 92.0 |
| InfLLM-V2-8B, 64K | 16.0 | 64.0 | 69.0 | 75.5 | 82.5 |
| Task | Dense | Sparse | Full Attention, Flagged Heads | 4x Budget, Flagged Layers | Scorer, Every Token |
|---|---|---|---|---|---|
| Variable tracking | 46.9 | 74.4 | 60.9 | 64.1 | 74.2 |
| Common word extraction | 40.5 | 11.8 | 11.6 | 13.5 | 13.8 |
| Benchmark | Model | Dense | Sparse | Scorer, Every Token |
| LongFuncEval, 32K | InfLLM-V2-8B | 62.9 | 58.4 | 60.0 |
| LongFuncEval, 64K | 48.3 | 39.7 | 46.1 | |
| LongCodeU | MiniCPM4.1-8B | 75.1 | 74.1 | 74.3 |
| RepoQA, 16K | 84.0 | 82.0 | 82.6 | |
| RepoQA, 32K | 79.0 | 78.4 | 79.4 | |
| OpenAI MRCR, 16K to 32K | 19.6 | 19.9 | 19.3 |
| ms per Output Token | Prefill Attention (s) | ||||||
| Context | Dense | Sparse | Full Attention, Flagged Heads | Dense | Sparse | Full Attention, Flagged Heads | Scorer |
| 32K | 10.9 | 12.1 | 12.2 | 0.74 | 0.94 | 0.98 | 1.06 |
| 64K | 14.4 | 14.0 | 14.4 | 2.98 | 2.25 | 2.40 | 2.67 |
| 128K | 20.9 | 19.3 | 19.9 | 13.04 | 5.61 | 6.23 | 7.13 |
Sparse Attention
Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.
See all 274 papers