Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse
Organizations: Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China
Abstract
Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B models and four datasets of dialogue memory, open-domain QA and API documentation. Reuse loses 19-100% of the context-dependent questions that full prefill answers, and it loses more when the other chunks are of the same kind as the target. Because the recoverable damage is spread over these other chunks, restoring a few of them recovers little, and the budget is better spent on the target. This works through contextual compatibility: recomputing the target against the reused cache beats inserting its full-prefill KV in all seven settings tested, although the recomputed KV stays far from the full-prefill KV. Question attention over reused caches, however, selects the target in at most 18.8% of development questions. A retriever finds it more often, and recomputing its top chunk after moving it next to the question significantly beats InfoFlow-style reordering at the same 5% budget in six of nine open-domain QA and API-documentation settings. On dialogue memory, though, recomputing chunks never significantly beats the token-level selector chosen for each setting.
Figures & tables
| 5% budget | 10% budget | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retention | Gain over Token | Retention | Gain over Token | |||||||||
| Dataset | Model | Token | Target | Target +4 | Retrieved | Retrieved +4 | Token | Target | Target +4 | Retrieved | Retrieved +4 | |
| LoCoMo | Qwen3-8B | 105 | 0.71 | −0.04 | −0.06 | −0.10 | −0.10 | 0.72 | −0.04 | −0.04 | −0.10 | −0.09 |
| Llama-3.1-8B | 102 | 0.85 | +0.01 | +0.01 | −0.01 | −0.05 | 0.86 | +0.00 | −0.02 | −0.02 | −0.05 | |
| Mistral-7B | 36 | 0.78 | +0.03 | −0.03 | −0.03 | +0.03 | 0.75 | +0.00 | +0.00 | +0.00 | +0.08 | |
| NQ-Open | Qwen3-8B | 146 | 0.16 | +0.35 | +0.29 | +0.22 | +0.21 | 0.24 | +0.27 | +0.33 | +0.14 | +0.24 |
| Loss and terms | Gain over Token | |||||
|---|---|---|---|---|---|---|
| Dataset | Model | Loss | Same | Sim. | Target | Chunk moved |
| LoCoMo | 14B | 0.36 | +0.20 | +0.03 | +0.14 | – |
| 32B | 0.46 | +0.16 | +0.18 | +0.07 | – | |
| NQ-Open | 14B | 0.79 | +0.17 | +0.08 | +0.22 | +0.34 |
| 32B | 0.78 | +0.37 | +0.08 | +0.38 | +0.35 | |
| Gorilla | 14B | 0.90 | +0.58 | −0.10 † | +0.45 | +0.38 |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Model | No context | Full prefill | Reuse | Full reuse | Reuse no context | middle | Retention middle | |
|---|---|---|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 250 | 0.11 | 0.61 | 0.36 | +0.26 [+0.19, +0.32] | +0.24 [+0.18, +0.32] | 129 | 0.49 |
| Llama-3.1-8B | 250 | 0.10 | 0.58 | 0.50 | +0.08 [+0.04, +0.13] | +0.40 [+0.32, +0.50] | 123 | 0.74 | |
| Mistral-7B | 250 | 0.01 | 0.20 | 0.15 | +0.04 [+0.00, +0.09] | +0.14 [+0.08, +0.20] | 48 | 0.60 | |
| NQ-Open | Qwen3-8B | 500 | 0.29 | 0.56 | 0.22 | +0.34 [+0.30, +0.39] | −0.07 [−0.11, −0.03] | 164 | 0.15 |
| Llama-3.1-8B | 500 | 0.41 | 0.61 | 0.15 | +0.46 [+0.41, +0.51] | −0.26 [−0.30, −0.21] | 136 | 0.15 | |
| Mistral-7B | 500 | 0.36 | 0.53 | 0.20 | +0.33 [+0.28, +0.37] | −0.15 [−0.20, −0.10] | 128 | 0.18 |
| Dataset | Model | Questions | No context wrong | + full prefill right (SIM) | + right under RAND, OTHER ( ) |
|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 250 | 222 | 129 | 105 |
| Llama-3.1-8B | 250 | 225 | 123 | 102 | |
| Mistral-7B | 250 | 247 | 48 | 36 | |
| NQ-Open | Qwen3-8B | 500 | 354 | 164 | 146 |
| Llama-3.1-8B | 500 | 296 | 136 | 128 | |
| Mistral-7B | 500 | 322 | 128 | 101 |
| Comparison | Reported in | Uncorrected | Holm | |
|---|---|---|---|---|
| Loss, | Table 10 | 12 | 12 / 0 | 12 / 0 |
| Same-category term | Table 10 | 12 | 9 / 0 | 9 / 0 |
| Similarity term, open-domain QA | Table 10 | 6 | 5 / 0 | 5 / 0 |
| Competing answers, SIM RAND | Table 16 | 6 | 6 / 0 | 6 / 0 |
| All distractors replaced target replaced | Table 13 | 12 | 11 / 0 | 11 / 0 |
| Reuse no context, all questions, open-domain QA | Table 3 | 6 | 0 / 6 | 0 / 6 |
| Dataset | Model | 2nd chunk | first third | last third | last chunk | Spearman slot, | Position term / cross term | Max. residual | |
|---|---|---|---|---|---|---|---|---|---|
| NQ-Open | Qwen3-8B | 30 | 0.19 | 0.27 | 0.36 | 0.36 | 0.99 | 0.45 | 2e-7 |
| Llama-3.1-8B | 30 | 0.24 | 0.40 | 0.59 | 0.60 | 1.00 | 0.63 | 1e-7 | |
| Mistral-7B | 30 | 0.21 | 0.38 | 0.60 | 0.62 | 1.00 | 0.38 | 5e-7 | |
| Gorilla | Qwen3-8B | 30 | 0.17 | 0.26 | 0.34 | 0.34 | 0.94 | 0.39 | 2e-7 |
| Llama-3.1-8B | 30 | 0.20 | 0.39 | 0.58 | 0.58 | 0.98 | 0.61 | 2e-7 | |
| Mistral-7B | 30 | 0.21 | 0.38 | 0.58 | 0.60 | 1.00 | 0.31 | 6e-7 |
| Measured | Predicted | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | 4 | 8 | 16 | 24 | Odds | Probit | Linear | Most similar first, 16 | |||
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.19 | 0.23 | 0.32 [+0.25, +0.40] | 0.51 | 0.71 | 0.27 | 0.44 | 0.46 | 0.27 |
| Llama-3.1-8B | 128 | 0.16 | 0.20 | 0.20 | 0.30 [+0.22, +0.38] | 0.55 | 0.70 | 0.28 | 0.44 | 0.46 | 0.29 | |
| Mistral-7B | 101 | 0.18 | 0.25 | 0.29 | 0.32 [+0.23, +0.42] | 0.49 | 0.66 | 0.30 | 0.43 | 0.45 | 0.29 | |
| TriviaQA | Qwen3-8B | 151 | 0.27 | 0.31 | 0.30 | 0.43 [+0.35, +0.51] | 0.60 | 0.80 | 0.43 | 0.58 | 0.56 | 0.40 |
| Llama-3.1-8B | 127 | 0.20 | 0.29 | 0.35 | 0.43 [+0.34, +0.51] | 0.60 | 0.74 | 0.34 | 0.49 | 0.50 | 0.46 | |
| Retention | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model ( ) | Reuse | OTHER | RAND | SIM | Loss | Universal | Same- category | Similarity |
| NQ-Open | Tulu3-RAG (83) | Pos.-corr. | 0.39 | 0.40 | 0.18 | +0.82 [+0.73, +0.90] | +0.61 [+0.51, +0.71] | −0.01 [−0.13, +0.10] | +0.22 [+0.08, +0.35] |
| Block | 0.07 | 0.08 | 0.10 | +0.90 [+0.83, +0.96] | +0.93 [+0.87, +0.98] | −0.01 [−0.08, +0.06] | −0.01 [−0.10, +0.06] | ||
| Tulu3-Block-FT (111) | Pos.-corr. | 0.74 | 0.60 | 0.20 | +0.80 [+0.73, +0.87] | +0.26 [+0.18, +0.34] | +0.14 [+0.03, +0.24] | +0.41 [+0.31, +0.51] | |
| Block | 0.77 | 0.79 | 0.40 | +0.60 [+0.51, +0.69] | +0.23 [+0.16, +0.32] | −0.03 [−0.12, +0.05] | +0.40 [+0.29, +0.50] | ||
| Gorilla | Tulu3-RAG (102) | Pos.-corr. | 0.44 | 0.00 | 0.04 | +0.96 [+0.92, +0.99] | +0.56 [+0.46, +0.66] | +0.44 [+0.34, +0.54] | −0.04 † [−0.08, −0.01] |
| Method | Sparse tokens | Full prefill as reference or explanation | Attention locates |
|---|---|---|---|
| CacheBlend | ✓ | ✓ | ✗ |
| EPIC | ✓ | ✗ | ✗ |
| ProphetKV | ✓ | partly h | ✓ |
| InfoFlow KV | ✓ | partly a | ✓ |
| CacheTune | ✓ | ✓ b | ✗ |
| SurroKV | ✓ c | ✓ d | ✓ c |
| Dataset | Model | Loss | Universal | Same-category | Similarity | |
|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | 0.47 | 0.11 | +0.19 [+0.10, +0.26] | +0.16 [+0.08, +0.24] |
| Llama-3.1-8B | 102 | 0.19 | 0.14 | +0.11 [+0.04, +0.18] | −0.06 [−0.13, +0.02] | |
| Mistral-7B | 36 | 0.28 | 0.31 | −0.11 [−0.30, +0.08] | +0.08 [−0.03, +0.20] | |
| NQ-Open | Qwen3-8B | 146 | 0.85 | 0.43 | +0.23 [+0.14, +0.32] | +0.19 [+0.12, +0.27] |
| Llama-3.1-8B | 128 | 0.84 | 0.72 | −0.04 [−0.13, +0.06] | +0.16 [+0.07, +0.25] | |
| Mistral-7B | 101 | 0.82 | 0.65 | +0.07 [−0.02, +0.16] | +0.10 [−0.01, +0.21] |
| Dataset | Model | None | Pad in-domain | Pad SurroKV | Pad neutral | SurroKV full | Token 10% | Token 10% + pad | |
|---|---|---|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | 0.53 | 0.72 | 0.66 | 0.59 | 0.90 | 0.72 | 0.91 |
| Llama-3.1-8B | 102 | 0.81 | 0.85 | 0.80 | 0.85 | 0.95 | 0.86 | 0.97 | |
| Mistral-7B | 36 | 0.72 | 0.78 | 0.81 | 0.81 | 0.83 | 0.75 | 0.92 | |
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.22 | 0.17 | 0.18 | 0.41 | 0.24 | 0.38 |
| Llama-3.1-8B | 128 | 0.16 | 0.17 | 0.13 | 0.15 | 0.63 | 0.38 | 0.55 | |
| Mistral-7B | 101 | 0.18 | 0.22 | 0.23 | 0.24 | 0.48 | 0.34 | 0.42 |
| Dataset | Model | Chance | reuse | reuse | full | full | reuse: last chunk | ||
|---|---|---|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 29 | 6.4 | 10.3 | 17.2 | 20.7 | 31.0 | 51.7 | 86.2 |
| Llama-3.1-8B | 28 | 6.4 | 0.0 | 14.3 | 46.4 | 89.3 | 50.0 | 100.0 | |
| Mistral-7B | 14 | 6.3 | 7.1 | 7.1 | 14.3 | 42.9 | 50.0 | 92.9 | |
| NQ-Open | Qwen3-8B | 36 | 3.4 | 11.1 | 16.7 | 11.1 | 27.8 | 50.0 | 77.8 |
| Llama-3.1-8B | 38 | 3.3 | 10.5 | 13.2 | 23.7 | 55.3 | 55.3 | 100.0 | |
| Mistral-7B | 33 | 3.4 | 6.1 | 6.1 | 9.1 | 54.5 | 63.6 | 100.0 |
| Dataset | Model | Target KV | All distr. | 1 similar | 1 random | All distr. Target | 1 similar Target | ||
|---|---|---|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | 0.53 | 0.61 | 0.79 | 0.54 | 0.58 | +0.18 [+0.07, +0.29] | −0.07 [−0.11, −0.03] |
| Llama-3.1-8B | 102 | 0.81 | 0.71 | 0.82 | 0.79 | 0.82 | +0.12 [+0.03, +0.20] | +0.09 [+0.00, +0.17] | |
| Mistral-7B | 36 | 0.72 | 0.61 | 0.64 | 0.78 | 0.72 | +0.03 [−0.10, +0.18] | +0.17 [+0.05, +0.31] | |
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.18 | 0.71 | 0.15 | 0.14 | +0.53 [+0.44, +0.62] | −0.03 [−0.10, +0.03] |
| Llama-3.1-8B | 128 | 0.16 | 0.23 | 0.70 | 0.18 | 0.16 | +0.46 [+0.36, +0.56] | −0.05 [−0.13, +0.02] | |
| Mistral-7B | 101 | 0.18 | 0.46 | 0.66 | 0.17 | 0.17 | +0.21 [+0.08, +0.35] | −0.29 [−0.39, −0.19] |
| Dataset | Model | Target KV | All distr. | 1 similar | 1 random | |
|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | +0.08 [+0.03, +0.12] | +0.26 [+0.17, +0.35] | +0.01 [+0.00, +0.03] | +0.05 [+0.01, +0.08] |
| Llama-3.1-8B | 102 | −0.11 [−0.16, −0.04] | +0.01 [−0.07, +0.07] | −0.02 [−0.05, +0.02] | +0.01 [−0.02, +0.04] | |
| Mistral-7B | 36 | −0.11 [−0.26, +0.00] | −0.08 [−0.19, −0.02] | +0.06 [+0.00, +0.12] | +0.00 [−0.09, +0.07] | |
| NQ-Open | Qwen3-8B | 146 | +0.03 [−0.03, +0.10] | +0.56 [+0.48, +0.64] | +0.00 [−0.02, +0.02] | −0.01 [−0.03, +0.01] |
| Llama-3.1-8B | 128 | +0.07 [+0.00, +0.15] | +0.53 [+0.44, +0.62] | +0.02 [−0.01, +0.05] | −0.01 [−0.03, +0.02] | |
| Mistral-7B | 101 | +0.28 [+0.18, +0.38] | +0.49 [+0.38, +0.59] | −0.01 [−0.06, +0.04] | −0.01 [−0.05, +0.02] |
| Model, data | Question | Gold answer | Reuse answer | Similar passage (title: sentence) |
|---|---|---|---|---|
| Qwen3-8B, NQ-Open | Who was the actor that played ben stone on law and order | Michael Moriarty | Philip Winchester | Peter Stone (Chicago and Law & Order character) : …saw it develop and we saw the actor, it made sense. Peter Stone (Chicago and Law & Order character) Peter Stone is a fictional character portrayed by Philip Winchester. |
| Llama-3.1-8B, NQ-Open | Who won the most medals in the 1924 winter olympics | Norway | Ville Ritola | Ville Ritola : …win the 3000 m team race. |
| Mistral-7B, NQ-Open | Dendrites and cell bodies are components of what type of matter found in the brain | gray | Neuronal matter | Neuron : Other structural proteins that are important for neuronal function are actin and the tubulin of microtubules. |
| Qwen3-8B, TriviaQA | Liberty Bell was the theme tune for which British TV show? | Monty Python’s Flying Circus | EastEnders | EastEnders theme tune : …the theme is still popular and in the national psyche. The theme tune was used in the 2012 Summer Olympics opening ceremony. |
| Llama-3.1-8B, TriviaQA | What Tory MP was found to have claimed unsuccessfully on his parliamentary expenses for a floating duck island? | Sir Peter Viggers | Gordon Brown | United Kingdom parliamentary expenses scandal : Among those who had to repay expenses were Prime Minister Gordon Brown who claimed £12,415 for cleaning and gardening costs and Liberal Democrat leader Nick Clegg who was… |
| Mistral-7B, TriviaQA | What was the character name of Commander Shore’s daughter in the 60’s series Stingray (voiced by Lois Maxwell) | Atlanta | Lois Lane | Lois Lane : …before was one of Luthor’s robotic duplicates. |
| Dataset | Model | SIM | RAND | SIM RAND | Abstain SIM | Abstain RAND | Conditional (entity) | Conditional (phrase) | |
|---|---|---|---|---|---|---|---|---|---|
| NQ-Open | Qwen3-8B | 146 | 61.6 | 17.1 | +44.5 [+34.9, +53.4] | 4.0 | 25.0 | +40.9 [+27.5, +53.3] | +47.6 [+35.1, +60.1] |
| Llama-3.1-8B | 128 | 63.3 | 39.8 | +23.4 [+13.3, +33.6] | 0.0 | 14.9 | +6.8 [−5.8, +19.1] | +8.3 [−3.5, +19.8] | |
| Mistral-7B | 101 | 58.4 | 12.9 | +45.5 [+33.7, +57.4] | 10.8 | 72.6 | +14.7 [−7.8, +40.4] | +32.8 [+8.9, +56.8] | |
| TriviaQA | Qwen3-8B | 151 | 40.4 | 17.2 | +23.2 [+13.2, +32.5] | 8.2 | 17.9 | +22.7 [+7.9, +37.5] | +31.2 [+16.6, +44.7] |
| Llama-3.1-8B | 127 | 43.3 | 19.7 | +23.6 [+12.6, +33.1] | 3.0 | 9.8 | +10.7 [−5.3, +25.4] | +21.9 [+7.7, +35.9] | |
| Mistral-7B | 117 | 40.2 | 7.7 | +32.5 [+23.1, +41.9] | 26.1 | 59.3 | +31.4 [+8.5, +54.0] | +40.7 [+17.6, +62.8] |
| SIM | Competing removed | Others removed | Gap closed | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Reuse | Full | Reuse | Full | Reuse | Competing | Others | Competing Others | |
| NQ-Open | Qwen3-8B | 131 | 0.13 | 0.94 | 0.22 | 0.94 | 0.16 | +0.15 [+0.08, +0.22] | +0.09 [+0.04, +0.15] | +0.06 [−0.01, +0.14] |
| Llama-3.1-8B | 109 | 0.15 | 0.98 | 0.34 | 0.97 | 0.11 | +0.21 [+0.12, +0.31] | −0.01 [−0.06, +0.04] | +0.22 [+0.13, +0.31] | |
| Mistral-7B | 86 | 0.19 | 0.93 | 0.22 | 0.95 | 0.17 | +0.10 [+0.01, +0.20] | +0.03 [−0.05, +0.12] | +0.07 [−0.03, +0.17] | |
| TriviaQA | Qwen3-8B | 122 | 0.22 | 0.99 | 0.28 | 0.97 | 0.23 | +0.07 [+0.00, +0.13] | +0.04 [−0.01, +0.10] | +0.02 [−0.04, +0.10] |
| Llama-3.1-8B | 96 | 0.16 | 0.94 | 0.24 | 0.99 | 0.17 | +0.15 [+0.06, +0.24] | +0.02 [−0.02, +0.06] | +0.12 [+0.04, +0.22] | |
| Dataset | Model | reuse | Reuse (%) | full | Full prefill (%) | Difference |
|---|---|---|---|---|---|---|
| NQ-Open | Qwen3-8B | 124 | 4.0 | 218 | 1.4 | +2.7 [−0.9, +6.3] |
| Llama-3.1-8B | 107 | 0.0 | 193 | 0.5 | −0.5 [−1.6, +0.0] | |
| Mistral-7B | 83 | 10.8 | 235 | 6.8 | +4.0 [−3.3, +11.6] | |
| TriviaQA | Qwen3-8B | 110 | 8.2 | 199 | 3.0 | +5.2 [+0.0, +10.7] |
| Llama-3.1-8B | 101 | 3.0 | 171 | 4.1 | −1.1 [−5.4, +3.4] | |
| Mistral-7B | 88 | 26.1 | 222 | 8.6 | +17.6 [+8.0, +26.9] |
| Dataset | Model | Token 10% | Retrieved+4 | Retrieved | Retrieved+4 Retrieved | Retrieved+4 Token | |
|---|---|---|---|---|---|---|---|
| NQ-Open | Qwen3-8B | 66 | 0.24 | 0.26 | 0.14 | +0.12 [+0.03, +0.21] | +0.02 [−0.09, +0.12] |
| Llama-3.1-8B | 68 | 0.31 | 0.21 | 0.16 | +0.04 [−0.01, +0.12] | −0.10 [−0.22, +0.01] | |
| Mistral-7B | 50 | 0.34 | 0.36 | 0.20 | +0.16 [+0.02, +0.30] | +0.02 [−0.14, +0.16] | |
| Gorilla | Qwen3-8B | 42 | 0.14 | 0.12 | 0.00 | +0.12 [+0.02, +0.24] | −0.02 [−0.14, +0.10] |
| Llama-3.1-8B | 47 | 0.26 | 0.11 | 0.06 | +0.04 [−0.02, +0.13] | −0.15 [−0.30, −0.02] | |
| Mistral-7B | 57 | 0.30 | 0.23 | 0.09 | +0.14 [+0.04, +0.25] | −0.07 [−0.19, +0.04] |
| Dataset | Model | Target Token | Target +4 Token | Retrieved Token | Retrieved +4 Token | |
|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | −0.04 [−0.13, +0.03] | −0.06 [−0.15, +0.01] | −0.10 [−0.19, −0.03] | −0.10 [−0.20, −0.04] |
| Llama-3.1-8B | 102 | +0.01 [−0.06, +0.07] | +0.01 [−0.06, +0.08] | −0.01 [−0.09, +0.07] | −0.05 [−0.13, +0.02] | |
| Mistral-7B | 36 | +0.03 [−0.09, +0.15] | −0.03 [−0.19, +0.13] | −0.03 [−0.14, +0.10] | +0.03 [−0.12, +0.19] | |
| NQ-Open | Qwen3-8B | 146 | +0.35 [+0.27, +0.43] | +0.29 [+0.22, +0.38] | +0.22 [+0.14, +0.29] | +0.21 [+0.13, +0.28] |
| Llama-3.1-8B | 128 | +0.15 [+0.06, +0.23] | +0.18 [+0.09, +0.27] | +0.08 [+0.00, +0.16] | +0.12 [+0.04, +0.21] | |
| Mistral-7B | 101 | +0.31 [+0.18, +0.43] | +0.21 [+0.09, +0.34] | +0.14 [+0.02, +0.25] | +0.15 [+0.04, +0.26] |
| Dataset | Model | Target Token | Target +4 Token | Retrieved Token | Retrieved +4 Token | |
|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | −0.04 [−0.14, +0.06] | −0.04 [−0.15, +0.05] | −0.10 [−0.20, −0.01] | −0.09 [−0.18, −0.01] |
| Llama-3.1-8B | 102 | +0.00 [−0.06, +0.06] | −0.02 [−0.08, +0.04] | −0.02 [−0.08, +0.05] | −0.05 [−0.14, +0.04] | |
| Mistral-7B | 36 | +0.00 [−0.11, +0.12] | +0.00 [−0.18, +0.16] | +0.00 [−0.11, +0.11] | +0.08 [−0.06, +0.24] | |
| NQ-Open | Qwen3-8B | 146 | +0.27 [+0.20, +0.36] | +0.33 [+0.25, +0.41] | +0.14 [+0.07, +0.23] | +0.24 [+0.16, +0.32] |
| Llama-3.1-8B | 128 | −0.02 [−0.12, +0.06] | +0.00 [−0.09, +0.09] | −0.09 [−0.19, −0.01] | −0.07 [−0.17, +0.02] | |
| Mistral-7B | 101 | +0.25 [+0.14, +0.36] | +0.25 [+0.14, +0.37] | +0.08 [−0.03, +0.20] | +0.14 [+0.02, +0.26] |
| Dataset | Model | Target+4 Target (5%) | Target+4 Target (10%) | Target+4 random (10%) | Similar Token (10%) | |
|---|---|---|---|---|---|---|
| LoCoMo | Qwen3-8B | 105 | −0.02 [−0.06, +0.04] | +0.00 [−0.05, +0.05] | −0.01 [−0.07, +0.03] | −0.14 [−0.21, −0.07] |
| Llama-3.1-8B | 102 | +0.00 [−0.05, +0.05] | −0.02 [−0.06, +0.02] | −0.05 [−0.08, −0.02] | −0.13 [−0.21, −0.04] | |
| Mistral-7B | 36 | −0.06 [−0.17, +0.05] | +0.00 [−0.13, +0.10] | +0.00 [−0.13, +0.10] | −0.03 [−0.13, +0.06] | |
| NQ-Open | Qwen3-8B | 146 | −0.05 [−0.12, +0.02] | +0.05 [+0.00, +0.11] | +0.05 [−0.01, +0.11] | −0.09 [−0.14, −0.03] |
| Llama-3.1-8B | 128 | +0.03 [−0.02, +0.09] | +0.02 [−0.03, +0.08] | −0.03 [−0.10, +0.04] | −0.20 [−0.29, −0.12] | |
| Mistral-7B | 101 | −0.10 [−0.19, −0.01] | +0.00 [−0.07, +0.07] | −0.01 [−0.09, +0.07] | −0.16 [−0.26, −0.06] |
| First third | Middle third | Last third | ||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Slot 1 | Gain | Gain | Gain | |||
| NQ-Open | Qwen3-8B | 6 | 50 | +0.20 | 36 | +0.47 | 54 | +0.48 |
| Llama-3.1-8B | 5 | 43 | +0.05 | 36 | +0.22 | 44 | +0.34 | |
| Mistral-7B | 1 | 29 | +0.14 | 32 | +0.47 | 39 | +0.56 | |
| Gorilla | Qwen3-8B | 9 | 23 | +0.09 | 30 | +0.23 | 27 | +0.41 |
| Llama-3.1-8B | 8 | 23 | +0.04 | 33 | +0.12 | 36 | +0.36 | |
| Dataset | Model | Token 5% (own rule) | Token 5% (ProphetKV) | Target Token | Target+4 Token | Retrieved Token | Retrieved +4 Token | |
|---|---|---|---|---|---|---|---|---|
| LoCoMo | Mistral-7B | 36 | 0.78 | 0.69 | +0.11 [+0.02, +0.24] | +0.08 [−0.06, +0.24] | +0.06 [−0.05, +0.16] | +0.08 [−0.05, +0.23] |
| Gorilla | Qwen3-8B | 89 | 0.06 | 0.09 | +0.13 [+0.04, +0.22] | +0.15 [+0.07, +0.24] | +0.06 [−0.02, +0.13] | +0.12 [+0.04, +0.20] |
| Gorilla | Llama-3.1-8B | 100 | 0.12 | 0.07 | +0.16 [+0.08, +0.25] | +0.12 [+0.04, +0.20] | +0.08 [+0.01, +0.16] | +0.07 [−0.01, +0.14] |
| Gorilla | Mistral-7B | 108 | 0.10 | 0.12 | +0.46 [+0.35, +0.56] | +0.35 [+0.24, +0.46] | +0.18 [+0.08, +0.27] | +0.19 [+0.10, +0.29] |
| TriviaQA | Mistral-7B | 117 | 0.44 | 0.38 | +0.30 [+0.21, +0.38] | +0.24 [+0.15, +0.32] | +0.17 [+0.09, +0.26] | +0.18 [+0.09, +0.26] |
| 5% budget | 10% budget | |||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Retrieved +4 (%) | Gated Token | Gated Retrieved+4 | Retrieved +4 (%) | Gated Token | Gated Retrieved+4 | |
| LoCoMo | Qwen3-8B | 105 | 0 | +0.00 [+0.00, +0.00] | +0.10 [+0.04, +0.20] | 1 | −0.01 [−0.03, +0.00] | +0.08 [+0.01, +0.16] |
| Llama-3.1-8B | 102 | 0 | +0.00 [+0.00, +0.00] | +0.05 [−0.02, +0.13] | 1 | +0.00 [+0.00, +0.00] | +0.05 [−0.04, +0.14] | |
| Mistral-7B | 36 | 0 | +0.00 [+0.00, +0.00] | −0.03 [−0.19, +0.12] | 3 | +0.00 [+0.00, +0.00] | −0.08 [−0.24, +0.06] | |
| NQ-Open | Qwen3-8B | 146 | 79 | +0.17 [+0.11, +0.24] | −0.03 [−0.08, +0.00] | 84 | +0.23 [+0.16, +0.32] | −0.01 [−0.03, +0.01] |
| Llama-3.1-8B | 128 | 77 | +0.11 [+0.02, +0.20] | −0.02 [−0.04, +0.00] | 84 | −0.08 [−0.16, +0.01] | −0.01 [−0.05, +0.03] | |
| TTFT, ms: median (p95) | Ratio to full prefill | |||||||
|---|---|---|---|---|---|---|---|---|
| Condition | LoCoMo (10.8K) | NQ-Open (4.6K) | Gorilla (5.7K) | TriviaQA (4.7K) | LoCoMo | NQ-Open | Gorilla | TriviaQA |
| Full prefill | 2554 (3136) | 840 (898) | 1072 (1938) | 842 (959) | 1.00 | 1.00 | 1.00 | 1.00 |
| Reuse | 94 (120) | 53 (66) | 58 (81) | 53 (57) | 0.04 | 0.06 | 0.06 | 0.06 |
| Token 5% | 309 (363) | 162 (198) | 129 (189) | 165 (176) | 0.12 | 0.19 | 0.12 | 0.19 |
| Target 5% | 268 (324) | 92 (114) | 103 (149) | 94 (104) | 0.11 | 0.11 | 0.09 | 0.11 |
| Retrieved 5% | 282 (352) | 105 (130) | 117 (165) | 108 (114) | 0.11 | 0.13 | 0.11 | 0.13 |
| Retrieved chunk, 5% budget | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Hit (%) | Token 5% | In place Token | Moved Token | Moved only, no recomp. | Moved in place | Moved Token (hits) | Moved Token (misses) | |
| NQ-Open | Qwen3-8B | 146 | 55 | 0.16 | +0.22 [+0.14, +0.29] | +0.40 [+0.31, +0.49] | +0.00 [−0.02, +0.02] | +0.18 [+0.11, +0.25] | +0.75 [+0.65, +0.85] | −0.03 [−0.12, +0.06] |
| Llama-3.1-8B | 128 | 47 | 0.21 | +0.08 [+0.00, +0.16] | +0.25 [+0.15, +0.35] | +0.04 [−0.01, +0.09] | +0.17 [+0.10, +0.24] | +0.63 [+0.50, +0.75] | −0.09 [−0.18, +0.00] | |
| Mistral-7B | 101 | 50 | 0.28 | +0.14 [+0.02, +0.25] | +0.27 [+0.14, +0.39] | +0.09 [+0.02, +0.16] | +0.13 [+0.03, +0.22] | +0.65 [+0.49, +0.78] | −0.12 [−0.26, +0.00] | |
| Gorilla | Qwen3-8B | 89 | 53 | 0.06 | +0.09 [+0.00, +0.18] | +0.21 [+0.12, +0.30] | +0.02 [+0.00, +0.06] | +0.12 [+0.03, +0.22] | +0.43 [+0.28, +0.57] | −0.02 [−0.07, +0.00] |
| Llama-3.1-8B | 100 | 53 | 0.12 | +0.02 [−0.07, +0.10] | +0.15 [+0.06, +0.24] | +0.00 [−0.04, +0.04] | +0.13 [+0.04, +0.22] | +0.34 [+0.23, +0.47] | −0.06 [−0.17, +0.02] | |
| Retention, 5% budget | Chunk moved Token moved | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Token original | Token moved | Chunk moved | Token moved Token | All | Hits | Misses | |
| NQ-Open | Qwen3-8B | 146 | 0.16 | 0.27 | 0.56 | +0.11 [+0.05, +0.17] | +0.29 [+0.21, +0.37] | +0.56 [+0.45, +0.68] | −0.05 [−0.12, +0.03] |
| Llama-3.1-8B | 128 | 0.21 | 0.46 | 0.46 | +0.25 [+0.16, +0.34] | +0.00 [−0.09, +0.09] | +0.08 [−0.05, +0.22] | −0.07 [−0.18, +0.03] | |
| Mistral-7B | 101 | 0.28 | 0.51 | 0.54 | +0.24 [+0.13, +0.34] | +0.03 [−0.08, +0.13] | +0.22 [+0.08, +0.35] | −0.16 [−0.30, −0.02] | |
| Gorilla | Qwen3-8B | 89 | 0.06 | 0.06 | 0.27 | +0.00 [−0.03, +0.03] | +0.21 [+0.12, +0.31] | +0.45 [+0.30, +0.60] | −0.05 [−0.12, +0.00] |
| Llama-3.1-8B | 100 | 0.12 | 0.34 | 0.27 | +0.22 [+0.12, +0.32] | −0.07 [−0.16, +0.02] | −0.09 [−0.25, +0.06] | −0.04 [−0.13, +0.04] | |
| Hit (%) | Retention | Chunk moved InfoFlow style | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Retrieved | InfoFlow | Token 5% | InfoFlow 5% | InfoFlow 15% | Chunk moved 5% | at 5% | at 15% | InfoFlow 5% Token | InfoFlow chunk Chunk moved | |
| NQ-Open | Qwen3-8B | 146 | 55 | 52 | 0.16 | 0.40 | 0.66 | 0.56 | +0.16 [+0.07, +0.26] | −0.10 [−0.20, +0.00] | +0.23 [+0.16, +0.32] | −0.01 [−0.11, +0.08] |
| Llama-3.1-8B | 128 | 47 | 39 | 0.21 | 0.40 | 0.59 | 0.46 | +0.06 [−0.04, +0.16] | −0.12 [−0.22, −0.02] | +0.19 [+0.10, +0.27] | −0.08 [−0.17, +0.02] | |
| Mistral-7B | 101 | 50 | 43 | 0.28 | 0.38 | 0.53 | 0.54 | +0.17 [+0.02, +0.31] | +0.01 [−0.13, +0.16] | +0.10 [+0.00, +0.20] | −0.07 [−0.20, +0.07] | |
| Gorilla | Qwen3-8B | 89 | 53 | 24 | 0.06 | 0.10 | 0.28 | 0.27 | +0.17 [+0.07, +0.27] | −0.01 [−0.11, +0.09] | +0.04 [−0.02, +0.11] | −0.13 [−0.24, −0.03] |
| Llama-3.1-8B | 100 | 53 | 15 | 0.12 | 0.13 | 0.27 | 0.27 | +0.14 [+0.05, +0.23] | +0.00 [−0.11, +0.11] | +0.01 [−0.08, +0.10] | −0.15 [−0.24, −0.06] | |
| Dataset | Model | Hit (%) top-1/3 | chunk | chunks | tokens | chunks | Predicted | Chunks tokens, | chunks Token, orig. | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| NQ-Open | Qwen3-8B | 146 | 55/75 | 0.93 0.12 | 0.56 | 0.67 | 0.50 | +0.11 [+0.04, +0.18] | +0.16 | +0.17 [+0.09, +0.25] | +0.43 [+0.34, +0.53] |
| Llama-3.1-8B | 128 | 47/70 | 0.83 0.13 | 0.46 | 0.52 | 0.65 | +0.06 [−0.02, +0.14] | +0.16 | −0.12 [−0.20, −0.05] | +0.14 [+0.04, +0.23] | |
| Mistral-7B | 101 | 50/71 | 0.92 0.16 | 0.54 | 0.66 | 0.66 | +0.12 [+0.03, +0.21] | +0.16 | +0.00 [−0.09, +0.09] | +0.33 [+0.21, +0.46] | |
| TriviaQA | Qwen3-8B | 151 | 64/85 | 0.84 0.19 | 0.60 | 0.74 | 0.69 | +0.13 [+0.07, +0.20] | +0.14 | +0.05 [−0.03, +0.12] | +0.25 [+0.15, +0.34] |
| Llama-3.1-8B | 127 | 60/79 | 0.89 0.16 | 0.60 | 0.69 | 0.74 | +0.09 [+0.01, +0.17] | +0.14 | −0.06 [−0.13, +0.02] | +0.13 [+0.03, +0.24] | |
| Mistral-7B | 117 | 62/85 | 0.88 0.18 | 0.62 | 0.77 | 0.74 | +0.15 [+0.09, +0.22] | +0.15 | +0.03 [−0.04, +0.09] | +0.25 [+0.15, +0.35] |
| Hit top-1/top-3 (%) | Top-1 moved, retention at 10% | Top-3 top-1 moved at 10% | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | BGE-M3 | BM25 | Predicted | Measured | Predicted | Measured | Top-1 moved, BM25 BGE | |
| NQ-Open | Qwen3-8B | 146 | 55/75 | 23/47 | 0.31 | 0.32 [+0.25, +0.40] | +0.19 | +0.18 [+0.12, +0.26] | −0.24 [−0.33, −0.15] |
| Llama-3.1-8B | 128 | 47/70 | 23/46 | 0.29 | 0.28 [+0.20, +0.35] | +0.16 | +0.19 [+0.11, +0.27] | −0.18 [−0.27, −0.09] | |
| Mistral-7B | 101 | 50/71 | 25/41 | 0.35 | 0.38 [+0.28, +0.48] | +0.12 | +0.07 [−0.02, +0.15] | −0.17 [−0.28, −0.05] | |
| Gorilla | Qwen3-8B | 89 | 53/74 | 54/72 | 0.27 | 0.31 [+0.22, +0.42] | +0.08 | +0.11 [+0.01, +0.21] | +0.03 [−0.06, +0.12] |
| Llama-3.1-8B | 100 | 53/69 | 52/67 | 0.27 | 0.28 [+0.19, +0.37] | +0.06 | +0.07 [−0.02, +0.16] | +0.01 [−0.06, +0.09] | |
| Loss and two terms | 5%, original order | Retrieved chunk moved last | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Loss | Same | Sim. | Target Token | Retrieved Token | Chunk moved Token | Token moved Token | Chunk moved Token moved | |
| LoCoMo | 14B | 74 | +0.36 [+0.21, +0.51] | +0.20 [+0.08, +0.31] | +0.03 [−0.07, +0.12] | +0.14 [+0.05, +0.22] | +0.05 [−0.05, +0.17] | – | – | – |
| 32B | 67 | +0.46 [+0.35, +0.54] | +0.16 [+0.02, +0.30] | +0.18 [+0.05, +0.31] | +0.07 [−0.01, +0.16] | +0.00 [−0.04, +0.04] | – | – | – | |
| 8B | 105 | +0.47 | +0.19 | +0.16 | −0.04 | −0.10 | – | – | – | |
| NQ-Open | 14B | 150 | +0.79 [+0.72, +0.85] | +0.17 [+0.09, +0.25] | +0.08 [−0.01, +0.17] | +0.22 [+0.13, +0.30] | +0.13 [+0.05, +0.21] | +0.34 [+0.24, +0.43] | +0.24 [+0.16, +0.32] | +0.10 [+0.03, +0.17] |
| 32B | 138 | +0.78 [+0.71, +0.85] | +0.37 [+0.28, +0.46] | +0.08 [−0.01, +0.18] | +0.38 [+0.28, +0.46] | +0.23 [+0.14, +0.31] | +0.35 [+0.25, +0.44] | +0.25 [+0.17, +0.33] | +0.09 [+0.01, +0.17] | |
| Retention | KV dist. | Attention on target | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | None | Target KV | Recomp. | Recomp. on full | Recomp. Target KV | None | Recomp. | None | Target KV | Rec. | Full | |
| NQ-Open | Qwen3-8B | 40 | 0.15 | 0.15 | 0.53 | 0.97 | +0.38 [+0.23, +0.53] | 0.36 | 0.26 | 0.035 | 0.021 | 0.050 | 0.060 |
| NQ-Open | Llama-3.1-8B | 30 | 0.20 | 0.20 | 0.33 | 1.00 | +0.13 [+0.03, +0.27] | 0.40 | 0.28 | 0.029 | 0.020 | 0.025 | 0.064 |
| NQ-Open | Mistral-7B | 40 | 0.23 | 0.42 | 0.62 | 0.97 | +0.20 [+0.05, +0.35] | 0.40 | 0.27 | 0.031 | 0.019 | 0.030 | 0.059 |
| Gorilla | Qwen3-8B | 40 | 0.00 | 0.05 | 0.25 | 0.95 | +0.20 [+0.10, +0.33] | 0.42 | 0.30 | 0.040 | 0.027 | 0.065 | 0.053 |
| Gorilla | Mistral-7B | 40 | 0.05 | 0.38 | 0.55 | 1.00 | +0.17 [+0.03, +0.33] | 0.45 | 0.29 | 0.044 | 0.034 | 0.045 | 0.068 |
| Earlier chunks reused | Earlier chunks full | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | Target reused | Target KV | Target reused | Target KV | Target recomp. | Full target effect (reused) | Full target effect (full) | Interaction | |
| NQ-Open | Qwen3-8B | 40 | 0.15 | 0.15 | 0.28 | 0.55 | 0.53 | +0.00 [−0.12, +0.12] | +0.28 [+0.12, +0.42] | +0.28 [+0.15, +0.40] |
| NQ-Open | Llama-3.1-8B | 30 | 0.20 | 0.20 | 0.27 | 0.43 | 0.33 | +0.00 [−0.17, +0.17] | +0.17 [+0.00, +0.33] | +0.17 [−0.03, +0.37] |
| NQ-Open | Mistral-7B | 40 | 0.23 | 0.42 | 0.33 | 0.55 | 0.62 | +0.20 [+0.03, +0.40] | +0.23 [+0.10, +0.35] | +0.03 [−0.12, +0.17] |
| Gorilla | Qwen3-8B | 40 | 0.00 | 0.05 | 0.15 | 0.45 | 0.25 | +0.05 [+0.00, +0.12] | +0.30 [+0.17, +0.45] | +0.25 [+0.12, +0.38] |
| Gorilla | Mistral-7B | 40 | 0.05 | 0.38 | 0.07 | 0.60 | 0.55 | +0.33 [+0.17, +0.47] | +0.53 [+0.38, +0.68] | +0.20 [+0.03, +0.35] |
KV-Cache Reuse
Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.
See all 81 papers