EchoKV: Self-Corrective KV Cache Eviction via Iterative Query Feedback
Organizations: Beijing University of Technology
Abstract
KV cache eviction is not merely a token-ranking problem: the workspace retained for generating eviction queries also determines the quality of those queries. Consequently, an early workspace error can distort later query observations and reinforce an incorrect final eviction decision, particularly when evidence is dispersed across a long context. We present EchoKV, which discovers and addresses this failure mode with iterative correction. It initializes a feedback-compatible bootstrap workspace using proposals at future decode positions, then uses early decode-generated queries to recover the workspace before the remaining lookahead steps. It uses the same number of workspace forward passes as LAQ while adding a tensor re-scoring operation for workspace repair. On LongBench (16 tasks, Mistral-7B-Instruct-v0.2 and Qwen2.5-7B-Instruct), EchoKV shows notable improvements over LAQ on several tasks requiring dispersed evidence, including multi-hop cross-document QA and long single-document comprehension. Across budgets and backbones, EchoKV selectively improves evidence-intensive tasks while maintaining comparable performance on the remaining LongBench subsets. These results identify query--workspace coupling as a practical failure mode of query-driven KV eviction.
Figures & tables
| KV overlap at step | ||||
|---|---|---|---|---|
| Workspace strategy | ||||
| Frozen ( steps on ) | 0.92 | 0.88 | 0.83 | 0.77 |
| Refreshed at (ours) | 0.92 | 0.88 | 0.85 | 0.80 |
| Layer | Locked | Linear | Prompt |
|---|---|---|---|
| 0 | 0.804 | 0.853 | 0.927 |
| 8 | 0.472 | 0.430 | 0.631 |
| 16 | 0.275 | 0.334 | 0.599 |
| 24 | 0.447 | 0.368 | 0.541 |
| 31 | 0.325 | 0.328 | 0.427 |
| Budget | Single-Doc | Multi-Doc | Summ | Few-shot | Synth | Code | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Nrtv | Qas | MF | Hot | 2Wiki | Mus | Gov | QM | MN | TREC | TQA | SAM | PC | PR | Lcc | RB | Avg | |
| FullKV | – | 26.90 | 33.14 | 49.26 | 42.77 | 27.35 | 18.77 | 32.95 | 24.21 | 27.13 | 71.00 | 86.23 | 42.72 | 2.75 | 86.98 | 57.12 | 54.51 | 42.74 |
| StreamingLLM | 128 | 17.12 | 13.43 | 27.31 | 29.54 | 21.91 | 11.94 | 15.61 | 19.18 | 17.72 | 44.00 | 79.92 | 37.37 | 3.50 | 23.77 | 51.44 | 45.95 | 28.73 |
| H 2 O | 128 | 21.62 | 21.34 | 38.61 | 30.46 | 20.38 | 12.20 | 20.59 | 22.51 | 22.03 | 39.00 | 82.33 | 40.64 | 3.06 | 80.56 | 51.17 | 48.46 | 34.69 |
| SnapKV | 128 | 19.71 | 21.13 | 42.75 | 36.45 | 22.36 | 15.76 | 19.05 | 21.81 | 21.36 | 47.50 | 84.15 | 40.29 | 2.41 | 68.26 | 52.26 | 48.75 | 35.25 |
| PyramidKV | 128 | 21.98 | 22.78 | 43.78 | 32.30 | 22.31 | 15.81 | 20.41 | 21.82 | 21.23 | 66.00 | 83.51 | 39.83 | 2.99 | 65.81 | 51.61 | 46.42 | 36.16 |
| Budget | Single-Doc | Multi-Doc | Summ | Few-shot | Synth | Code | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Nrtv | Qas | MF | Hot | 2Wiki | Mus | Gov | QM | MN | TREC | TQA | SAM | PC | PR | Lcc | RB | Avg | |
| FullKV | – | 17.07 | 43.76 | 52.61 | 57.70 | 47.13 | 29.85 | 32.00 | 23.61 | 23.95 | 39.50 | 87.65 | 40.08 | 8.50 | 100.00 | 6.62 | 9.79 | 38.74 |
| StreamingLLM | 128 | 11.77 | 23.65 | 26.44 | 40.90 | 37.09 | 16.42 | 16.12 | 18.03 | 14.69 | 12.00 | 76.26 | 35.50 | 8.50 | 24.50 | 10.26 | 13.44 | 24.10 |
| H 2 O | 128 | 16.61 | 29.58 | 37.25 | 51.04 | 42.14 | 20.82 | 20.33 | 22.05 | 18.07 | 18.50 | 80.85 | 39.04 | 8.50 | 98.00 | 6.29 | 9.77 | 32.43 |
| SnapKV | 128 | 16.23 | 32.72 | 45.14 | 52.95 | 44.10 | 24.16 | 19.27 | 20.81 | 17.27 | 21.50 | 85.83 | 37.75 | 8.50 | 95.00 | 7.26 | 11.28 | 33.74 |
| PyramidKV | 128 | 15.40 | 30.67 | 44.89 | 51.54 | 41.22 | 25.62 | 19.51 | 19.71 | 16.69 | 25.75 | 84.22 | 34.08 | 8.50 | 96.00 | 5.29 | 6.32 | 32.84 |
| Method | MultifieldQA | HotpotQA | AVG |
|---|---|---|---|
| LAQ ( , baseline) | 45.43 | 40.35 | 39.29 |
| + EchoKV-Stage1 only | 46.15 | 41.24 | 39.38 |
| + Iterative only | 46.27 | 40.98 | 39.42 |
| EchoKV | 46.45 | 41.89 | 39.55 |
| Task | LAQ W | EchoKV W1 | EchoKV W2 |
|---|---|---|---|
| HotpotQA | 68.4 | 72.1 | 76.8 |
| 2WikiMQA | 66.9 | 71.2 | 75.4 |
| MultifieldQA | 70.6 | 73.5 | 77.0 |
| Qasper | 74.2 | 76.0 | 78.1 |
| Average | 70.0 | 73.2 | 76.8 |
KV-Cache Eviction
Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.
See all 104 papers