cs.CLNov 4, 2026

Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse

Authors: Yuxing Peng, Zhiqing Tang, Weijia Jia

Organizations: Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China

Abstract

Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B models and four datasets of dialogue memory, open-domain QA and API documentation. Reuse loses 19-100% of the context-dependent questions that full prefill answers, and it loses more when the other chunks are of the same kind as the target. Because the recoverable damage is spread over these other chunks, restoring a few of them recovers little, and the budget is better spent on the target. This works through contextual compatibility: recomputing the target against the reused cache beats inserting its full-prefill KV in all seven settings tested, although the recomputed KV stays far from the full-prefill KV. Question attention over reused caches, however, selects the target in at most 18.8% of development questions. A retriever finds it more often, and recomputing its top chunk after moving it next to the question significantly beats InfoFlow-style reordering at the same 5% budget in six of nine open-domain QA and API-documentation settings. On dialogue memory, though, recomputing chunks never significantly beats the token-level selector chosen for each setting.

Figures & tables

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Budgeted Cache Repair for Cross-Context KV-Cache Reuse

    Sep 27, 2026Haeyong Kang, Chang D. YooKV-Cache ManagementKV-Cache Reuse

  2. KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

    Oct 9, 2026Taiga Ikeda, Daisuke Miyashita, Jun DeguchiKV-Cache ReuseKV-Cache Management

  3. Budgeted KV-Cache Repair for Evolving Contexts

    Oct 12, 2026Han Yu, Wenhui Zhu, Jincheng Cao +15KV-Cache ManagementLLM Inference Acceleration