Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.
Figures & tables
Figure 1. Top: KV cache and attention profile with full-prefill. Middle: CLC with naive KV cache reuse lacks cross-chunk attention, and obtains different attention profiles. Bottom: selective recomputation aims to recover key cross-chunk attention information; attention reshaping aims to obtain an attention profile similar to full prefill’s.
Taxonomy
System
Technique
Selective Recomputation
Cacheblend ( Yao et al., 2025 )
Recompute KV cache entries with highest deviation from full-prefill
Epic ( Hu et al., 2024 )
Recompute beginning-of-chunk KV cache entries
Cacheclip ( Yang et al., 2025b )
Choose recomputation tokens at last layer of auxiliary model
Droidspeak ( Liu et al., 2025 )
Re-evaluate chosen tokens at mid layer
Reshape attention: modify attn. scores
Link0 ( Hu et al., 2024 )
Absorbs abnormally high attention to beginning-of-chunk-tokens
APE ( Yang et al., 2025d )
Rescales attention to resemble full-prefill one
Table 1. State-of-the-art CLC approaches.
Figure 2. Cacheblend accuracy.
Figure 3. Ideal token selection using ΔK ( r=15% ) for one query in Llama-2WikiMQA (left) and Qwen-Musique (right). Black/white: a token is/is not selected at that layer. The clusters of chosen tokens at early layers correspond to beginning-of-chunk tokens.
Figure 4. Left: EPIC/Link0 accuracy. Center: cumulative percentage of tokens’ ΔK for Llama on 2WikiMQA and Qwen on Musique, with and without the first 10 tokens (sink) of each chunk. Right: CacheClip accuracy.
Figure 5. Average percentage of the tokens chosen (by ΔK ) for recomputation at a reference layer that would be chosen also in other layers ( r=15% ).
Figure 6. Droidspeak accuracy.
Figure 7. APE accuracy.
Figure 8. BlockAttention’s accuracy on fine-tuned LLama.