Recent work in machine learning has demonstrated that transformer-based language model agents can communicate state not only through text, but also directly in latent space via key-value (KV) caches. In this paper, we argue that such cache-to-cache translation can provide utility beyond the original motivating application of real-time agent communication and can instead be utilized for retrieval-augmented generation (RAG). Building on the Cache-to-Cache (C2C) fusion framework, our proposed system CacheGraft encodes each corpus passage offline into a fused KV cache representation between a larger sharer model (4B parameters in our experiments) and a smaller receiver model (0.6B parameters) such that this cache can be injected into the receiver at query time with no need for the larger model. Across five QA benchmarks and two independent locally-run judges, CacheGraft improves answer quality over the text-only receiver baseline in 29 out of 30 comparisons, of which 17 of the 30 are statistically significant. Notably, these gains are zero-shot with respect to the fuser, which was never trained on retrieval data. These initial results suggest that latent-space document enrichment may be a promising avenue for enhancing the performance of RAG systems, particularly in compute-constrained settings where it is only feasible to deploy a small model at inference time.
Figures & tables
Figure 1. The sharer and fuser run only offline, once per corpus passage, with no knowledge of any query. The fused KV cache is the only artifact that crosses the offline/online boundary: at query time the receiver loads it and decodes alone, reading the passage [pd] in latent space and the question [q] as text.
phi-4
Gemma
contains
Dataset
Setting
n
Top-1 Recall
Receiver-Only
CacheGraft
Sharer-Only
Δ
Receiver-Only
CacheGraft
Sharer-Only
Δ
Receiver-Only
CacheGraft
Sharer-Only
Δ
AdversarialQA
retrieved
416
59.4
30.8
36.8
41.8
+6.0∗
34.9
38.9
48.3
+4.1
24.5
30.3
33.7
+5.8∗
oracle
—
42.1
49.3
59.1
+7.2∗
44.2
51.4
63.9
+7.2∗
33.9
42.5
48.3
+8.7∗
SQuAD
retrieved
800
68.1
51.8
59.6
66.4
+7.9∗
59.2
62.5
70.8
+3.2∗
48.6
55.5
61.1
+6.9∗
oracle
—
73.0
79.4
86.5
+6.4∗
78.6
81.8
89.4
+3.1
69.9
76.9
81.2
+7.0∗
CovidQA
retrieved
800
53.2
33.1
36.9
41.9
+3.8∗
50.8
53.8
59.8
+3.0
21.0
22.6
24.4
+1.6
Table 1. Performance of CacheGraft in comparison to using only the 0.6B receiver model and only the 4B sharer model across two LLM judges and the contains metric, which measures if the correct answer is included in the output text. CacheGraft improvements over the receiver-only baseline are marked in bold. Δ reports CacheGraft minus Receiver-Only for each metric.
Over how many species of trees can be found in the Saxon Garden? (gold: 100)
text-RAG : 15.5 ×
CacheGraft : 100 ✓
Where was the Continental Edison Company located? (gold: France)
text-RAG : New York City. ×
CacheGraft : The Continental Edison Company was located in France. ✓
Who did Rollo sign the treaty of Saint-Clair-sur-Epte with? (gold: King Charles III)
Table 2. Example outputs of CacheGraft and the receiver-only Text-RAG baseline. The last two rows are failure cases: in each, CacheGraft answers with a different span of the same passage.
Dataset
Headroom
Answer-in-doc
Gain
AdversarialQA
+14.4
+34.6
+7.2
SQuAD
+11.3
+69.0
+6.4
CovidQA
+4.4
+34.1
+5.4
TriviaQA
+20.5
+40.7
+3.2
QASPER
+2.0
+30.2
+2.9
Table 3. Relationship between CacheGraft’s gains and the opportunity for improvement, all computed in the oracle-retrieval setting. Headroom is Sharer-Only minus Receiver-Only under contains. Answer-in-doc is the sharer’s contains improvement when given the gold passage versus answering closed-book (no passage). Gain is CacheGraft’s improvement over Receiver-Only under the phi-4 judge.
System
Latency (ms)
GPU Memory (GB)
# Tokens
Receiver text-RAG (0.6B)
334
1.22
8.2
CacheGraft (0.6B + Cache)
574
1.27
14.3
Sharer text-RAG (4B)
652
7.62
12.6
Table 4. Query-time cost, batch size 1, greedy decoding with 24 maximum new tokens, warm, over 100 SQuAD questions on one A100-40GB.
Multi-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality. Selective recomputation methods recover the missing cross-chunk context by rerunning the target LLM on selected tokens, incurring substantial online computation. We introduce CacheRepair, a lightweight network that learns the difference between independently computed KV caches and those produced by processing the chunks together. The network combines compressed KV features with token embeddings and uses attention that is bidirectional within each chunk and flows from earlier to later chunks. Each repair block receives the compressed cache features, and the predicted residual is added to every document token's cache. Each repair network is trained for a specific frozen target LLM on a generic retrieval corpus and reused across downstream datasets. Our analysis shows that repair reduces KV errors both near chunk boundaries and throughout chunk interiors. Evaluation across three target LLMs and four downstream datasets places CacheRepair on the measured answer-quality-latency Pareto frontier in eleven of twelve model-dataset combinations. Reported time to first token (TTFT) includes online cache transfer and repair. Across all twelve combinations, the largest repairers achieve 1.69-4.61× speedups in median TTFT over full prefill and improve mean F1 by 2.1-26.1 percentage points over direct cache reuse.
Retrieval-augmented generation (RAG) improves large language model (LLM) answer quality by grounding generation in external evidence, but processing retrieved contexts makes the prefill stage a dominant serving cost. RAG cache fusion reduces this cost by reusing precomputed key-value (KV) caches for retrieved chunks and selectively recomputing tokens under the current prompt. Existing selectors, however, face a dilemma between quality and efficiency: fast query-agnostic or final-layer query-to-context selectors can miss request-relevant evidence, whereas full-view query-aware selectors require broad context and layer visibility before recomputation and therefore stall the layer-wise cache-fusion pipeline. We present QCFuse, a compressed-view query-aware selector for RAG cache fusion. QCFuse uses chunk-anchor query probing to condition user-query states on compact per-chunk anchors and critical-layer profiling to identify recomputation tokens without all-layer inspection. We implement QCFuse in SGLang and evaluate it on four open-weight LLMs across six datasets. QCFuse reaches full-prefill-level quality. At matched quality, QCFuse achieves an average prefill-time speedup of 1.7x over full prefill and 1.5x over ProphetKV, the strongest quality-preserving baseline.
Jianxin Yan, Wangze Ni, Zhenxin Li +8
Zhejiang University, Hangzhou, China · Ant Group, Shanghai, China · The Hong Kong Polytechnic University, Hong Kong, China +5
Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing large language models (LLMs) with external knowledge, mitigating hallucinations, and improving factuality. However, existing systems rely on generating natural language queries at each hop and maintaining a strict architectural separation between retriever and generator, preventing them from leveraging the full representational capacity of the LLM. We propose LAnR (Latent Abstraction for RAG), a unified framework in which a single LLM jointly performs encoding, retrieval, and generation entirely within its own latent space. Rather than generating textual queries, LAnR produces dense retrieval vectors from the hidden states of a designated [PRED] token and uses them to match against encoded document representations from the same model. Furthermore, LAnR adaptively decides when sufficient evidence has been retrieved using a lightweight MLP control head over those same hidden states, eliminating both the separate retriever and explicit token-level stopping reasoning. This design is motivated by our empirical observation that answer token entropy reliably signals retrieval sufficiency. Extensive experiments on six QA benchmarks spanning single-hop and multi-hop settings demonstrate that LAnR outperforms existing RAG methods, while achieving improved inference efficiency through reduced number of retrieval calls and tighter model integration.