Recent work in machine learning has demonstrated that transformer-based language model agents can communicate state not only through text, but also directly in latent space via key-value (KV) caches. In this paper, we argue that such cache-to-cache translation can provide utility beyond the original motivating application of real-time agent communication and can instead be utilized for retrieval-augmented generation (RAG). Building on the Cache-to-Cache (C2C) fusion framework, our proposed system CacheGraft encodes each corpus passage offline into a fused KV cache representation between a larger sharer model (4B parameters in our experiments) and a smaller receiver model (0.6B parameters) such that this cache can be injected into the receiver at query time with no need for the larger model. Across five QA benchmarks and two independent locally-run judges, CacheGraft improves answer quality over the text-only receiver baseline in 29 out of 30 comparisons, of which 17 of the 30 are statistically significant. Notably, these gains are zero-shot with respect to the fuser, which was never trained on retrieval data. These initial results suggest that latent-space document enrichment may be a promising avenue for enhancing the performance of RAG systems, particularly in compute-constrained settings where it is only feasible to deploy a small model at inference time.
Figures & tables
Figure 1. The sharer and fuser run only offline, once per corpus passage, with no knowledge of any query. The fused KV cache is the only artifact that crosses the offline/online boundary: at query time the receiver loads it and decodes alone, reading the passage [pd] in latent space and the question [q] as text.
phi-4
Gemma
contains
Dataset
Setting
n
Top-1 Recall
Receiver-Only
CacheGraft
Sharer-Only
Δ
Receiver-Only
CacheGraft
Sharer-Only
Δ
Receiver-Only
CacheGraft
Sharer-Only
Δ
AdversarialQA
retrieved
416
59.4
30.8
36.8
41.8
+6.0∗
34.9
38.9
48.3
+4.1
24.5
30.3
33.7
+5.8∗
oracle
—
42.1
49.3
59.1
+7.2∗
44.2
51.4
63.9
+7.2∗
33.9
42.5
48.3
+8.7∗
SQuAD
retrieved
800
68.1
51.8
59.6
66.4
+7.9∗
59.2
62.5
70.8
+3.2∗
48.6
55.5
61.1
+6.9∗
oracle
—
73.0
79.4
86.5
+6.4∗
78.6
81.8
89.4
+3.1
69.9
76.9
81.2
+7.0∗
CovidQA
retrieved
800
53.2
33.1
36.9
41.9
+3.8∗
50.8
53.8
59.8
+3.0
21.0
22.6
24.4
+1.6
Table 1. Performance of CacheGraft in comparison to using only the 0.6B receiver model and only the 4B sharer model across two LLM judges and the contains metric, which measures if the correct answer is included in the output text. CacheGraft improvements over the receiver-only baseline are marked in bold. Δ reports CacheGraft minus Receiver-Only for each metric.
Over how many species of trees can be found in the Saxon Garden? (gold: 100)
text-RAG : 15.5 ×
CacheGraft : 100 ✓
Where was the Continental Edison Company located? (gold: France)
text-RAG : New York City. ×
CacheGraft : The Continental Edison Company was located in France. ✓
Who did Rollo sign the treaty of Saint-Clair-sur-Epte with? (gold: King Charles III)
Table 2. Example outputs of CacheGraft and the receiver-only Text-RAG baseline. The last two rows are failure cases: in each, CacheGraft answers with a different span of the same passage.
Dataset
Headroom
Answer-in-doc
Gain
AdversarialQA
+14.4
+34.6
+7.2
SQuAD
+11.3
+69.0
+6.4
CovidQA
+4.4
+34.1
+5.4
TriviaQA
+20.5
+40.7
+3.2
QASPER
+2.0
+30.2
+2.9
Table 3. Relationship between CacheGraft’s gains and the opportunity for improvement, all computed in the oracle-retrieval setting. Headroom is Sharer-Only minus Receiver-Only under contains. Answer-in-doc is the sharer’s contains improvement when given the gold passage versus answering closed-book (no passage). Gain is CacheGraft’s improvement over Receiver-Only under the phi-4 judge.
System
Latency (ms)
GPU Memory (GB)
# Tokens
Receiver text-RAG (0.6B)
334
1.22
8.2
CacheGraft (0.6B + Cache)
574
1.27
14.3
Sharer text-RAG (4B)
652
7.62
12.6
Table 4. Query-time cost, batch size 1, greedy decoding with 24 maximum new tokens, warm, over 100 SQuAD questions on one A100-40GB.