cs.AIOct 8, 2026

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Authors: Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel

Organizations: University of Southern California · University of California, Irvine · Intel Labs

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23x parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8x gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04x prefill speedup. For online serving, it handles 1.8x the request throughput of target prefill on a single GPU, and at the target's saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0x and 6.4x respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.

Figures & tables

Figure 1: The figure illustrates our RaReCache pipeline where target cache constitutes mapped cache tokens from source and recomputed tokens in target (left). RaReCache closes accuracy gap to target while yielding 2.6×2.6\times lower prefill FLOPs (right).
Figure 2: GSM8K acc. for mapping source KV to target on Qwen3. Marker size is proportional to the parameter count of prefill model.
Figure 3: Token-by-token scores for each selector on a single GSM8K prompt. Columns are selected tokens from four sections of the prompt (template, question, instruction, and header). The first four rows marks the tokens in the top 20% positions determined by four different selection metrics. The last three rows are tokenwise scores for three of the four metrics, excluding the random selection metric which has no associated score.
Figure 4: GSM8K accuracy of Qwen3-0.6B →\to 14B when 30% of prompt positions are recomputed, chosen by four selectors. Arrows give the remaining gap to the 14B alone.
Figure 5: The RaReCache Pipeline: The source model prefills the prompt, concatenating KK selected layers into input xix_{i} . An offline-fitted linear map generates both a full-rank target cache prediction ( y^i\hat{y}_{i} ) and a reduced-rank projection ( y^i(r)\hat{y}_{i}^{(r)} ). Rank disagreement ( sis_{i} ) calculates the squared gap between these predictions. As illustrated, tokens whose predictions project outside the familiar calibration subspace span(Vr)\text{span}(V_{r}) yield high sis_{i} scores. To save compute, the target model only runs a forward pass on the top ⌈ρT⌉\lceil\rho T\rceil positions, ρ\rho being the recompute ratio, with the highest sis_{i} , merging these recomputed KV tensors with the directly mapped ones to form the final target model cache.
Figure 6: Accuracy or F1 score against the recompute fraction ρ\rho for three Qwen3 sources mapped into Qwen3-14B. Dashed lines are each source answering alone; the solid grey line is the 14B alone. ρ=0\rho{=}0 reproduces results from Heo et al. (2026) in our setting.
   Transfer, recompute fraction ρ\rho 
Pair ( nn )GapSource alone000.20.20.30.30.40.4Target alone
3.2-3B →\to 3.1-8B (1319)2.5×2.5\times82.381.7 (95.2)85.7 (99.8)84.2 (98.1)86.4 (100.6)85.8
3.1-8B →\to 3.3-70B (300)8.8×8.8\times82.781.0 (85.0)85.3 (89.5)88.3 (92.7)92.0 (96.5)95.3
Table 1: GSM8K Evaluation on Llama 3: Accuracy (%) against the recompute fraction ρ\rho , retention relative to the target is in parentheses. The 8B →\to 70B accuracies are reported on the first 300 test items.
Figure 7: Selectors at a matched budget ( ρ=0.3\rho{=}0.3 ), for 300 samples on GSM8K, transfer mapping Qwen3-0.6B →\to 14B. The shaded band marks random selection.
TargetRatioAloneTransfer ( ρ=0\rho{=}0 )LiftTransfer ( ρ=0.2\rho{=}0.2 )
Qwen3-0.6B23×23\times35.027.2−7.8-7.827.9
Qwen3-1.7B8.2×8.2\times57.571.9+14.4\mathbf{+14.4}71.4
Qwen3-4B3.5×3.5\times70.273.3+3.1+3.173.2
Table 2: Large-to-small transfer , Qwen3-14B →\to each smaller model on MMLU-Redux. Alone is the small model prefilling its own cache. Lift is transfer minus alone.
Figure 8: Comparison against existing selective recomputation strategies for Qwen3-0.6B →\to Qwen3-14B for different recompute ratios for 300 GSM8K samples.
Figure 9: (a) Prefill speedup across recompute fractions ρ\rho (from § 4.2 shaded); (b, c) Median and p99 TTFT vs request rate under Poisson arrivals ( T=149T{=}149 , ρ=0.3\rho{=}0.3 ). The shaded region shows the divergence of the target prefill queue while RaReCache operates at lower device utilization.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: What RaReCache costs, relative to the target prefill. Phase-by-phase runtime across implementation variants at increasing context lengths ( T=149T{=}149 is the mean GSM8K prompt length), with numbers above the bars showing overall speedups. While naive FP32 mapping consumes nearly as much time as a full prefill, BF16 execution combined with source block sharing reduces mapping and selection overhead to a small fraction of total latency.
Prompt length TT1495121,0242,0484,0968,192
fp32 map0.630.670.810.850.810.81
++ bf16 map0.780.951.271.381.361.34
++ block sharing1.161.321.821.951.891.78
++ saturating batch2.362.342.302.232.091.89
Appendix
Table 3: Prefill speedup over the target baseline, at recomputation budget ρ=0.3\rho{=}0.3 . Rows are cumulative, each adding an optimization to the row above. The first three rows are single requests (batch size 1); the last is the largest batch the device holds, which is 384,128,64,32,16384,128,64,32,16 and 88 requests from left to right ( ≈\approx 65k tokens in flight).
Figure 11: (a) Prefill throughput scaling at T=149T{=}149 , where target only prefill saturates near batch size 44 while RaReCache continues to scale. (b) Prefill speedup as a function of batch size across three prompt lengths; the dashed horizontal line indicates the theoretical speedup ceiling.
  Mean Batchp50 TTFTp99 TTFT
λ\lambda (req/s)utargetu_{\text{target}}tgt / ourstgt →\to ours (ms)Ratiotgt →\to ours (ms)Ratio
50.121.1 / 1.251 →\to 950.54 ×\times113 →\to 1820.62 ×\times
100.231.2 / 1.751 →\to 1180.43 ×\times130 →\to 1930.67 ×\times
200.472.1 / 2.891 →\to 1410.65 ×\times200 →\to 2030.99 ×\times
300.704.0 / 4.1154 →\to 1560.99 ×\times374 →\to 2221.68 ×\times
350.827.4 / 4.7274 →\to 1631.68 ×\times640 →\to 2392.67 ×\times
Appendix
Table 4: TTFT under Poisson load ( T=149T{=}149 , ρ=0.3\rho{=}0.3 , max batch size 3232 , wall clock). Offered load is normalized to target saturation capacity ( utarget=λ/μtargetu_{\text{target}}=\lambda/\mu_{\text{target}} ). Speedup ratios represent target ÷\div RaReCache ( >1×>1\times denotes RaReCache is faster).
  Transfer, recompute fraction ρ\rho 
Benchmark ( nn )0.6B alone000.20.20.30.30.40.40.50.514B alone
GSM8K (1319)56.171.6 (75.3)—90.2 (94.8)93.8 (98.6)95.3 † (100.2)95.1
MMLU-Redux (5700)35.047.4 (60.3)54.4 (69.3)61.2 (77.9)67.1 (85.4)72.6 (92.4)78.6
ARC-Challenge (1172)56.781.3 (87.5)89.4 (96.2)90.0 (96.9)91.2 (98.2)—92.9
ARC-Easy (2376)72.387.6 (92.9)91.8 (97.3)93.0 (98.6)93.6 (99.2)—94.3
LongBench-E QA (467)33.525.4 ‡ (38.5)——53.6 (81.3)59.7 (90.7)65.9
Appendix
Table 5: Qwen3-0.6B →\to Qwen3-14B across five benchmarks. Accuracy (%, F1 for LongBench-E) as a function of the fraction ρ\rho of prompt positions recomputed in the target. Retention relative to the 14B is in parentheses. ρ=0\rho{=}0 is plain transfer. † Not significantly different from the 14B ( p=0.86p{=}0.86 ). ‡ Below the source model alone. Each recompute budget improves over ρ=0\rho{=}0 . A dash means the arm was not run.
Figure 12: Layerwise agreement scores with the shared recompute set determined by ℓ2\ell_{2} mapping error and rank disagreement.
Figure 13: What the map gets wrong in one GSM8K prompt, and which tokens each signal would recompute. Columns are prompt tokens from four windows, one per section (position ranges above). Tokens in blue are those rank disagreement recomputes at ρ=0.2\rho{=}0.2 . The two lanes mark the top 20% of positions by rank disagreement and by mapping error. The four rows below are relative mapping error ∥mapped−true∥2/∥true∥2\lVert\mathrm{mapped}-\mathrm{true}\rVert^{2}/\lVert\mathrm{true}\rVert^{2} (keys and values together, averaged over layers), the rank-disagreement score, and the attention each token receives from the 14B and from the 0.6B (last 32 query positions, last quarter of layers). Each row is scaled to its own 95th percentile over the whole prompt; a black tick marks a clipped value, and the <im_start> sink’s share of its row is printed.

Explore similar work

CardsList
Aug 4, 2026cs.LG

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Taekyung Heo, Rasoul Shafipour, Ritchie Zhao +6

NVIDIA

KV CachingLLM Inference AccelerationKV-Cache Reuse
AskPDF
Sep 27, 2026cs.AI

RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse

Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing methods selectively recompute token states to recover these missing interactions, but primarily allocate the recomputation budget to selecting which states to recompute, while fixing the recomputation context to the full causal prefix. We introduce RelaxKV, which formulates selective cache repair as a joint allocation problem over repair targets and recomputation context. Guided by the user query, RelaxKV identifies layer-specific repair targets and restricts their recomputation to a query-relevant context, reducing attention computation. Across four decoder models, RelaxKV at a 15% anchor ratio improves aggregate LongBench performance over ProphetKV on all models. On Qwen3-14B, RelaxKV provides a stronger quality-TTFT trade-off than ProphetKV across a 5%-30% anchor-ratio sweep, and achieves the best selective results on RULER-MV and LV-Eval at 16K and 32K context lengths. Controlled ablations further demonstrate the importance of recomputation context selection.

Ruoling Qi, Yirui Liu, Xuaner Wu +5

Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI) · State University of New York at Buffalo

Retrieval-Augmented GenerationKV-Cache ManagementSparse AttentionKV-Cache Reuse
AskPDF
Aug 31, 2026cs.LG

A Universal Context-Reuse Layer for Cross-Model KV Sharing

Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B →\rightarrow Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B →\rightarrow Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B →\rightarrow Qwen2.5-7B setting, cross-family handoff achieves 44.0% accuracy compared with 45.7% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.

Yi Li, Dongming Jiang, Yi Zhao +1

University of Texas at Dallas

LLM InferenceKV CachingLLM Inference AccelerationKV-Cache Reuse
AskPDF