RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

Yingsheng Geng; Yuchong Gao; Weihong Wu; Guyue Liu; Jiang Liu

cs.LGFeb 28, 2026

RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

Authors: Yingsheng Geng, Yuchong Gao, Weihong Wu, Guyue Liu, Jiang Liu

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Tsinghua University, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China · Peking University, Beijing, China

Abstract

The increasing complexity of AI tasks has shifted the paradigm from monolithic models toward multi-agent large language model (LLM) systems. However, these collaborative architectures introduce a critical bottleneck: redundant prefill computation for shared content generated by previous agents, which significantly increases KV cache memory usage and time-to-first-token (TTFT). While various KV cache methods have been proposed to mitigate prefill redundancy, they either fail to maintain accuracy on agent-generated outputs or exhibit low reuse rates due to rigid constraints. We present RelayCaching, a training-free inference method that directly reuses decoding phase KV caches from previous agents in subsequent prefill phases. Our key insight is that KV caches for identical content are highly consistent across phases, while prefix-induced deviations are sparse and localized within a limited range of layers and token positions. By selectively recomputing KV caches at these positions, RelayCaching preserves model accuracy with minimal overhead, yielding a superior accuracy-efficiency trade-off over existing methods. Experiments on diverse collaborative LLM tasks spanning mathematical reasoning, general knowledge, and code generation demonstrate that RelayCaching achieves over 80% KV cache reuse, reduces TTFT by up to $4.7\times$ compared to the standard pipeline, all with negligible accuracy degradation.

Figures & tables

Figure 1: Overview of KV cache reuse in multi-agent LLM collaboration. Top: A typical workflow where an Architect generates {spec} , a Developer produces {code} based on it, and a Reviewer evaluates the result, so each agent’s output becomes the next agent’s input, causing cascading redundant prefill. Bottom: Comparison of caching strategies. (a) Prefix Caching fails due to prefix mismatch; (b) Pre-computed Caching cannot handle dynamic content; (c) RelayCaching enables decode-to-prefill KV reuse via selective rectification.

Figure 2: Macro-level KV similarity. Average cosine and norm similarity between Decoding KV caches and Full-Prefill KV caches across layers and tokens on 2WikiMQA, compared against Random KV and Independent KV .

Panel of Figure 3 — Figure 3: Layer-wise structured deviations. (a) Layer-wise similarity between decoding and full-prefill KV caches, highlighting a U-shaped value cosine similarity profile across layers. (b) Relative recovery of value cosine similarity for tokens after the reuse position when decoding KV caches are replaced with their full-prefill counterparts either up to or from a given layer index.

Panel of Figure 4 — Figure 4: Token-wise structured deviations. (a) Token-wise similarity between decoding and full-prefill KV caches, averaged over layers, revealing a sparse set of high-deviation positions. (b) Spearman rank correlation of token-wise value cosine deviations between adjacent layers, showing that high-deviation positions, once emerged, tend to persist.

Figure 5: Overview of RelayCaching. (1) An offline layer-range profiler analyzes calibration data to identify a critical layer range

[L_{\text{start}},L_{\text{end}}]

from the similarity profile and a detection layer

L_{\text{det}}

from the inter-layer correlation profile. (2) During stage-

(i{-}1)

decoding, the cache manager stores pre-RoPE KV caches, saves hidden states at

L_{\text{start}}

, and accumulates influence scores

s_{\text{inf}}(j)

. (3) At stage-

i

prefilling, RelayCaching re-applies RoPE to align reused KV caches to new positions, performs token selection at

L_{\text{det}}

via combined deviation- and influence-based criteria, and finally applies sparse rectification on selected tokens within

[L_{\text{start}},L_{\text{end}}]

Panel of Figure 6 — Figure 6: Computational reuse efficiency across diverse benchmarks. We compare the Reuse Rate of RelayCaching against baselines on (a) GSM8K, (b) MMLU, and (c) HumanEval as the number of agents increases.

Llama-3.1-8B-Instruct (Accuracy %)
Dataset	Method	2 Agents	3 Agents	4 Agents
GSM8K	Full Prefill	85.97	84.69	84.69
	Zero Reuse	40.33	40.03	53.68
	CacheBlend	76.72	77.56	74.73
	EPIC	58.73	66.46	58.42
	KVComm	84.23	82.34	84.91

Table 1: RelayCaching consistently maintains performance comparable to the Full-prefill . Columns indicate the number of agents in the workflow. Bold indicates the best performance; underline marks the second best.

Panel of Figure 7 — Figure 7: Ablation on key RelayCaching hyperparameters. We analyze the trade-off between Accuracy (left axis) and Reuse Rate (right axis) by varying: (a) the influence score factor $\tau_{\text{down}}$ , (b) the deviation score factor $\tau_{\text{diff}}$ , and (c) the suffix length $L_{\text{suf}}$ . Red markers denote the default configuration used in the main experiments.

Method	Agent 2	Agent 3	Agent 4	Agent 5
Full Prefill	85.2	180.2	321.5	493.9
CacheBlend	44.9	76.7	112.5	159.3
Speedup	1.90 $\times$	2.35 $\times$	2.86 $\times$	3.10 $\times$
RelayCaching	40.6	57.0	73.2	104.8
Full recompute	1.6	2.8	4.3	7.9
Index selection	0.5	0.6	0.7	0.8

Table 2: Per-agent TTFT breakdown and speedup. Prefix: 512 tokens; Output: 2,048 tokens; Model: Qwen3-0.6B.

Figure 8: Scalability of TTFT with context growth. We measure the average TTFT of downstream agents as the cumulative output length scales from 512 to 12,288 tokens.

Method	Reuse (%)	Acc (%)
Full Prefill	0.00	85.97
+ Layer Range Profiler	46.88	85.90
+ $\mathbf{I}_{\text{dev}}$	90.37	79.68
+ $\mathbf{I}_{\text{inf-score}}$	85.84	81.65
+ $\mathbf{I}_{\text{inf-suffix}}$ (Ours)	85.35	84.84

Table 3: Ablation study on GSM8K (Llama-3.1-8B-Instruct).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Universality of layer-wise structured deviations. Left Column: Layer-wise cosine similarity and norm similarity profiles. The distinct U-shaped drop in value cosine similarity (red lines) is consistent across diverse models. Right Column: Relative recovery curves. In all cases, recomputing the middle layers (where the similarity drops) yields the most efficient recovery of downstream token fidelity.

Figure 10: Universality of token-wise structured deviations. Left Column: Token-wise similarity profiles averaged over middle layers. The deviations are consistently sparse, with most tokens retaining high fidelity. Right Column: Spearman rank correlation of value cosine deviation between adjacent layers. The high correlation confirms that deviation patterns are persistent and predictable across depths.

Symbol	Description	Default
$\tau_{\text{low}}$	Lower similarity threshold	0.99
$T$	Right-tail layers	5
$\lambda$	Stable multiplier	2.0
$N_{\min}$	Min rise samples	3
$C$	Consecutive stables	2

Appendix

Table 4: Hyperparameters for Critical Layer Selection. These values are used to identify the deviation-prone middle layers based on the U-shaped similarity profile.

Model	$L_{\text{start}}$	$L_{\text{det}}$	$L_{\text{end}}$
Llama-3.1-8B-Instruct	1	3	18
Qwen2.5-Coder-7B-Instruct	3	4	22
Qwen3-0.6B	2	3	19

Appendix

Table 5: Model-specific layer configurations derived from offline profiling.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction
Sep 28, 2026Hyesung Jeon, Hyeongju Ha, Jae-Joon KimMulti-Agent LLM Systems Memory-Efficient Inference
Department of Electrical and Computer Engineering Seoul National University
RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
Oct 8, 2026Sreetama Sarkar, Saptarshi Mitra, Sitao Huang +2KV-Cache Reuse LLM Inference Acceleration
University of Southern California · University of California, Irvine · Intel Labs
LatentRemedy: Efficient Remedy For Non-Prefix KV Cache Reuse in Multi-Agent Systems
Oct 13, 2026Zhiwei Chen, Zhongtao Rao, Waikit Xiu +4KV-Cache Reuse Multi-Agent LLM Systems
The Hong Kong University of Science and Technology (Guangzhou) · Microsoft · The University of Hong Kong

Sep 28, 2026J/K move · Enter open · S save

PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim

Department of Electrical and Computer Engineering Seoul National University

Multi-Agent LLM Systems Memory-Efficient Inference LLM Inference Acceleration KV-Cache Reuse

Open paper AskPDF arXiv

Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present PReCache, a training-free KV cache sharing framework with two designs, namely PreLRShared and ReBaseShared, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a 3.1x TTFT speedup and a 2.3x improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1 points relative to inference without cache sharing.

Abstract

LLM Inference Acceleration

Explore similar work

PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

LatentRemedy: Efficient Remedy For Non-Prefix KV Cache Reuse in Multi-Agent Systems