Multi-agent large language model (LLM) systems are increasingly deployed at the edge to support complex tasks that require collaboration and information exchange among multiple agents. However, when task contexts are passed across agents, shared context that has already been processed by upstream agents often needs to be prefilled again, resulting in substantial redundant computation. Reusing an already prefilled key--value (KV) cache can avoid this recomputation, but requires the reusable KV state to be transferred across the network before the receiving agent can begin generation. Under strict time-to-first-token (TTFT) deadlines and unreliable edge links, delivering the complete KV cache may be infeasible, making it necessary to prioritize and protect the most valuable KV information within a limited transmission budget. This paper proposes PriorityFEC, a utility-aware KV-cache handoff scheme that jointly exploits model-side KV importance and transport-side reliability. PriorityFEC first constructs a utility-aware source stream by placing high-utility KV blocks at earlier transmission positions. It then protects the ordered stream using a feedback-adaptive Streaming Forward Erasure Correction (FEC) mechanism, which schedules repair packets according to receiver feedback before the TTFT deadline. We implement PriorityFEC on a distributed edge LLM serving prototype and evaluate it with Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct across multiple datasets. Results show that PriorityFEC achieves superior task quality over the compared transport schemes, with its advantages becoming particularly pronounced under stringent deadline constraints. These results demonstrate the importance of jointly optimizing KV-cache utility and deadline-aware transport reliability when reusable LLM state must be handed off across unreliable edge networks.
Figures & tables
Fig. 1: Motivation for deadline-constrained KV-cache handoff. (a) Reliable transport waits for loss recovery before the receiving agent can use the complete reusable KV state, delaying agent activation and TTFT. (b) PriorityFEC places high-utility KV blocks earlier in the source stream and uses FEC, allowing useful KV state to be reconstructed before the TTFT deadline.
Work
Target object
Primary mechanism
Loss-aware
Model changes
Objective
DistServe [ 9 ]
Serving phases
Resource placement
–
–
TTFT/TPOT SLOs
Splitwise [ 10 ]
Phase-split state
Phase placement
–
–
Service latency
Mooncake [ 11 ]
KV cache
Disaggregated KV store
–
–
Serving SLOs
vLLM [ 12 ]
GPU KV cache
Memory management
–
–
Runtime efficiency
LMCache [ 4 ]
KV cache
Offloading and reuse
–
–
Transfer efficiency
KVDirect [ 3 ]
KV cache
Tensor-aware transfer
–
–
Transfer latency
TABLE I: Positioning of PriorityFEC relative to representative related work.
Fig. 2: Characterization of non-uniform KV-cache utility. (a) Normalized sensitivity across layer groups. (b) Normalized sensitivity across relative token-position bins. (c) Teacher-forced top-1 agreement under partial KV retention on held-out prompts. Profile-ranked retention consistently outperforms random retention; error bars denote 95% confidence intervals.
Fig. 4: Double-DQN training framework for adaptive repair scheduling. The online network evaluates the executed action and selects the feasible next action, while the target network evaluates this selected action to construct the DDQN target.
Fig. 5: Normalized KV utility recovered before the decode-start deadline on the 100-Mbps burst-loss GE link. PriorityFEC consistently preserves the highest-utility KV regions, with its advantage being most pronounced under the tight ρ=0.8 deadline.
Fig. 6: End-to-end token-level F1 under different decode-start deadlines on the 100-Mbps burst-loss GE link. PriorityFEC provides substantial gains under ρ=0.8 , where complete loss-free KV serialization is infeasible. As the deadline is relaxed to ρ=1.0 , the performance gap decreases because more KV state can be recovered regardless of source position.
Fig. 7: Ablation of utility-aware source ordering and adaptive Streaming FEC on Qwen2.5-7B-Instruct under the 100-Mbps heavy-GE channel. The first row reports downstream token-level F1, while the second row reports normalized KV utility recovered before the decode-start deadline. PriorityFEC consistently achieves the best performance by combining favorable source ordering with adaptive repair scheduling.
Model
ρ
State/FB
RL
Enc.
Dec.
Total
Control Mean/P99
Overrun
(ms per 1k packets)
( μ s)
(%)
Qwen2.5-7B
0.8
9.42
3.41
10.31
7.19
30.34
75.98 / 94.35
0.009
Qwen2.5-7B
1.0
9.28
3.36
11.14
8.76
32.53
76.99 / 95.67
0.011
Llama-3.1-8B
0.8
9.38
3.43
10.32
7.29
30.41
76.53 / 95.18
0.019
Llama-3.1-8B
1.0
9.28
3.33
11.34
9.46
33.41
78.00 / 97.58
0.016
TABLE II: Measured online computation overhead of PriorityFEC. Component-wise overhead is normalized by the number of transmitted packets. The control latency measures the online decision path from transport-state construction to scheduling output.
Fig. 8: Token-level F1 under measured 5G SA uplink loss replay. Bars average eight examples and two paired replay windows per example; the dashed line is the clean Full-KV reference.
Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings. We cast the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: a sigmoid that fits measurements on the RULER benchmark with R2>0.99 across models and context lengths. Because importance ordering front-loads the high-value entries, the concave region of the accuracy curve spans nearly the entire cache. Our proposed allocator keeps served users within this region, making each per-slot allocation problem convex. The optimum is derived via a closed-form weighted water-filling solution that generalizes information-theoretic water-filling and enables online scheduling. The proposed allocator attains over 93.7% average accuracy in a 500ms transfer window, within 0.5pp of the full-cache ceiling, and reaches 98.2-99.5% of a clairvoyant upper bound.
Minwoo Kim, Soochang Song, Namyoon Lee +2
Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea
Multi-agent LLM systems on edge devices need to hand off latent context efficiently, but the practical choices today are expensive re-prefill or full-precision KV transfer. We study QKVShare, a framework for quantized KV-cache handoff between agents that combines token-level mixed-precision allocation, a self-contained CacheCard representation, and a HuggingFace-compatible cache injection path. Our current results support a narrower but clearer story than the original draft: on 150 GSM8K problems with Llama-3.1-8B-Instruct, adaptive quantization remains competitive under repeated handoff and shows its clearest gains against uniform quantization in deeper-hop, higher budget settings; for handoff latency, the QKVShare path reduces TTFT relative to full re prefill at every tested context, from 130.7 ms vs. 150.2 ms at nominal 1K context to 397.1 ms vs. 1029.7 ms at nominal 8K context;. Stage timing shows that post-injection generation, not card creation, dominates the current QKVShare latency path. These results position quantized KV handoff as a promising on-device systems direction while also highlighting the need for stronger controller ablations and apples-to-apples runtime comparisons.
Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory