ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference

Hung-Chun Lin; Ting-Wei Hsu; Chung-En Ho; Ahmed Saeed

cs.NIOct 30, 2026

ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference

Authors: Hung-Chun Lin, Ting-Wei Hsu, Chung-En Ho, Ahmed Saeed

Organizations: Georgia Institute of Technology, Atlanta, GA, USA

Abstract

Disaggregated LLM inference separates the prefill and decode phases across GPU pools, generating massive KV-cache transfers. Because these transfers last hundreds of milliseconds to seconds, they behave as mega elephant flows that dominate link bandwidth utilization. In this regime, stateless ECMP can perform poorly: hash collisions may overload one spine link while leaving others idle, stretching transfer times by seconds. Yet this same persistence makes coordination practical. Since these flows are long-lived, even lightweight one-to-all coordination can be amortized over their lifetime. We present ARK, a distributed elephant-flow path reservation mechanism. ARK coordinates senders to choose source ports whose hashes map concurrent flows onto distinct spines, without requiring switch changes or receiver-side packet reordering. Packet-level RDMA simulations show that ARK reduces mean and P95 FCT by up to 27.3% and 34.0% under moderate load, and further reduces mean TTFT by up to 12.9%.

Figures & tables

Figure 1. (a) Network topology (b) ARK path reservation controller (c) ARK deployment with separated control and data planes

Panel of Figure 2 — Figure 2. Mean and P95 flow completion time (FCT) as a function of request rate (QPS) under Poisson arrival. ARK consistently reduces both mean and tail FCT, with the largest improvement at QPS = 25 (mean FCT $-$ 27.3%, P95 $-$ 34.0%). Two line plots overlaid on the same axes. The y-axis shows FCT in milliseconds (mean and P95), and the x-axis shows request rate from 1 to 30 QPS. Both metrics rise with QPS for ECMP and \system; the \systemcurves consistently sit below the ECMP curves, with the gap widening as QPS increases.

Figure 3. Per-spine active-flow heatmap at QPS = 25 (top: ECMP, bottom: ARK). ECMP drives persistent hotspots of up to 18 concurrent flows onto Spines 2 and 3 while Spines 1 and 4 stay lightly loaded; ARK keeps peaks below ${\sim}$ 8 across all four spines.

Panel of Figure 4 — Figure 4. TTFT across the same QPS sweep as Fig. 2 . Two line plots sharing a common QPS axis from 1 to 30. The left panel shows mean TTFT and the right panel shows P95 TTFT, both rising with load. The ARK curve sits below ECMP at every QPS, with the gap widening from a few percent at low load to roughly twelve percent at QPS twenty-five for the mean and twelve percent for P95 at QPS thirty.

Microbenchmarks (per call)
Phase / Operation	Time
Hash computation	4.10 $\mu$ s
Reservation-table lookup	0.87 $\mu$ s
Boot phase (one-time per peer)
Initialize ( $\delta_{\max}=2^{16}$ )	$\sim$ 326 ms
Runtime per flow (worst case)

Table 1. ARK’s controller microbenchmarks on an Intel i9 CPU (single thread, single core), and a reference KV transmission time.

KV-Cache Transfer

See all 21 papers

ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover
Aug 11, 2026Minwoo Kim, Soochang Song, Namyoon Lee +2KV-Cache Transfer KV Caching
Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Jun 2, 2026Mubarak Adetunji OjewaleLLM Inference KV Caching
Cloud Competency Centre, National College of Ireland
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM Serving LLM Inference Scheduling
Meta Platforms

Aug 11, 2026J/K move · Enter open · S save

ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

Minwoo Kim, Soochang Song, Namyoon Lee, Bang Chul Jung+1

Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea

KV-Cache Transfer KV Caching Edge Computing LLM Inference Scheduling KV-Cache Management Online Resource Allocation

Open paper AskPDF arXiv

Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings. We cast the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: a sigmoid that fits measurements on the RULER benchmark with

R^2>0.99

across models and context lengths. Because importance ordering front-loads the high-value entries, the concave region of the accuracy curve spans nearly the entire cache. Our proposed allocator keeps served users within this region, making each per-slot allocation problem convex. The optimum is derived via a closed-form weighted water-filling solution that generalizes information-theoretic water-filling and enables online scheduling. The proposed allocator attains over 93.7% average accuracy in a 500ms transfer window, within 0.5pp of the full-cache ceiling, and reaches 98.2-99.5% of a clairvoyant upper bound.

Abstract

KV-Cache Transfer

Explore similar work

ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference

Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving