Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN

Tianhang Ding; Jianchun Liu; Hongli Xu

cs.LGAug 17, 2026

Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN

Authors: Tianhang Ding, Jianchun Liu, Hongli Xu

Organizations: University of Science and Technology of China

Abstract

AI-RAN brings large language model (LLM) serving close to mobile users, but cellular handover can separate an active request from its inference state: the user attaches to a target base station (gNB) while the large and growing key-value (KV) cache remains at the source. Retaining inference at the source preserves service continuity but persistently increases inter-token latency (ITL), whereas recovering the state at the target restores serving locality but requires KV-cache transfer, recomputation, or a combination of both only after handover, directly prolonging service interruption time (SIT). This work presents Pallas, a proactive KV-cache migration framework that prepares the inference state at the predicted target before handover, in parallel with ongoing source-side inference and token delivery. At the preparation trigger, Pallas partitions the token sequence into a stable historical prefix and an evolving suffix. The target reconstructs the prefix through local prefill, while the source streams the KV blocks generated for the suffix. At handover, the target assembles both portions into an up-to-date KV cache and resumes decoding locally, leaving only unfinished preparation to contribute to SIT. An online scheduler selects the prefetching window, which determines how early preparation begins before handover, based on mobility predictions and runtime telemetry. Across three LLMs and $100$ -- $500~\mathrm{Mbps}$ inter-gNB links, our vLLM-based prototype reduces average SIT by factors of $2.28$ -- $89.68$ over target-side recovery approaches and lowers average ITL by $16.0\%$ -- $50.0\%$ compared with source-side forwarding.

Figures & tables

Figure 1. CDF of ITL under the Detour strategy. Increased routing hops lead to severe ITL degradation. The vertical red dashed line at 100 ms marks the SLO.

Strategy	1K	2K	4K
Full-Copy	7165	14310	29066
Recomputation	581	1208	2103
ctHO	554	1119	2058

Table 1. Target-side recovery time (ms) of representative recovery strategies with Qwen3-32B at 300 Mbps inter-gNB goodput.

Figure 2. Prefetching-window trade-off under a representative operating condition (300 Mbps inter-gNB goodput, Qwen3-14B, and a 3,000-token context).

Figure 4. KV-cache recovery timelines of Full-Copy , Recomputation , ctHO , and Pallas during handover.

Figure 5. Single-handover performance of five methods across three models: (a)–(c) SIT under fixed goodputs of 100, 300, and 500 Mbps; (d) ITL across all fixed-goodput settings; (e) SIT under measured goodput traces.

Model	Metric	Detour	Full-Copy	Recomputation	ctHO	Pallas
Llama-3-8B	SIT Avg.	287.7	19645.7	837.3	812.7	153.8
	SIT P50	279.1	18018.7	828.8	812.3	152.8
	SIT P90	481.7	26931.4	1082.8	1040.2	190.9
	SIT P99	652.2	36163.7	1252.2	1230.3	216.3
	ITL Avg.	241.3	25.0	25.1	25.1	25.1
	ITL P90	433.7	26.4	26.4	26.4	26.4

Table 2. Repeated-handover SIT and ITL performance across three models.

Figure 6. Prefetching-window selection across different goodputs, models, and context lengths.

Cell (m)	$T_{\mathrm{avail}}\!\geq\!2$ s (%)	$T_{\mathrm{avail}}\!\geq\!3$ s (%)	$\hat{g}_{\mathrm{tar}}\neq g^{*}_{\mathrm{tar}}$ (%)	$E_{\mathrm{early}}$ P90 (s)	$T_{\mathrm{waste}}$ Mean (s)	Upper Bound Mean (s)
50	51.9	34.6	37.3	4.19	1.46	3.79
100	57.9	42.2	19.3	2.16	1.02	2.93
150	62.8	48.0	11.2	0.94	0.58	2.57

Table 3. Prediction availability and failure overhead.

Figure 7. Performance of Pallas under controlled reductions in

T_{\mathrm{avail}}

Figure 8. Concurrency performance under (a) HH, (b) HI, and (c) joint HH–HI contention.

Figure 9. Ablation study of single-handover SIT across three goodputs and three models.

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 72 papers

Aug 11, 2026

ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings. We cast the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: a sigmoid that fits measurements on the RULER benchmark with

R^2>0.99

across models and context lengths. Because importance ordering front-loads the high-value entries, the concave region of the accuracy curve spans nearly the entire cache. Our proposed allocator keeps served users within this region, making each per-slot allocation problem convex. The optimum is derived via a closed-form weighted water-filling solution that generalizes information-theoretic water-filling and enables online scheduling. The proposed allocator attains over 93.7% average accuracy in a 500ms transfer window, within 0.5pp of the full-cache ceiling, and reaches 98.2-99.5% of a clairvoyant upper bound.

Minwoo Kim, Soochang Song, Namyoon Lee +2

Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea

KV Caching Edge Computing LLM Inference Scheduling KV-Cache Management Online Resource Allocation

Ask PDF

Sep 18, 2026

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications often require long contexts, making the Key-Value (KV) cache a dominant memory bottleneck because it grows linearly with sequence length and is accessed at every decoding step. Prior work reduces KV-cache footprint through low-rank compression, token eviction, or flash offloading, but the resulting reconstruction overhead, irreversible token loss, or I/O stalls can offset the benefit of saving memory. We present TierKV, a mobile LLM inference framework built on Predictive Multi-Tier Cache Optimization (PMCO). Before decoding starts, PMCO predicts future cache demand from prefill hidden states and jointly assigns tokens to exact, low-rank, and flash-offloaded tiers under the device memory and accuracy budgets. This formulation retains access to the full context, removes the circular dependency of reactive eviction, and admits a closed-form solver that selects tier boundaries and per-layer ranks at runtime. Across eight text, vision, and audio models on three mobile SoCs, TierKV improves prefill throughput by up to 17.6x over existing mobile LLM frameworks, reduces RAM-resident KV cache by 12.5-34%, thereby enabling substantially longer contexts under the same memory budget, while incurring only minor accuracy degradation.

Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu +4

University of Georgia Athens, GA, USA · Peking University Beijing, China · Western Digital Research San Jose, CA, USA

On-Device Language Model Inference Long-Context Language Model Inference KV-Cache Management Memory-Efficient Inference

Ask PDF

Jul 9, 2026

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.

Jiantong Jiang, Peiyu Yang, Rui Zhang +1

School of Computing and Information Systems, The University of Melbourne · School of Computer Science and Technology, Huazhong University of Science and Technology (www.ruizhang.info)

LLM Inference Efficiency LLM Serving KV Caching KV-Cache Management

Ask PDF

Abstract

LLM Inference Serving

Explore similar work

ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization