Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving

Jiantong Jiang; Yue Yang; Peiyu Yang; Feng Liu

cs.LGSep 27, 2026

Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving

Authors: Jiantong Jiang, Yue Yang, Peiyu Yang, Feng Liu

Organizations: University of Melbourne · Maincode

Abstract

Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that target fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity management that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0 $\times$ lower time-to-first-token (TTFT) and 9.1 $\times$ lower P90 TTFT than vLLM while preserving generation quality.

Figures & tables

Figure 1 : Comparison of target-only execution contract and mixed-fidelity execution contract.

Figure 2 : Overview of ElasticKV runtime architecture across control, KV memory, and GPU execution planes.

Panel of Figure 3 — Figure 3 : Pair-structured KV layout for reclaimable BASE capacity.

Setting	Llama-3.1-8B	Qwen3-8B
Static FP16	49.89	49.59
Static FP8	48.72	49.07
ElasticKV (5% cap)	50.06	49.59
ElasticKV (10% cap)	50.06	49.52
ElasticKV (20% cap)	50.04	49.42
All-BASE	48.45	47.87

Table 1: Average LongBench scores over 21 tasks. ElasticKV uses BASE-exposure caps. All-BASE is a 100% BASE stress endpoint. Higher is better.

Figure 4 : Serving performance under increasing concurrency for (a) Llama-3.1-8B-Instruct and (b) Qwen3-8B. Columns show mean/P90 TTFT, mean/P90 TPOT, and output-token throughput. Lower is better for latency; higher is better for throughput. First preemption points are also reported.

Figure 5 : P90 TTFT on ShareGPT under increasing request rates for (a) Llama-3.1-8B-Instruct and (b) Qwen3-8B. Lower is better.

Figure 6 : Mooncake production trace replay with

d=16

. (a) Eff@300 across three trace windows, higher is better; (b) native request preemptions, lower is better; (c) Window C runtime behavior showing offered input load, waiting requests, and ElasticKV’s resident BASE ratio over time.

Figure 7 : Production-trace replay with Llama-3.1-70B-Instruct on 8

\times

AMD MI355X (TP2

\times

DP4). (a) Eff@300, higher is better; (b) native request preemptions, lower is better; and (c) drain time, lower is better.

Figure 8 : Mean TTFT under varying GPU KV budgets (a, b) and input context lengths (c, d) for Llama-3.1-8B-Instruct (a, c) and Qwen3-8B (b, d). Lower is better. “B” denotes ElasticKV’s BASE exposure (%).

Activity	Cost (ms/step)
Online control	0.32
Sync. demotion	2.82
Elastic metadata	1.23

Table 2: ElasticKV-specific cost under high pressure.

Variant	P90 TTFT	P90 TPOT	BASE
ElasticKV	2.8s	40.3ms	14.7%
Watermark demotion	3.4s	46.5ms	17.2%
Eager promotion	4.4s	52.0ms	15.7%
No promotion	2.6s	40.5ms	28.8%
General elastic path	3.1s	49.5ms	14.7%

Table 3: Ablation under high pressure. Lower is better for P90 TTFT and P90 TPOT. BASE denotes exposure.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Single-Document QA
		Reference settings			ElasticKV exposure cap
Task	Metric	FP16	FP8	All-BASE	$\leq 5\%$		$\leq 10\%$		$\leq 20\%$
					Score	Exp.	Score	Exp.	Score	Exp.
NarrativeQA	QA F1	27.85	28.34	27.61	28.87	5.3%	28.95	6.8%	28.95	6.8%
Qasper	QA F1	44.76	43.81	44.88	44.83	5.2%	44.69	8.6%	44.61	20.5%
MultiFieldQA-en	QA F1	55.62	53.38	55.48	56.01	3.7%	56.09	10.5%	56.08	20.5%

Appendix

Table 4 : Per-task LongBench generation-quality results for Llama-3.1-8B-Instruct. For each ElasticKV exposure cap, we report the official task score and the observed BASE exposure ( Exp. ) of the selected run. Higher scores are better for all task metrics.

Single-Document QA
		Reference settings			ElasticKV exposure cap
Task	Metric	FP16	FP8	All-BASE	$\leq 5\%$		$\leq 10\%$		$\leq 20\%$
					Score	Exp.	Score	Exp.	Score	Exp.
NarrativeQA	QA F1	26.32	24.39	24.73	26.88	4.8%	26.64	6.7%	26.64	6.7%
Qasper	QA F1	47.70	45.87	45.08	47.36	5.5%	47.03	10.3%	47.4	18.0%
MultiFieldQA-en	QA F1	53.34	54.34	51.33	52.83	3.9%	52.93	10.5%	52.56	19.6%

Appendix

Table 5 : Per-task LongBench generation-quality results for Qwen3-8B. For each ElasticKV exposure cap, we report the official task score and the observed BASE exposure ( Exp. ) of the selected run. Higher scores are better for all task metrics.

Figure 9 : Serving performance on ShareGPT under increasing request rates for (a) Llama-3.1-8B-Instruct and (b) Qwen3-8B. Columns show mean TTFT, mean/P90 TPOT, and output-token throughput. Lower is better for latency; higher is better for throughput. First preemptions are also reported.

Model	System	Preemptions	BASE exposure
Llama-3.1-8B-Instruct	Full-GPU	1297	–
	Exact-offload	1392	–
	ElasticKV	0	2.6%
Qwen3-8B	Full-GPU	1574	–
	Exact-offload	1594	–
	ElasticKV	0	3.6%

Appendix

Table 6: Runtime behavior at the highest ShareGPT request rate (i.e., 22 requests/s).

Window	Region	Trace interval	Requests	Input tokens	Output tokens
A	Early	[450,750)	920	11.82M	0.324M
B	Middle	[1650,1950)	1027	11.54M	0.357M
C	Late	[2850,3150)	1122	13.37M	0.358M

Appendix

Table 7: Mooncake conversation-trace windows used for the single-A100 replay. Token counts are totals over the original 300s trace intervals before arrival-time dilation.

Figure 10 : Mooncake production trace replay with

d=20

. (a) Eff@300 across three trace windows. (b) Number of native request preemptions. (c) Case study of Window C showing offered input load, waiting requests, and ElasticKV’s resident BASE ratio over time.

Window	Region	Trace interval (s)	Requests	Input tokens	Output tokens
A	Early	[300,900)	1810	24.34M	0.627M
B	Middle	[1500,2100)	2093	23.45M	0.731M
C	Late	[2700,3300)	2207	25.48M	0.720M

Appendix

Table 8 : Mooncake production-trace windows used for the large-scale MI355X evaluation. The three 600s windows sample the early, middle, and late portions of the trace.

Window	System	Eff@300	Mean TTFT	P90 TTFT	Preempt.	Drain	BASE exp. (%)
A	Full-GPU	27.79%	272.49s	924.63s	649	1454s	–
	Exact-offload	25.47%	84.42s	251.64s	450	665s	–
	ElasticKV	38.12%	40.16s	118.63s	0	299s	4.84
B	Full-GPU	36.69%	96.79s	356.03s	484	477s	–
	Exact-offload	35.88%	24.89s	76.60s	116	403s	–
	ElasticKV	55.90%	12.89s	39.73s	0	109s	0.01

Appendix

Table 9 : Detailed results for the TP2

\times

DP4 Mooncake replay on 8

\times

AMD MI355X GPUs.

Figure 11 : Production-trace replay on 8

\times

AMD MI355X with Llama-3.1-70B-Instruct (TP1

\times

DP8).

Model	Setting	Executable KV states	Backend	P90 TTFT (s)	Out. tok/s
Llama-3.1-8B	Static FP8	TARGET FP8	FlashInfer	1.47	1003.12
Llama-3.1-8B	ElasticKV	TARGET 16 + BASE 8	Elastic Triton	2.87	914.25
Qwen3-8B	Static FP8	TARGET FP8	FlashInfer	1.50	949.04
Qwen3-8B	ElasticKV	TARGET 16 + BASE 8	Elastic Triton	1.98	854.92

Appendix

Table 10: Serving performance comparison with static FP8. On our A100 setup, vLLM serves FP8 KV with FlashInfer, while ElasticKV uses our Triton-based elastic backend. The results are therefore not a backend-matched kernel comparison.

Online control
Activity	Trigger (%)	Cost (ms/step)
Shortage probe	78.67	0.072
Demotion planning	11.06	0.175
Promotion planning	100.00	0.023
Restore-source prep.	100.00	0.021
Async promotion issue	13.63	0.029

Appendix

Table 11: Detailed breakdown of ElasticKV-specific critical-path work in the profiled representative high-pressure run. Cost is normalized by all steps.

Statistic	Value
Scheduling steps	16586
BASE exposure	14.71%
Successful demotion batches	1835
Demoted pairs	29427
Promotion issues	2260
Promoted pairs	25278

Appendix

Table 12: Transition statistics for the profiled high-pressure run.

Metric	Profile off	Profile on
Duration (s)	673.01	673.08
Mean TPOT (ms)	39.39	39.38
Output throughput (tok/s)	912.92	912.82
Native preemptions	0	0

Appendix

Table 13: Serving performance with runtime-overhead profiling disabled and enabled.

Explore similar work

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
May 13, 2026Zedong Liu, Xinyang Ma, Dejun Luo +9Key-Value Cache Compression Large Language Model Serving
University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Shanghai Jiao Tong University
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
Sep 7, 2026Michael Wang, Keith Li, Roozbeh BostandoostKey-Value Cache Compression LLM Inference Optimization
Lake Washington School District · XSchool · UMass Amherst
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
Jul 9, 2026Jiantong Jiang, Peiyu Yang, Rui Zhang +1Kv-Cache Management Large Language Model Serving
School of Computing and Information Systems, The University of Melbourne · School of Computer Science and Technology, Huazhong University of Science and Technology (www.ruizhang.info)

May 13, 2026, cs.DCJ/K move · Enter open · S save

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

Zedong Liu, Xinyang Ma, Dejun Luo, Hairui Zhao+8

University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Shanghai Jiao Tong University

Key-Value Cache Compression Large Language Model Serving Large Language Model Compression Data Compression Methods Kubernetes Bottlenecks

Open paper AskPDF arXiv

LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present \emph{KVServe}, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing

50\times

offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to

9.13\times

JCT speedup in PD-separated serving and up to

32.8\times

TTFT reduction in KV-disaggregated serving.

Abstract

Explore similar work

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization