SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

Jieyun Deng; Keming Hu; Yanghe Liu; Li Feng

cs.CLOct 13, 2026

SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

Authors: Jieyun Deng, Keming Hu, Yanghe Liu, Li Feng

Organizations: Southwest Jiaotong University, Chengdu, China · Wuhan Institute of Technology, Wuhan, China · Southwest Petroleum University, Chengdu, China

Abstract

Running large language models (LLMs) on consumer hardware is bottlenecked by the key-value (KV) cache, which grows with every decoded token and quickly exhausts the memory and bandwidth of a single consumer GPU. We present SparPR (Sparse Partial Recomputation), a runtime KV-cache recovery mechanism that does not modify model weights and requires no offline training. SparPR couples three online decisions: Query-Aware key-block selection scores historical blocks against the current query; dynamic budget control estimates per-block copy and recompute costs to decide how many blocks to retain; and on-demand local recomputation with cost-aware fallback merges retained blocks into continuous spans and switches between sparse recomputation and full recovery according to a runtime cost comparison. Implemented on FlexGen/FlexLLMGen, SparPR is evaluated on a single consumer laptop (RTX 3070 Ti 8GB, Ryzen 9 6900HX, 16GB RAM). It cuts end-to-end latency by 31%, 52%, and 63% at 1k, 2k, and 4k prompt tokens respectively, with 1.4x-2.7x higher throughput, and completes an 8k-token workload in 44 minutes where the FlexGen baseline times out. Under a fixed cache budget, SparPR retains a Needle Accuracy that stays well above StreamingLLM and H2O at every keep ratio, and ablation studies confirm that the three mechanisms act jointly.

Figures & tables

Panel of Figure 1 — Figure 1: The SparPR runtime decision loop: query-aware scoring, dynamic budget control, span merging, and cost-aware path selection.

Prompt	FlexGen lat.	FlexGen thr.	SparPR lat.	SparPR thr.	Lat. reduction
1k	1064	.466	737	.673	31%
2k	2101	.236	1009	.491	52%
4k	4174	.119	1553	.319	63%
8k	timeout	—	2641	.187	—

Table 1: End-to-end latency (s) and throughput (tok/s) of FlexGen and SparPR at increasing prompt length (OPT-6.7B, batch 4, RTX 3070 Ti 8 GB). “timeout” denotes failure to complete within the experiment’s time budget.

Method	Keep ratio	2k	4k	8k
StreamingLLM	50%	.445	.467	.485
	40%	.291	.333	.345
	30%	.153	.197	.208
	20%	.020	.051	.070
	10%	.020	.020	.020
	5%	.020	.020	.020

Table 2: Needle Accuracy at fixed cache keep ratios, at 2k/4k/8k prompt tokens (mean of three runs).

Floor	Avg. keep	Throughput	Needle acc.	Sparse cost (ms)
1.00	100.0%	0.482	0.993	952.33
0.90	90.0%	0.532	0.987	891.67
0.80	80.0%	0.588	0.983	821.33
0.75	75.5%	0.642	0.973	761.67
0.70	72.6%	0.681	0.963	712.33
0.65	69.7%	0.728	0.947	671.67

Table 3: Effect of Minimum Keep Floor on keep ratio, throughput, accuracy, and sparse cost (needle task).

Threshold $\gamma$	Throughput	Needle acc.	Bypass count
0.72	0.520	0.985	12.00
0.74	0.560	0.983	10.00
0.76	0.610	0.980	8.00
0.78	0.650	0.977	6.00
0.80	0.680	0.971	5.00
0.85	0.700	0.962	3.00

Table 4: Effect of Dense Fallback Threshold on throughput, accuracy, and bypass count (needle task).

Method	Latency (s)	Throughput	Needle acc.
FlexGen	991.33	0.50	0.983
Quest-only	845.33	0.59	0.777
w/o Budget	838.67	0.60	0.963
w/o Span Merge	936.67	0.53	0.973
w/o Bypass	812.33	0.62	0.917
SparPR-full	756.67	0.66	0.957

Table 5: Ablation: external results (needle task, 1000 tokens, mean of three runs).

Method	Avg. keep	Bypass	Blocks skip	Recompute ok	Sparse cost (ms)
FlexGen	100.0%	0.00	0.00	0.00	952.33
Quest-only	58.7%	0.00	726.67	3.00	805.67
w/o Budget	85.0%	3.33	266.33	4.33	818.33
w/o Span Merge	73.5%	16.33	468.33	8.67	1085.33
w/o Bypass	73.8%	0.00	462.67	12.33	821.67
SparPR-full	73.2%	5.67	473.33	9.33	728.33

Table 6: Ablation: runtime statistics (needle task, 1000 tokens, mean of three runs).

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,090 papers

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
Apr 23, 2026Hongyao Liu, Liuqun Zhai, Junyi Wang +3On-Device Language Model Inference KV-Cache Offloading
Department of Computer Science, City University of Hong Kong, Hong Kong SAR.
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
S $^4$ R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Aug 1, 2026Jialong Han, You Wu, Kewei TuKV Caching Long-Context Language Model Inference
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging

Apr 23, 2026J/K move · Enter open · S save

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

Hongyao Liu, Liuqun Zhai, Junyi Wang, Zhengru Fang+2

Department of Computer Science, City University of Hong Kong, Hong Kong SAR.

On-Device Language Model Inference KV-Cache Offloading Edge-Cloud Collaborative Inference KV-Cache Management LLM Inference Acceleration

Open paper AskPDF arXiv

Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment.

Abstract

LLM Inference Acceleration

Explore similar work

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

S4^44R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching

S $^4$ R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching