Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

Hiroki Matsutani; Naoki Matsuda; Naoto Sugiura

cs.LGFeb 26, 2026

Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

Authors: Hiroki Matsutani, Naoki Matsuda, Naoto Sugiura

Organizations: Dept. of ICS, Keio University, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, Japan

Abstract

Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.

Figures & tables

Figure 1: System overview. A local LLM on Client 1 (left) first processes a prompt and uploads its internal states to Cache Server (middle), accelerating a subsequent query on Client 2 (right).

Figure 2: Data structures. The internal states and the master catalog are stored in the remote server. Local inference on the clients utilizes the cached states when available. Each local catalog is synchronized with the master on the server.

Figure 3: Example of catalog . Four distinct ranges of a prompt consisting of the instruction, few-shot examples, and target question are registered in the catalog .

	Server	Client (high-end)	Client (low-end)
Platform	Raspberry Pi 5 Model B		Zero 2W
CPU	ARM Cortex-A76 2.4GHz		Cortex-A53 1GHz
DRAM	16GB	4GB	512MB
OS	Raspberry Pi OS 13 (Trixie), 64-bit

Table 1: Specifications of server and client devices.

	TTFT			TTLT
Case	1	5	[%]	1	5	[%]
Low-end	12.59	0.87	6.88	23.74	11.86	49.93
High-end	2.70	2.89	107.08	2.77	2.97	107.10

Table 2: TTFT and TTLT [sec] on low-end and high-end settings under Case 1 (cache miss) and Case 5 (full hit).

Figure 4: Performance comparison between cache miss and hit cases (based on Table 2 ).

	Token	Bloom	P-decode	Redis	R-decode	Sample	$N$	# tokens	State size [MB]
Low-end (Case 1)	3.46	0.30	12580.85	2.42 $\dagger$	11061.04	95.69	1	65.27	2.25
Low-end (Case 5)	3.46	0.19	0.00	861.92	10904.67	84.82	1	65.27	2.25
High-end (Case 1)	1.61	0.00	2688.17	7.84 $\dagger$	72.59	1.45	5	334.11	9.94
High-end (Case 5)	1.56	0.00	0.00	2887.04	78.12	1.67	5	334.11	9.94
$\dagger$ Induced by rate false positives.

Table 3: Latency breakdowns [msec] on low-end and high-end settings under Case 1 (cache miss) and Case 5 (full hit).

	# matched	% matched	T-decode
Low-end (Case 1)	1	0.25	27203.96
Low-end (Case 2)	10	2.47	26288.23
Low-end (Case 3)	57	14.07	24590.09
Low-end (Case 4)	340	83.95	13344.96
Low-end (Case 5)	405	100.00	11220.95
High-end (Case 1)	1	0.25	3361.88

Table 4: Total decoding time [msec] on low-end and high-end settings under partial matching cases.

Figure 5: Performance comparison between partial matching cases on low-end setting (based on Table 4 ).

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
Jul 14, 2026Yi Li, Chen Li, Jiexiong LiuOn-Device Language Model Inference Privacy-Preserving Language Models
KunlunMeta
DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Jul 25, 2025Jiawen Qi, Jiayu Liu, Chang Gao +2On-Device Language Model Inference LLM Inference Acceleration
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2300 RA Leiden, The Netherlands · Department of Microelectronics, Delft University of Technology, 2628 CD Delft, The Netherlands
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1

Jul 14, 2026J/K move · Enter open · S save

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

Yi Li, Chen Li, Jiexiong Liu

KunlunMeta

On-Device Language Model Inference Privacy-Preserving Language Models LLM Inference Edge-Cloud Collaborative Inference Efficient Language Model Inference

Open paper AskPDF arXiv

On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1% and downlink payloads by up to 67.4% over baseline split inference, retaining comparable performance to full cloud inference.

Abstract

LLM Inference Acceleration

Explore similar work

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters