Large Language Model Deployment on Resource-Constrained Edge Devices: A Practitioners Survey

Rahul Maliakkal; Yashasvi Makin; Plawan Rath; Rohit Jain; Ankur Sadhoo

cs.LGNov 6, 2026

Large Language Model Deployment on Resource-Constrained Edge Devices: A Practitioners Survey

Authors: Rahul Maliakkal, Yashasvi Makin, Plawan Rath, Rohit Jain, Ankur Sadhoo

Abstract

Large Language Models (LLMs) have transformed many areas of artificial intelligence (AI), but their substantial resource requirements limit their deployment on mobile and edge devices. Although LLMs have demonstrated impressive capabilities in various applications, the next challenge is deploying these models on resource-limited edge devices. Recent innovations in software optimizations and specialized hardware have made it feasible to deploy LLMs on edge devices. Edge deployment offers benefits including local data processing, cost savings, reduced latency, and personalized responses. This paper reviews the latest techniques that make the deployment of LLMs on edge devices feasible.

Figures & tables

Fig. 1: Edge LLMs combine efficient inference and fine-tuning. These are supported by specialized runtimes and hardware adaptations.

Device Class	$<$ 2GB	2-4GB	4-8GB	$>$ 8GB
Device	ESP32,	Budget	Mid-range	Jetson,
Examples	RPi Zero	phones	phones	High-end
Model	TinyLlama	Phi-2	Mistral	Llama
Examples	1.1B	2.7B	7B	13B
Quantization	INT4/2	INT4	INT4/8	INT8/16
Pruning	70-90%	50-70%	20-50%	0-20%

TABLE I: Edge Device Class Optimization Guidelines

Local LLM Deployment

See all 9 papers

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3LLM Serving Prefill-Decode Disaggregation
Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway
Efficient Reasoning on the Edge
Mar 17, 2026Yelysei Bondarenko, Thomas Hehn, Rob Hesselink +15Efficient Language Model Reasoning On-Device Language Model Inference
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints
May 9, 2026Jiaxiang Geng, Yiyi Lu, Lunyu Zhao +3LLM Evaluation LLM Fine-Tuning
Duke Kunshan University · Flower Labs · University of Cambridge

Jun 2, 2026J/K move · Enter open · S save

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La, Frank Eliassen+2

Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway

LLM Serving Prefill-Decode Disaggregation Edge Computing LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging. Beyond executing the models themselves, practical deployment must address cost efficiency, low latency, and optimal resource utilization. Conventional approaches typically assume that an entire model can be hosted on a single device, which does not hold in many real-world scenarios, particularly in Edge and Fog environments where device resources are constrained. In this paper, we introduce E2LLM, a framework designed to enable efficient LLM deployment in such resource limited settings. Rather than simply partitioning a single model across all available devices, E2LLM replicates the full model across multiple groups of devices (replicas) and applies model parallelism within each replica. Each replica is assigned a specialized role PREFILL or DECODER based on its efficiency in handling input and output tokens. This separation leverages the inherent differences between these two phases of LLM inference. To effectively organize devices, we utilize a Genetic Algorithm to form clusters that maximize system performance. Within each cluster, we apply Dynamic Programming to determine an optimal partitioning strategy that minimizes bottlenecks in model-parallel execution. Experimental results demonstrate that our approach adapts robustly to varying workloads, including scenarios with significant variation in input and output token lengths. Compared to the Splitwise baseline, E2LLM reduces average waiting time by over 50% under high-demand conditions

Abstract

Local LLM Deployment

Explore similar work

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

Efficient Reasoning on the Edge

EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints