Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.
Figures & tables
Figure 1. System-level profile during LLM inference on Jetson AGX Orin. Four time-series panels comparing llama.cpp and NeuroPrefetcher during decoding, showing major page faults per second, CPU iowait percentage, GPU utilization percentage, and NVMe bandwidth.
Figure 2. High-level NeuroPrefetcher architecture with predictor-guided weight prefetching and sparse MLP execution. Block diagram showing the GPU execution path with a dense prefix and sparse suffix, the NeuroPrefetcher runtime with predictor, centroid bank, delta runtime, and I/O path, and the NVMe storage tier supplying delta rows.
Figure 3. One-shot shared predictor for gate and up centroid IDs. Diagram of the shared predictor. Layer 0 features feed four projection blocks and a centroid embedding, a deep residual trunk combines them, and separate gate and up heads emit one centroid ID per downstream layer.
Figure 4. Offline sparse centroid preparation.
Feature
Input form
Token embedding
4096-d vector
Pre-attention LayerNorm
4096-d vector
Attention-head sketch
128-d vector
Post-attention RMS norm
1 scalar
Gate projection structure
centroid ID
Table 1. Layer 0 features consumed by the shared predictor.
Component
Params
Size (MB)
Input projections
4.3M
8.6
Centroid embedding
5.2M
10.4
Residual trunk
67.1M
134.2
Output heads
130.2M
260.4
Total
206.8M
413.6
Table 2. Predictor parameter breakdown. Sizes are FP16.
Figure 5. Token-to-token delta update for one sparse MLP layer.
Kernel mem=
CUDA total (GiB)
CUDA free (GiB)
11G
10.69
9.02
12G
11.67
9.99
13G
12.65
11.02
14G
13.64
11.94
15G
14.62
13.19
16G
15.60
13.88
Table 3. Memory levels and measured CUDA availability.
LLM in a Flash, LMDeploy, TGI, HF Optimum + ONNX GPU
Table 4. Capability results at a 14 GiB memory cap with 11.9 GiB CUDA-available memory.
Figure 6. Throughput versus dense prefix length at 11.9 GiB CUDA-available memory. Dashed lines mark configurations whose sparse suffix falls back to whole-layer NVMe reads. Circled markers denote the peak d∗ . Line chart of decoding throughput against dense prefix length for three sparsity levels and an mmap fallback, showing a rising regime, a peak, a cliff of more than three times, and a falling regime.
Figure 7. Throughput comparison across memory levels. (a) In the model-exceeds-memory range. (b) Near and after the memory crossover, llama.cpp page-fault overhead drops and the speedup narrows. The dashed line marks the 13.49 GiB base model footprint.
Figure 8. Delta reuse and NVMe read volume at 62% sparsity. (a) Sparse suffix rows reused from the previous token versus incoming rows that require I/O. (b) NVMe read volume per token on a log scale. The dashed line marks the full 13.49 GiB model footprint.
Figure 9. Per-token latency breakdown at 62% sparsity.
Figure 10. Model quality across dense/sparse configurations for Mistral-7B and Llama-3-8B. (a) WikiText-2 perplexity, where lower values are better. (b) HellaSwag accuracy retention relative to each model’s dense baseline.
On-device LLM inference is increasingly important for edge platforms but remains challenging due to high computational and memory demands and tight resource constraints. Modern applications often involve long prompts. Although compact models reduce linear-layer costs through weight compression, attention remains largely unoptimized and becomes the dominant bottleneck, especially during prefilling. Existing attention pruning methods primarily target highly parallel hardware (e.g., GPUs) and long context windows, limiting their suitability for edge scenarios. We present DeltaLLM, a training-free LLM framework based on algorithm-hardware-dataflow co-design to accelerate prefilling attention. DeltaLLM introduces a causal-attention-aware K-only delta formulation that enables Delta-Key reuse across query heads under Grouped-Query Attention (GQA). A context-aware hybrid strategy preserves exact attention near the diagonal while applying delta approximation to nonlocal regions. For score-value multiplication, we propose a vectorized structured sparse online softmax (VSSOS) that converts fine-grained spatial score sparsity into hardware-friendly column-level skipping. A specialized vector-processor architecture and unified sparse/dense attention dataflow exploit the resulting sparsity. Evaluated on BitNet-b1.58-2B-4T and LLaMA3.2-1B-Instruct, DeltaLLM achieves up to 60% Delta-Key sparsity and 63% structured attention-score sparsity with negligible average accuracy loss. On the KV260 FPGA board, it delivers a 1.62x speedup over TeLLMe-v2 while reducing DSP and BRAM usage. These results show that DeltaLLM supports efficient edge LLM deployment and integration with existing pipelines without fine-tuning.
Jiawen Qi, Jiayu Liu, Chang Gao +2
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2300 RA Leiden, The Netherlands · Department of Microelectronics, Delft University of Technology, 2628 CD Delft, The Netherlands
Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices. Existing approaches to model compression and optimization often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectInfer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1
Department of Computer Networks, Paderborn University, Paderborn, Germany
On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8× to 7.5× at the best throughput achieved by prior work, while further improving token generation throughput by 1.6× to 2.1×.
Renyuan Liu, Yuyang Leng, Kaiyan Liu +7
Richard · Global Technology Applied Research, JPMorganChase · University of Kansas +1