On-Device Language Model Inference

Overview

The field studies how to run useful language models locally, where limited memory, battery life, heat, and compute can constrain both model size and response speed. Common approaches include quantization and pruning, compact or mixture-of-experts models, KV-cache compression, speculative decoding, and hardware-aware runtimes for mobile CPUs, GPUs, and NPUs. Other systems split work between a device and the cloud or route tasks among local models to balance privacy, cost, and capability. A central tension is that gains in speed or memory do not always preserve accuracy, reliability, or energy efficiency, so evaluation increasingly measures full workloads on real devices rather than relying on model size or peak throughput alone.

Latest papers

CardsList

190 papers23 papers in the last four weeks, down 26% on the four weeks before. 0.2% of all new papers.

  1. Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval

    Feb 18, 2026Subrit DikshitAI LawSmall Language Models

  2. Arapai: An Offline-First LLM Architecture for Adaptive Learning in Low-Connectivity Environments

    Feb 14, 2026Joseph Walusimbi, Ann Move Oguti, Joshua Benjamin Ssentongo +1On-Device Language Model InferenceIntelligent Tutoring Systems

  3. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

    Nov 11, 2025Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12LLM Inference EfficiencyAI Accelerator Inference

  4. OpenPhone: Mobile Agentic Foundation Models

    Oct 24, 2025Yangqin Jiang, Chao HuangOn-Device Language Model InferenceGUI Agents

  5. K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

    Oct 15, 2025Donald Shenaj, Ondrej Bohdal, Taha Ceritli +3Continual Learning for LLMsOn-Device Language Model Inference

  6. DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

    Jul 25, 2025Jiawen Qi, Jiayu Liu, Chang Gao +2On-Device Language Model InferenceLLM Inference Acceleration

  7. A Survey on Collaborative Mechanisms Between Large and Small Language Models

    May 12, 2025Yi Chen, Haiquan Wang, JiaHao Zhao +1Small Language ModelsMulti-LLM Collaboration

  8. LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities

    May 8, 2025Kalyan Nakka, Jimmy Dani, Ausmit Mondal +1LLM GuardrailsLLM Safety

  9. Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

    Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model InferenceLLM Inference

  10. FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

    Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM QuantizationOn-Device Language Model Inference