On-Device Language Model Inference

Ask about this topic

Overview

The field studies how to run useful language models locally, where limited memory, battery life, heat, and compute can constrain both model size and response speed. Common approaches include quantization and pruning, compact or mixture-of-experts models, KV-cache compression, speculative decoding, and hardware-aware runtimes for mobile CPUs, GPUs, and NPUs. Other systems split work between a device and the cloud or route tasks among local models to balance privacy, cost, and capability. A central tension is that gains in speed or memory do not always preserve accuracy, reliability, or energy efficiency, so evaluation increasingly measures full workloads on real devices rather than relying on model size or peak throughput alone.

Latest papers

CardsList

170 papers26 papers in the last four weeks, down 10% on the four weeks before. 0.2% of all new papers.

  1. OpenJarvis: Personal AI, On Personal Devices

    May 16, 2026Jon Saad-Falcon, Avanika Narayan, Robby Manihani +10On-Device Language Model InferenceLLM Agents

    Stanford University · Lambda Labs

  2. Lever: Speculative LLM Inference on Smartphones

    May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model InferenceLLM Inference

    Tsinghua University · Beijing, China · Beihang University +1

  3. CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference

    May 12, 2026Nan Xue, Shengkang Chen, Zhiyong Chen +4On-Device Language Model InferenceLLM Inference

    Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China · Department of Broadband Communication, Pengcheng Laboratory, Shenzhen 518000, China · Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen), Shenzhen 518172, China +1

  4. CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration

    May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1On-Device Language Model InferenceMemory-Efficient Inference

    University of Florida Gainesville, FL 32611

  5. Agent-X: Full Pipeline Acceleration of On-device AI Agents

    May 11, 2026Jinha Chung, Byeongjun Shin, Jiin Kim +1LLM Inference EfficiencyOn-Device Language Model Inference

    KAIST Republic of Korea, Daejeon

  6. GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference

    May 11, 2026Zengzipeng Tang, Yuxuan Sun, Wei Chen +2On-Device Language Model InferenceEdge Computing

    School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China

  7. Large Language Models over Networks: Collaborative Intelligence under Resource Constraints

    May 9, 2026Liangqi Yuan, Wenzhi Fang, Shiqiang Wang +2LLM Inference EfficiencyOn-Device Language Model Inference

    Purdue University, IN, USA. · University of Exeter, UK. · Princeton University, USA.

  8. Benchmarking Local Language Models for Social Robots using Edge Devices

    May 4, 2026Dorian Lamouille, Matevž B. Zorec, Farnaz Baksh +1LLM EvaluationOn-Device Language Model Inference

    University of Tartu Tartu, Estonia · Institute of Computer Science University of Tartu Tartu, Estonia · Institute of Technology University of Tartu Tartu, Estonia

  9. Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI

    May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM PruningOn-Device Language Model Inference

    Meta

  10. AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devices

    May 1, 2026Dzung Pham, Kleomenis Katevas, Ali Shahin Shamsabadi +1LLM Inference EfficiencyOn-Device Language Model Inference

    University of Massachusetts Amherst Amherst, Massachusetts, USA · Brave Software London, UK · Brave Software, Imperial College London London, UK

  11. Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference

    May 1, 2026Abdurrahman Javat, Allan KazakovLLM Inference EfficiencyLLM Quantization

    Bahcesehir University, Istanbul, Türkiye

  12. DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference

    Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model InferenceKV-Cache Offloading

    1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.

  13. AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices

    Apr 28, 2026Ma Zirui, Fan Zhihua, Li Wenxing +4Compute-in-MemoryOn-Device Language Model Inference

    State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  14. Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices

    Apr 27, 2026Zihuai Xu, Ruofei Hou, Yang Xu +3Retrieval-Augmented GenerationOn-Device Language Model Inference

    Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123

  15. Less Is More: Engineering Challenges of On-Device Small Language Model Integration in a Mobile Application

    Apr 27, 2026William OliveiraOn-Device Language Model InferenceSmall Language Models

    Independent Researcher

  16. BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment

    Apr 27, 2026Md. Ashiq Ul Islam Sajid, Mohammad Sakib Mahmood, Md. Tareq Hasan +3LLM QuantizationRL for Language Models

    Department of Computer Science BRAC University Dhaka 1212, Bangladesh · Department of Computer Science Missouri State University Springfield, MO, USA · Department of Computer Science & Engineering Prime University Dhaka, Bangladesh +2

  17. An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

    Apr 25, 2026Jelena Ilić VulićevićLLM EvaluationOn-Device Language Model Inference

    Independent Researcher

  18. Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers

    Apr 24, 2026Harri Renney, Fouad Trad, Michael Mattarock +1AI Accelerator InferenceOn-Device Language Model Inference

    Kaze Technologies Kaze Consulting Bath, BA1 2HN, UK · Electrical and Computer Engineering Lebanese American University Byblos, Lebanon · Carnegie Mellon University Pittsburgh Pennsylvania USA +1

  19. Multilinguality at the Edge: Developing Language Models for the Global South

    Apr 23, 2026Lester James V. Miranda, Songbo Hu, Roi Reichart +1Multilingual Language ModelsOn-Device Language Model Inference

    Language Technology Lab, University of Cambridge, United Kingdom · Technion – Israel Institute of Technology, Israel

  20. SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

    Apr 23, 2026Hongyao Liu, Liuqun Zhai, Junyi Wang +3On-Device Language Model InferenceKV-Cache Offloading

    Department of Computer Science, City University of Hong Kong, Hong Kong SAR.

  21. Micro Language Models Enable Instant Responses

    Apr 21, 2026Wen Cheng, Tuochao Chen, Karim Helwani +3On-Device Language Model InferenceSmall Language Models

    Paul G. Allen School of Computer Science & Engineering, University of Washington · Meta AI

  22. Toward Zero-Egress Psychiatric AI: On-Device LLM Deployment for Privacy-Preserving Mental Health Decision Support

    Apr 20, 2026Eranga Bandara, Asanga Gunaratna, Ross Gore +9On-Device Language Model InferencePrivacy-Preserving Language Models

    Old Dominion University, Norfolk, VA, USA · AI Motion Labs, Melbourne, Australia · Department of Psychiatry and Neurobehavioral Sciences, University of Virginia School of Medicine, Charlottesville, VA, USA +3

  23. Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM

    Apr 20, 2026Sravanth Kodavanti, Sowmya Vajrala, Srinivas Miriyala +13On-Device Language Model InferenceLLM Inference

    Samsung Research Institute Bangalore, India · Samsung Electronics, Suwon, South Korea

  24. A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models

    Apr 15, 2026Jason Kong, Nilesh Prasad Pandey, Flavio Ponzina +1Quantization Sensitivity AnalysisLLM Quantization

  25. EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models

    Apr 13, 2026Jinane Bazzi, Mariam Rakka, Fadi Kurdahi +2On-Device Language Model InferenceEfficient Language Model Inference

    King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · the University of California Irvine, Irvine, CA, USA. · Rain Neuromorphics Inc., San Francisco, CA, USA

  26. SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUs

    Apr 8, 2026Jintao Zhang, Xuanyao FongOn-Device Language Model InferenceEfficient Language Model Inference

    Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  27. Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

    Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache QuantizationOn-Device Language Model Inference

    Clemson University, USA

  28. MUXQ: Mixed-to-Uniform Precision MatriX Quantization via Low-Rank Outlier Decomposition

    Apr 6, 2026Seoungsub Lee, In Seo Kim, Seon Wook KimLLM QuantizationNeural Network Quantization

    School of Electrical Engineering, Korea University, Seoul, Korea

  29. Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence

    Mar 27, 2026Eziyo Ehsani, Luca Giamattei, Ivano Malavolta +1On-Device Language Model InferenceEnergy-Efficient ML

    ∗University of Naples Federico II, Naples, Italy · †Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

  30. PLACID: Privacy-preserving Large language models for Acronym Clinical Inference and Disambiguation

    Mar 24, 2026Manjushree B. Aithal, Ph. D., Alexander Kotz +2Clinical NLPPrivacy-Preserving Language Models

    Department of Biomedical Informatics, University of Colorado Anschutz, Aurora, CO