On-Device Language Model Inference

Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 111

All topics
CardsList

  1. Enhancing Virtual Agents through SLMs and Edge-Computing: An Exploratory Evaluation of Think and Memory Processes

    Aug 13, 2026Aimilios Hadjiliasi, Louis NisiotisCognitive Architectures for AI AgentsOn-Device Language Model Inference

    Department of Computing, Engineering and Mathematics University of Central Lancashire, Cyprus

  2. MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices

    Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model InferenceLLM Inference Scheduling

    Kyungpook National University Daegu, Republic of Korea

  3. GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization

    Aug 6, 2026Sajjad Ghiasvand, Nader SehatbakhshOn-Device Language Model InferencePrivacy-Preserving Language Models

    Electrical and Computer Engineering Department, UC Santa Barbara1 · Electrical and Computer Engineering Department, UC Los Angeles2

  4. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator InferenceOn-Device Language Model Inference

    KAIST Daejeon, South Korea

  5. Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs

    Aug 5, 2026Kuanysh Akhmetzhanov, Jurn-Gyu ParkFine-TuningOn-Device Language Model Inference

    Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan

  6. Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

    Aug 3, 2026Anusha Madan Gopal, Aras Pirbadian, Kristofor D. Carlson +2Retrieval-Augmented GenerationOn-Device Language Model Inference

    BrainChip Inc. 23041 Avenida de la Carlota, Laguna Hills, CA

  7. Opt.Gear Technical Report

    Aug 2, 2026Juneyoung Park, Youngwook KwonOn-Device Language Model InferenceLong-Context Language Model Inference

    Opt.Gear Team

  8. CallScreenBench: Benchmarking On-Device Models as Phone Secretaries

    Aug 2, 2026Simiao RenOn-Device Language Model InferenceLLM Agent Evaluation

  9. From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference

    Jul 27, 2026Darina Gold, Alexander Schwirjow, Viktor Haag +4On-Device Language Model InferenceTraining Data Curation

    IIS Fraunhofer

  10. DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

    Jul 27, 2026Dengke HanExpert OffloadingOn-Device Language Model Inference

  11. Profiling Lightweight Large Language Models

    Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference EfficiencyLLM Evaluation

    Graduate School of Science and Engineering, Saitama University, Saitama, Japan

  12. BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

    Jul 21, 2026Fabian Waschkowski, Prabod Rathnayaka, Lukas WesemannAI Accelerator InferenceOn-Device Language Model Inference

    Base Compute, Melbourne, Australia

  13. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1On-Device Language Model InferenceMemory-Efficient Inference

    Department of Computer Networks, Paderborn University, Paderborn, Germany

  14. AdaHome: An Adaptive Smart Home Assistant using Local Small Language Models

    Jul 20, 2026Eu Jin Lim, Zhaoxing Li, Sebastian SteinAdaptive InferenceOn-Device Language Model Inference

    University of Southampton Southampton, United Kingdom

  15. Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

    Jul 19, 2026Param ChordiyaOn-Device Language Model InferenceSpeculative Decoding

    University of California, San Diego

  16. SmartRAG: Native Graph-Based RAG for Mobile Device

    Jul 16, 2026Zhihan Jiang, Meng Li, Shenghao Liu +5Named Entity RecognitionOn-Device Language Model Inference

    1Nanjing University · 2HUST · 3Southeast University

  17. PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

    Jul 16, 2026Hyunwoo Oh, Suyeon Jang, Hanning Chen +4Mixed-Precision QuantizationLLM Quantization

    University of California, Irvine

  18. Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

    Jul 14, 2026Yi Li, Chen Li, Jiexiong LiuOn-Device Language Model InferencePrivacy-Preserving Language Models

    KunlunMeta

  19. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1On-Device Language Model InferenceLLM Inference Scheduling

    School of Computer Science, Nanjing University Nanjing, China

  20. Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

    Jul 10, 2026Tsz-To WongOn-Device Language Model InferenceEfficient Language Model Inference

    National Yang Ming Chiao Tung University, Hsinchu, Taiwan

  21. Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

    Jul 9, 2026Vinay Kumar ChagantiOn-Device Language Model InferenceLLM Information Extraction

    independent researcher

  22. When Words Predict Workload

    Jul 6, 2026Anubhab BanerjeeOn-Device Language Model InferenceLLM Inference

    Nokia Germany

  23. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

    Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference EfficiencyAI Accelerator Inference

    Tsinghua University · Beijing Jiaotong University

  24. Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

    Jul 6, 2026Daeyeon SonOn-Device Language Model Inference

    Independent Researcher · Republic of Korea

  25. AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

    Jul 4, 2026Sadra SaremiOn-Device Language Model InferenceMemory-Efficient Inference

    Department of Physics, Iran University of Science and Technology, Tehran 16846-13114, Iran.

  26. STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

    Jul 3, 2026Nirhoshan Sivaroopan, Albert Zomaya, Kanchana ThilakarathnaOn-Device Language Model InferenceHuman Activity Recognition

    The University of Sydney · Australia

  27. BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

    Jul 1, 2026Prabod Rathnayaka, Fabian Waschkowski, Lukas WesemannAI Accelerator InferenceOn-Device Language Model Inference

    Base Compute, Melbourne, Australia

  28. Little Brains, Big Feats: Exploring Compact Language Models

    Jun 29, 2026Dari Baturova, Elena Bruches, Ivan Chernov +3Retrieval-Augmented GenerationOn-Device Language Model Inference

    Siberian Neuronets LLC, Novosibirsk, Russia

  29. MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar

    Jun 28, 2026Yi RenRetrieval-Augmented GenerationOn-Device Language Model Inference

    École Polytechnique Fédérale de Lausanne, Switzerland

  30. Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT

    Jun 25, 2026Jinghan Wang, Yanjun Chen, Wei Zhang +3LLM PruningOn-Device Language Model Inference

    School of Mechanical and Electrical Engineering, Harbin Institute of Technology, Harbin 150001, China · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China