On-Device Language Model Inference

Overview

The field studies how to run useful language models locally, where limited memory, battery life, heat, and compute can constrain both model size and response speed. Common approaches include quantization and pruning, compact or mixture-of-experts models, KV-cache compression, speculative decoding, and hardware-aware runtimes for mobile CPUs, GPUs, and NPUs. Other systems split work between a device and the cloud or route tasks among local models to balance privacy, cost, and capability. A central tension is that gains in speed or memory do not always preserve accuracy, reliability, or energy efficiency, so evaluation increasingly measures full workloads on real devices rather than relying on model size or peak throughput alone.

Latest papers

CardsList

189 papers23 papers in the last four weeks, down 26% on the four weeks before. 0.2% of all new papers.

  1. CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

    Nov 15, 2026Byeongju Kim, Hoonki Lee, Byungjun Kim +4On-Device Language Model InferenceMixture-of-Experts Inference

  2. Offline-First Quantized AI: Design and Evaluation of an On-Device Maternal-Health Assistant for Low-Connectivity Environments

    Nov 15, 2026Okenla AbdulQahhar Abimbola, Maurice Abraham John, Adepoju Jeremiah Moyondafoluwa +10Local LLM DeploymentHealthcare

  3. MeReader: An Offline, Privacy-Preserving, Progress-Aware AI Assistant for Narrative eBook Reading

    Nov 13, 2026Mohammed Efaz, Udo Bub, Itilekha PodderPrivacy-Preserving RAGOn-Device Language Model Inference

  4. LazyWeights: Decode-on-Demand Edge LLMs

    Nov 12, 2026Arnab Sanyal, Gourav Datta, Sandeep P. ChinchaliOn-Device Language Model InferenceLLM Compression

  5. EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

    Nov 10, 2026Zhengxiang Huang, Yujie Sun, Shengheng Chen +6On-Device Language Model InferenceLLM Inference

  6. HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios

    Nov 10, 2026Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman MushfiqueMedical QAClinical Language Model Evaluation

  7. Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

    Nov 10, 2026Evan Li, Krishna Deshpande, Wen Cheng +1On-Device Language Model Fine-TuningSmall Language Models

  8. Hibana: Fast Agentic Inference on Unified-Memory Devices

    Nov 7, 2026Yahya Emara, Chi-Chih Chang, Anthony Fei +5On-Device Language Model InferenceLLM Inference Acceleration

  9. MedPsy: State-of-the-Art Small Medical Language Models for Efficient Edge Deployment

    Nov 1, 2026Davide Vitabile, Alexandro Buffa, Akshay Nambiar +1Medical LLMsSmall Language Models

  10. MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference

    Nov 1, 2026Renjie Wei, Haochen Huang, Chuyu Qiu +3Mixture-of-Experts InferenceMixture-of-Experts Quantization

  11. BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

    Oct 31, 2026Hangyu Zheng, Zhihao Shu, Kunxiong Zhu +4On-Device Language Model InferenceLLM Inference Acceleration

  12. Compression Technique Interactions in Small Language Models: A Factorial Ablation Study

    Oct 30, 2026Bhushit Nema, Siddharth Krishnamurthy, Anjali KadiyalaSmall Language ModelsLLM Quantization

  13. TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

    Oct 30, 2026Wentao Ni, Hong Cai, Mingu Lee +7LLM Inference AccelerationOn-Device Language Model Inference

  14. Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

    Oct 22, 2026Tianxiang Xia, Kai Lion, Niao He +3LLM CompressionLLM Inference Acceleration

  15. Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

    Oct 22, 2026Kyrylo Yemets, Andrii Yaroshevych, Victor Muryn +5LLM InferenceOn-Device Language Model Inference

  16. Local Inference of LLMs on Long Contexts: A Data Systems Perspective

    Oct 20, 2026Mattias Dürrmeier, Alberto LernerLLM InferenceLong-Context Inference

  17. Tandem: Serving Hybrid Linear-Attention GUI Agents on Mobile NPUs

    Oct 19, 2026Zhuomin Chen, Jingchao Ni, Xu Zheng +4On-Device Language Model InferenceMobile GUI Agents

  18. Muse Glimmer 30B: An On-Device Agentic Model

    Oct 17, 2026Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed +326LLM AgentsLLM Tool Use

  19. MedPAO-Assist: Edge-Deployable, Modality-Aware Agentic AI for Radiology Report Structuring

    Oct 17, 2026Shrish Shrinath Vaidya, Minmini Selvam, Ganapathy KrishnamurthiRadiology Report GenerationMedical Report Generation

  20. BiShield-TEE: On the (In-)Security of Unilateral Weight Obfuscation in On-Device TEE-Shielded LLM Partition

    Oct 15, 2026Zhiyuan Wang, Yuezhi Zhou, Ziqi Zhang +3LLM SecurityOn-Device Language Model Inference

  21. HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding

    Oct 15, 2026Hanbin Bae, Heejin Choi, Joohyung Lee +3Speculative DecodingLLM Inference Acceleration

  22. PocketTSLM: Deploying an On-Device Soft-Prompt Time-Series Language Model for Privacy-Preserving Smartphone Reasoning

    Oct 15, 2026Max Rosenblattl, Nikhil Krishnaswamy, Leon Nissen +9Time Series Language ModelingOn-Device Language Model Inference

  23. ixi-sLM: On-Device sLM for AI Call Agent Service

    Oct 14, 2026Han-Sang Lee, Kyuho Lee, Juneyoung Park +10On-Device Language Model InferenceSmall Language Models