On-Device Language Model Inference

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Jun 22, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University
EnerInfer: Energy-Aware On-Device LLM Inference
Jun 22, 2026Bohua Zou, Nian Liu, Binqi Sun +6On-Device Language Model Inference LLM Inference Acceleration
Technical University of Munich · Shanghai Jiao Tong University · Huawei Hilbert Research Center (Dresden) +1
Enabling Cloud-Level Accuracy in Edge AI through IoT Data Preprocessing
Jun 21, 2026Aygün Varol, Katarzyna Kołodziej, Łukasz Sobczak +5On-Device Language Model Inference LLM Prompting
Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland · Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland · Department of Computer Science, University of Helsinki, Helsinki, Finland +1
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
Jun 19, 2026Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard +2LLM Inference Efficiency On-Device Language Model Inference
Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada · TSAI Lab, University of New Brunswick, Fredericton, NB, Canada
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
Jun 17, 2026Craig OpieExpert Offloading On-Device Language Model Inference
Holocron Security, Inc.
Efficient On-Device Diffusion LLM Inference with Mobile NPU
Jun 11, 2026Tuowei Wang, Yanfan Sun, Ju RenAI Accelerator Inference On-Device Language Model Inference
Beihang University China · Tsinghua University China
Closing the Social-Semantic Gap: SPSD for Edge-Based Prompt Compression in Cloud LLM Inference
Jun 10, 2026Abhinit Sen, Ajeet Kumar, Manaranjan PradhanLLM Inference Efficiency On-Device Language Model Inference
Indian School of Business, Hyderabad, India
TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs
Jun 9, 2026Wesley Pang, Gregory Hyegang Jun, Feiyang Liu +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign Department of Electrical and Computer Engineering USA
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Jun 9, 2026Wenxin Wang, Yule Hou, Yu Ji +2Expert Parallelism On-Device Language Model Inference
1Tsinghua University · 2Xingyun Integrated Circuits Co., Ltd. · 3Beijing National Research Center for Information Science and Technology
Distilling Safe LLM Systems via Soft Prompts for On Device Settings
Jun 8, 2026Motasem Alfarra, Cristina Pinneri, Dana Kianfar +2On-Device Language Model Inference LLM Safety Alignment
Qualcomm AI Research
Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge
Jun 3, 2026Haotian Zheng, Zhanwei Wang, Mingyao Cui +3On-Device Language Model Inference Edge Computing
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China
Operator Fusion for LLM Inference on the Tensix Architecture
Jun 3, 2026Qingbo Wu, Ke Li, Wenzhu Wang +3Efficient Transformer Inference AI Accelerator Inference
KylinSoft Co., Ltd, Tianjin, China · openKylin Community, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology, Changsha, China
LLM Compression with Jointly Optimizing Architectural and Quantization choices
Jun 2, 2026Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1Mixed-Precision Quantization LLM Quantization
UiT The Arctic University of Norway · University of Oslo, Norway
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows
May 29, 2026Yuri Balashov, Rex VanHorn, Mingxi Xu +1Multilingual Language Model Evaluation On-Device Language Model Inference
University of Georgia, Athens, Georgia, USA
When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systems
May 28, 2026Corrado Rainone, Davide Belli, Bence Major +1Multi-Agent LLM Systems On-Device Language Model Inference
Qualcomm Technologies, Inc. 1Qualcomm AI Research†
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
May 26, 2026Fengfa Li, Hongjin Ji, Yifeng Ding +2Mixture-of-Experts Pruning LLM Pruning
Li Auto · The Chinese University of Hong Kong, Shenzhen
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
May 22, 2026Pu Li, Jiawen Qi, Qinyu ChenAI Accelerator Inference On-Device Language Model Inference
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands.
Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
May 19, 2026Jinghe Zhang, Daliang Xu, Chenghua Wang +5Mixed-Precision Quantization LLM Quantization
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4On-Device Language Model Inference Hardware-Aware NAS
Brown University · University of Michigan · Google Research
OpenJarvis: Personal AI, On Personal Devices
May 16, 2026Jon Saad-Falcon, Avanika Narayan, Robby Manihani +10On-Device Language Model Inference LLM Agents
Stanford University · Lambda Labs
Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model Inference LLM Inference
Tsinghua University · Beijing, China · Beihang University +1
CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference
May 12, 2026Nan Xue, Shengkang Chen, Zhiyong Chen +4On-Device Language Model Inference LLM Inference
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China · Department of Broadband Communication, Pengcheng Laboratory, Shenzhen 518000, China · Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen), Shenzhen 518172, China +1
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1On-Device Language Model Inference Memory-Efficient Inference
University of Florida Gainesville, FL 32611
Agent-X: Full Pipeline Acceleration of On-device AI Agents
May 11, 2026Jinha Chung, Byeongjun Shin, Jiin Kim +1LLM Inference Efficiency On-Device Language Model Inference
KAIST Republic of Korea, Daejeon
GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference
May 11, 2026Zengzipeng Tang, Yuxuan Sun, Wei Chen +2On-Device Language Model Inference Edge Computing
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
Large Language Models over Networks: Collaborative Intelligence under Resource Constraints
May 9, 2026Liangqi Yuan, Wenzhi Fang, Shiqiang Wang +2LLM Inference Efficiency On-Device Language Model Inference
L. Yuan, W. Fang, and C. G. Brinton are with Purdue University, IN, USA. · S. Wang is with University of Exeter, UK. · H. V. Poor is with Princeton University, USA.
Benchmarking Local Language Models for Social Robots using Edge Devices
May 4, 2026Dorian Lamouille, Matevž B. Zorec, Farnaz Baksh +1LLM Evaluation On-Device Language Model Inference
University of Tartu Tartu, Estonia · Institute of Computer Science University of Tartu Tartu, Estonia · Institute of Technology University of Tartu Tartu, Estonia
Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Pruning On-Device Language Model Inference
Meta

Jun 22, 2026, cs.CRJ/K move · Enter open · S save

FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

Yinpeng Wu, Yitong Chen, Lixiang Wang, Jinyu Gu+2

Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

LLM Serving On-Device Language Model Inference Trusted Execution Environments LLM Security LLM Inference Acceleration

Open paper AskPDF arXiv

Device-side Large Language Models (LLMs) have grown explosively, offering stronger privacy and higher availability than their cloud-side counterparts. During LLM inference, both the model weights and the user data are valuable, and attackers may compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead to both the secure inference and the normal aplications, due to two challenges: the inflexible resource isolation and the inefficient secure resource management. To address these challenges, this paper presents FlexServe, a fast and secure LLM inference system for mobile devices. The key idea is to decouple the access permission from the management permission of secure resources, so that the normal-world OS cannot access them but can still manage them as usual. First, FlexServe introduces a Recallable Resource Isolation mechanism to construct Recallable Secure Memory (Flex-Mem) and a Recallable Secure NPU (Flex-NPU). They can only be accessed by the secure world, but can be efficiently allocated and reclaimed by the normal-world OS. Based on them, FlexServe further introduces a FlexServe Framework to run secure LLM inference in the secure world. It works together with the normal-world OS to perform cooperative secure memory management. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves average TTFT speedups of 10.05X over the strawman and 2.44X over an optimized strawman.

Momentum

Related topics

Latest papers 111