On-Device Language Model Inference

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 229% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

Decoupling Logic from Persona: Structural Immunity of Edge LLM Agents to Context Pollution
Oct 7, 2026Masaaki Nakatsu, Reno WangLLM Inference LLM Context Management
AO, Inc. / OrbLabs AG · AO, Inc.
Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Oct 5, 2026Arnav Kundu, Zhaoyang Xu, Bairu Hou +3On-Device Language Model Inference Dynamic Neural Networks
Apple
AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing
Sep 30, 2026Zhihui Gao, Tingjun Chen, Dirk EnglundOn-Device Language Model Inference LLM Inference Acceleration
Massachusetts Institute of Technology Cambridge, MA, USA · Duke University Durham, NC, USA
HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
Sep 29, 2026Kenan Alkiek, Moontae Lee, David Jurgens +1On-Device Language Model Inference Selective Prediction
School of Information University of Michigan Ann Arbor, MI 48109, USA · SNU-LG AI Research Center Seoul, South Korea
AS $^2$ D: Accelerating On-Demand Audio Understanding on Mobile Devices
Sep 29, 2026Yunzhe Li, Kyoungjun Park, Hongzi Zhu +1On-Device Language Model Inference Language Model Decoding
The University of Texas at Austin · Shanghai Jiao Tong University
IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence
Sep 29, 2026Changdi Yang, Fengquan Jiao, Haochih Lin +14Multi-Token Prediction On-Device Language Model Inference
Robotics Foundation Model Team, Xpeng Inc.
Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
Sep 28, 2026Zhengxiang Huang, Shengheng Chen, Chaoyue Niu +6AI Accelerator Inference On-Device Language Model Inference
Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China · Alibaba Group Hangzhou, Zhejiang, China
OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
Sep 28, 2026Zongshang Shen, Wangsong Yin, Daliang Xu +2Efficient Multimodal Inference On-Device Language Model Inference
Key Lab of High Confidence Software Technologies (Peking University), Beijing, China · State Key Laboratory of Networking and Switching Technology (BUPT), Beijing, China
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
Sep 27, 2026PhanTan Khanh Nguyen, Ashfaq Ali Shafin, Khandaker Mamun AhmedOn-Device Language Model Inference Small Language Models
Beacom College of Computer and Cyber Sciences, Dakota State University · Mathematics and Computer Science, Augustana College
Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
Sep 24, 2026Musa ShamsExpert Offloading On-Device Language Model Inference
Independent Researcher
Universal Fractal Natural Language Decision Map: Real-Time Edge Triage Across Heterogeneous Domains
Sep 21, 2026Volkan Dağlı, Zerrin Dağlı, Dağhan DağlıOn-Device Language Model Inference Efficient Language Model Inference
Anadolu University, Eski¸sehir, Turkey · ITouch Systems, Mersin, Turkey · Mersin University, Mersin, Turkey +1
Extending FunctionGemma for Practical On-Device Mobile Function Calling
Sep 21, 2026Ali Rezagholizadeh, Soheila SamieeOn-Device Language Model Inference Tool-Using Language Agents
UGrowAI
LoRA-generating hypernetworks for efficient on-device LLM generative personalization
Sep 21, 2026Sean Augenstein, Li Ding, Jihwan Lee +2On-Device Language Model Inference Low-Rank Adaptation
Google · Formerly at Google, work done while at Google
Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model
Sep 21, 2026MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan +4Retrieval-Augmented Generation Legal NLP
Computer Science and Engineering BRAC University Dhaka, Bangladesh
this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent
Sep 20, 2026Zehua Cheng, Wei Dai, Jiahao SunOn-Device Language Model Inference LLM Decision-Making
University of Oxford
ARID: A Deployable Edge AI System for Structured Information Extraction from Industrial Maintenance Work Orders
Sep 20, 2026Kuanlin Chen, Chen-Wei KuoOn-Device Language Model Inference Document Information Extraction
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
Sep 18, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu +4On-Device Language Model Inference Long-Context Language Model Inference
University of Georgia Athens, GA, USA · Peking University Beijing, China · Western Digital Research San Jose, CA, USA
Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions
Sep 16, 2026Janghoon LeeTool-Augmented Language Model Agents On-Device Language Model Inference
Redrob
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices
Sep 14, 2026Édouard Guégain, Tristan CoignionOn-Device Language Model Inference Energy-Efficient ML
DRI, GREENSPECTOR · Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France
On-Device Language Models for Privacy-Preserving Stress Prediction: A Multimodal Evaluation on Mobile Health
Sep 14, 2026Ibukunoluwa Soyebo, Alyssa Donawa, Rodrigo Aguilar Barrios +2On-Device Language Model Inference Privacy-Preserving Language Models
Ming Hsieh Department of Electrical & Computer Engineering, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
Sep 10, 2026Hamed Jafarzadeh Asl, Yuanhao Yu, Vahid Partovi NiaLLM Inference Efficiency On-Device Language Model Inference
Huawei Noah’s Ark Lab, Canada · Huawei Noah’s Ark Lab · Polytechnique Montreal, Canada
PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling
Sep 9, 2026Weisi Yang, Stephen XiaLLM Inference Efficiency On-Device Language Model Inference
Northwestern University Evanston, Illinois, USA
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
Sep 7, 2026Abrar Shahriar Qurat-Ul-Ain MastoiTool-Augmented Language Model Agents On-Device Language Model Inference
University of the West of England, Bristol,BS16 1QY United Kingdom
Diffusion Language Models for Mobile Edge Agentic AI: Foundations, Applications, and Challenges
Sep 7, 2026Chenqi Li, Minghui Min, Dusit Niyato +1On-Device Language Model Inference Diffusion Language Model Inference
School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, China · College of Computing and Data Science, Nanyang Technological University, Singapore, 639798, Singapore · School of Engineering, Edith Cowan University, Perth, Western Australia, 6027, Australia
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference Efficiency On-Device Language Model Inference
Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1
How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?
Sep 1, 2026Wei Hu, Xiaolong Tu, Dawei Chen +3LLM Inference Efficiency On-Device Language Model Inference
Georgia State University · Toyota Motor North America
mzCache: On-Device LLM Memory Management under Multitasking
Sep 1, 2026Hongseung Yu, Minsung Kim, Jongseok Park +1On-Device Language Model Inference Memory-Efficient Inference
Seoul National University · UC Berkeley
Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMs
Sep 1, 2026Jainil Dharmil ShahLLM Quantization LLM Evaluation
Elmore Family School of Electrical and Computer Engineering · Purdue University, West Lafayette, Indiana, USA
How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Engine
Aug 22, 2026Shahir M ALLM Quantization AI Accelerator Inference
Independent researcher

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

Decoupling Logic from Persona: Structural Immunity of Edge LLM Agents to Context Pollution

Masaaki Nakatsu, Reno Wang

AO, Inc. / OrbLabs AG · AO, Inc.

LLM Inference LLM Context Management On-Device Language Model Inference Persona-Conditioned Generation

Open paper AskPDF arXiv

Small language-model agents on edge devices must hold a persona and reason correctly at once, inside one context window that fills with conversational history and persona instructions. We study what happens to the logical part of such an agent when that history is long, misleading and persona-heavy (persona-logic interference), and present a Decoupling Architecture (AO-DA) that separates logical inference ("What") from persona expression ("How") into two inference paths on one INT4 base model with hot-swappable LoRA adapters. The logic path receives only the core turn and emits a verifiable structured state (Micro-State); the persona path renders it in character with the full history. In same-base-model ablations on an Apple M2 laptop (Llama-3.1-8B-Instruct and Gemma-3-4B-it, 4-bit; 480 runs over 4 pollution levels x 3 arms x 2 tasks x 2 personas x 5 seeds) we find: (i) the decoupled logic path is structurally invariant to pollution: its prompt stays at 180 (Llama) or 167 (Gemma) tokens while the mixed single-pass prompt grows from 242 to 1,203, and its outputs are byte-identical across levels (40/40); (ii) the mixed single pass degrades monotonically (composite logic score 0.669 to 0.150 on Llama, 0.487 to 0.150 on Gemma), mostly by failing to emit the required structured output (80-95% of runs on Llama, 100% on Gemma at the two highest levels); (iii) with the same pollution fed into the decoupled logic path, the dedicated-adapter, dedicated-format path is still more robust than the single pass on the 8B model (failure 0-20% vs 80-95%; paired

Δ

+0.30 to +0.50, Cliff's

δ

0.50-0.85, Holm-adjusted

p \le 0.03

) but not on the 4B model, where both collapse. Separation costs one extra decode on a topic's first turn (28.2 s vs 18.2 s on Llama) and buys persona hot-swapping in 1.7 ms without re-running the logic path. Code, rubric, fixtures, adapters and logs are released.

Momentum

Related topics

Latest papers 111