On-Device Inference

Ask about this topic

Overview

On-device inference studies how to run models locally on phones, wearables, robots, and tiny microcontrollers when connectivity, privacy, latency, memory, or battery life is limited. Common approaches include quantization, pruning and distillation, compact hardware-aware architectures, and runtimes that manage memory, accelerators, and computation across devices. Applications range from keyword spotting and wearable activity recognition to mobile language models, image generation, medical screening, and robot control, evaluated on target hardware and task benchmarks. A central challenge is balancing accuracy and capability against real latency, energy, and memory costs, which vary across hardware; deciding when to compute locally, adapt, or offload remains an open systems trade-off.

Latest papers

CardsList

190 papers18 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

  1. Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device

    Feb 23, 2026Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad +8Unified Multimodal ModelsMultimodal Generation

    Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University · Linköping University

  2. How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perf

    Feb 20, 2026Wenqi Jiang, Jason Clemons, Karu Sankaralingam +1Efficient VLA Model InferenceOn-Device Inference

    NVIDIA Research

  3. FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations

    Feb 17, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Hangyu Zheng +4Efficient Neural Network InferenceMemory-Efficient Inference

    University of Georgia Athens, GA, USA · University of Texas at Arlington Arlington, TX, USA

  4. FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices

    Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge InferenceMixture-of-Experts Inference

  5. SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices

    Jan 13, 2026Dongting Hu, Aarush Gupta, Magzhan Gabidolla +12Diffusion TransformerDiffusion Model Distillation

    Snap Inc. · University of Melbourne · MBZUAI

  6. An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile Devices

    Jan 6, 2026Jeiyoon Park, Daehwan Lee, Changmin Yeo +2Machine TranslationOn-Device Inference

    SOOP

  7. Animal Re-Identification on Microcontrollers

    Dec 9, 2025Yubo Chen, Di Zhao, Yun Sing Koh +1Efficient Neural Network InferenceAnimal Re-Identification

    University of Auckland Auckland, New Zealand

  8. AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devices

    Nov 27, 2025Mengyang Liu, Chenyu Lu, Haodong Tian +7Edge MLEfficient Neural Network Inference

    Southeast University · Hong Kong University of Science and Technology · Tsinghua University +1

  9. Energy Overhead in Sensor Nodes: Embedded Machine Learning and Data Offloading via Low Power Wide Area Networks

    Oct 28, 2025Benjamin Karic, Nina Herrmann, Jan Stenkamp +3Internet of ThingsEnergy-Efficient ML

    University of Münster · re:edu GmbH & Co. KG

  10. Robust Wake-Up Word Detection by Two-stage Multi-resolution Ensembles

    Oct 17, 2023Fernando López, Jordi Luque, Carlos Segura +1Keyword SpottingEnsemble Learning

    Telefónica I+D, Digital Life Disruption Lab, Spain · Universidad Autónoma de Madrid · Telefónica I+D, Research, Spain