On-Device Inference

Follow this topic Ask about this topic

On-device inference studies how to run models locally on phones, wearables, robots, and tiny microcontrollers when connectivity, privacy, latency, memory, or battery life is limited. Common approaches include quantization, pruning and distillation, compact hardware-aware architectures, and runtimes that manage memory, accelerators, and computation across devices. Applications range from keyword spotting and wearable activity recognition to mobile language models, image generation, medical screening, and robot control, evaluated on target hardware and task benchmarks. A central challenge is balancing accuracy and capability against real latency, energy, and memory costs, which vary across hardware; deciding when to compute locally, adapt, or offload remains an open systems trade-off.

Latest papers

190 papers18 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4
Oct 28, 2026Billal BoumedineAutomatic Speech Recognition On-Device Inference
Faculty of Computer Science, University of Science and Technology Houari Boumediene (USTHB), Algiers, Algeria
MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices
Oct 27, 2026Kailai Feng, Haoxiang Rao, Yuxiang Wei +5Image Editing Evaluation Image Generation Evaluation
University College London · Nanjing University · The Hong Kong Polytechnic University +1
AIPC: Evidence-Guided Agent Workflows for PyTorch Project Deployment
Oct 26, 2026Zhanwei Wu, Jianhao Su, Brian Zhang +11AI Accelerator Inference LLM Agent Workflow Optimization
Qualcomm Technologies, Inc · Independent Researcher
VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware
Oct 26, 2026Borui Xu, Hongshi Tan, Wenqi Jiang +4Efficient VLA Model Inference On-Device Inference
National University of Singapore
Energy and Latency First: How Deployment Constraints Reverse Time-Series Model Selection
Oct 22, 2026Shruti Bothe, Illyyne Saffar, Ayush K. VarshneyModel Selection Energy-Efficient ML
Ericsson Research, Santa Clara, USA · Ericsson Research, Massy, France · Ericsson Research, Kista, Sweden
AgentDP: Towards Automated Model Deployment for Heterogeneous Embedded AI Applications
Oct 22, 2026Leming Shen, Qiang Yang, Xinyu Huang +2MLOps On-Device Inference
The Hong Kong Polytechnic University · University College London · U2University of Cambridge
GANet: A Lightweight Group Attention Network for Plant Disease Identification on Embedded Systems
Oct 21, 2026Anubhab Tripathi, Bibek Das, Asfak Ali +1Plant Disease Classification Lightweight CNNs
Department of Electronics and Telecommunication Engineering, Jadavpur University, Kolkata, India
SAKURA: Serving Attention Key-Value State Under Rate Adaptation for Private Image Editing on Phones
Oct 21, 2026Shuncheng Jie, Desen Sun, Ri Hong +3Edge-Cloud Collaborative Inference Diffusion Model Serving
University of Waterloo Canada
Which Depth Estimator for Jetson AGX Orin? A 2026 Benchmark of Monocular and Stereo Methods
Oct 19, 2026Leaf JiangMetric Depth Estimation Monocular Depth Estimation
NODAR Inc.
Mind Your Boundaries: Boundary-Aware Model Obfuscation against Cross-node Attack
Oct 19, 2026Yuyang Zhang, Yihui Jin, Kangjie Chen +2Adversarial Robustness Adversarial Attacks
A Generic On-Board AI Processing Chain for Earth Observation: From Application Development to Embedded Deployment and Mission-Level Orchestration
Oct 19, 2026Michel-Fran{\c c}ois Foulon, Liz Ramos Medina, Thomas Goudemant +5On-Device Inference Remote Sensing
TAS · IRT Saint Exupéry, France
ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator
Oct 15, 2026Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem UlkarHardware-Aware Quantization Neural Network Quantization
Analog Devices, Inc.
Fault Detection and Bit-Error Rate Diagnostics in Optical Wireless Links Using Edge-AI
Oct 15, 2026Sarthak Bhattarai, Sheshadri Shikhar Bandyopadhyay, Suprabh Dhakal +1Fault Diagnosis Wireless Communications
Department of Electronics and Computer Engineering Pulchowk Campus, Institute of Engineering, Tribhuvan University Lalitpur, Nepal
Linear-Time Non-Local Codec Placement for Machine Learning Model Compression
Oct 10, 2026Matteo SpallanzaniEfficient Neural Network Inference Neural Network Compression
Axelera AI Zürich, Switzerland
16-bit Precision of Convolutional Neural Networks on Microcontroller Units for 8-bit Costs
Oct 2, 2026Rui Liu, Benjamin PaaßenNeural Network Quantization Efficient Neural Network Inference
Faculty of Technology, Bielefeld University
EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures
Oct 2, 2026Yuhai Long, Yuanxin Wei, Kai Wu +3On-Device Inference LLM Inference Scheduling
School of Computer Science and Engineering, Sun Yat-sen University · Sun Yat-sen University School of Computer Science and Engineering Guangzhou, China · China Mobile Internet Company Ltd +1
LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing
Oct 1, 2026Tianhao Wu, Xu Wu, Amirmohammad Radmehr +4EMG-Based Gesture Recognition Time-Series Foundation Models
University of Georgia, Athens, Georgia, United States · Computer Science, University of Oklahoma, Norman, Oklahoma, United States · Wireless and Sensor Systems Lab (WSSL), University of Massachusetts Amherst, Amherst, Massachusetts, United States +1
Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference
Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient Inference Convolutional Neural Networks
PowerLabs Technologies, Lagos, Nigeria
Bad: Taming the Bioacoustic Data Deluge with a Bat Activity Detector
Sep 29, 2026Stefano Ciapponi, Santiago Martinez Balvanera, Andrea Cesaretti +2Sound Event Detection Bioacoustics
Fondazione Bruno Kessler · University of Trento · University College London
Privacy-Friendly Cohort Determination: Sealed, CSP-Independent In-Browser ML Inference of Professional Segments for Identity-Less Advertising
Sep 28, 2026Om Shankar Tiwari, Navnit Shukla, Guanyu Wang +1Online Advertising Privacy-Preserving ML
Applied AI Technical Lead, Google · Principal AI Architect, Snowflake Inc. · Staff Software Engineer, TikTok +1
Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edge
Sep 28, 2026Yitong Li, Jincheng Yu, Junsong Chen +6Video Diffusion Models GPU Kernel Optimization
NVIDIA Research, Efficient AI Team & Singapore Lab.
Sub-Model Short-Term Memory Convolutions for Keyword Spotting Systems on Device
Sep 28, 2026Paweł Warlewski, Artur Czeczko, Artur Szumaczuk +2Keyword Spotting Speech Processing
Samsung R&D Institute Poland · Samsung AI Center Warsaw
On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation
Sep 27, 2026Moghis Fereidouni, Anthony Arnold, Sumit Gulwani +2LLM Inference Efficiency Tool-Augmented Language Model Agents
University of Kentucky · Microsoft
PastForward: Faster On-Device GUI Agents via Computational Experience Reuse
Sep 26, 2026Taehwan Park, Changmin Lee, Hayeon Lee +1Mobile GUI Agents On-Device Inference
Meta Superintelligence Labs
Empowering Hybrid Attention Models on NPUs
Sep 26, 2026Yinyuan Zhang, Daliang Xu, Xiaolong Huang +4LLM Inference Acceleration On-Device Inference
Peking University · Beijing University of Posts and Telecommunications
Cache-Aware Conv3D Lowering Across Embedded World-Model Decoders
Sep 25, 2026Jiaming Zhang, Wu Yang, Shuai Tao +1Inference Acceleration On-Device Inference
University of Wisconsin–Madison, Madison, WI, USA · Beta Infinity, Beijing, China
I-Parakeet: Integer-Only Conformer ASR on Mobile NPU
Sep 25, 2026Taichi NishimuraAutomatic Speech Recognition Efficient Neural Network Inference
Sony Interactive Entertainment, Tokyo, Japan
LUMO (Lightweight Unified Multilingual Orchestrator): A Privacy Preserving Offline Voice Assistant
Sep 25, 2026Md. Mehedi Hasan Naeem, Mst. Kamrunnahar Ruma, Nafiza Anjum +2Conversational Agents On-Device Inference
Department of Computer Science and Engineering Jatiya Kabi Kazi Nazrul Islam University Mymensingh, Bangladesh
Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement
Sep 24, 2026Clément Laroche, Riccardo MicciniEfficient Neural Network Inference Speech Enhancement
GN A/S, Denmark · Technical University of Denmark, Denmark
Edge AI on Constrained Devices for Binary Sleep-Wake Classification in Dynamic Environments
Sep 24, 2026Stefan Reitmann, Lena OdenTime Series Classification Wearable Sensing
Faculty of Computer Science, Chemnitz University of Technology, Chemnitz, Germany · Faculty of Computer Science, University of Hagen, Hagen, Germany

Oct 28, 2026J/K move · Enter open · S save

Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4

Billal Boumedine

Faculty of Computer Science, University of Science and Technology Houari Boumediene (USTHB), Algiers, Algeria

Automatic Speech Recognition On-Device Inference Streaming ASR Post-Training Quantization

Open paper AskPDF arXiv

We run Kroko, a 65.7M-parameter streaming Zipformer2 transducer for English, on the ESP32-P4, a microcontroller with two 400 MHz RISC-V cores, a 128-bit SIMD extension (PIE) and 32 MB of PSRAM, but no neural accelerator. Without any training, the encoder's 290 matrices are stored with 2 or 4 bits per weight in Hadamard-rotated groups: the width of each matrix comes from its measured sensitivity, and each matrix is first corrected toward the original model's output, then rounded with GPTQ and a search over the group norms. The released 71 MB of int8 weights become a 26.65 MB file. Activations are int16 between the matrix products and int8 into them. A C engine with hand-written PIE kernels processes each 1.28 s chunk of audio in 0.90 s on both cores, beam search included, and its PC build computes the same bits as the chip, so accuracy is measured on whole test sets. LibriSpeech WER is 4.34% on test-clean and 10.26% on test-other, against 3.94% and 9.17% for the released model; on FLEURS and Common Voice the gap grows to 2.3 and 1.1 points. A GPU simulator shows that the integer arithmetic costs nothing and that the whole gap comes from the weights, and an ablation on all of test-clean shows that GPTQ does most of the work. We also report the port's speed history, from 51.3 s to 0.62 s of encoder time per chunk, each design step measured again in one build, an attempt to reach 16 MB that stopped at 19.63 MB, and what did not work. We did not measure energy. The inference engine, the firmware and the 2/4-bit weights are available at https://github.com/BoumedineBillal/esp32-pie-asr.