On-Device Inference

Ask about this topic

Overview

On-device inference studies how to run models locally on phones, wearables, robots, and tiny microcontrollers when connectivity, privacy, latency, memory, or battery life is limited. Common approaches include quantization, pruning and distillation, compact hardware-aware architectures, and runtimes that manage memory, accelerators, and computation across devices. Applications range from keyword spotting and wearable activity recognition to mobile language models, image generation, medical screening, and robot control, evaluated on target hardware and task benchmarks. A central challenge is balancing accuracy and capability against real latency, energy, and memory costs, which vary across hardware; deciding when to compute locally, adapt, or offload remains an open systems trade-off.

Latest papers

CardsList

190 papers18 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

  1. Bridging the Rural Healthcare Gap: A Cascaded Edge-Cloud Architecture for Automated Retinal Screening

    May 13, 2026Nishi Doshi, Shrey ShahDiabetic Retinopathy GradingEdge-Cloud Computing

    University of Southern California · Los Angeles, USA

  2. Contextual Bandits for Resource-Constrained Devices using Probabilistic Learning

    May 13, 2026Marco Angioli, Kevin Johansson, Antonello Rosato +2Hyperdimensional ComputingContextual Bandits

    Sapienza University of Rome · Orebro University · Linkoping University +1

  3. DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

    May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Sparse Mixture-of-ExpertsMixture-of-Experts Inference

    Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China

  4. Split CNN Inference on Networked Microcontrollers

    May 10, 2026Junyu Lu, Shashwath Suresh, Hao Liu +2Efficient Neural Network InferenceEdge Inference

    Delft University of Technology, Delft, The Netherlands

  5. HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning

    May 8, 2026Jiseok Youn, You Rim Choi, Goodsol Lee +3Meta-LearningOOD Generalization

    Department of ECE and INMC, Seoul National University, Seoul, South Korea · Graduate School of Data Science, Seoul National University, Seoul, South Korea · Department of Computer Science, University of Colorado Boulder, CO, USA

  6. On-Orbit Real-Time Wildfire Detection Under On-Board Constraints

    May 7, 2026Matthias Rötzer, Veronika Pörtge, Martin Ickerott +6Efficient Neural Network InferenceWildfire Detection

    OroraTech GmbH, St.-Martin-Str. 112, 81669 Munich, Germany · Ludwig-Maximilians-Universität München, Munich, Germany

  7. PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs

    May 7, 2026Rappy Saha, Jude Haris, Nicolas Bohm Agostini +2Efficient Neural Network InferenceFPGA-Based Neural Network Acceleration

  8. ExecuTorch -- A Unified PyTorch Solution to Run AI Models On-Device

    May 5, 2026Mergen Nachin, Digant Desai, Sicheng Stephen Jia +36Heterogeneous ComputingEdge ML

    Meta · Work done while at Meta.

  9. QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

    May 5, 2026Pratik Honavar, Tejpratap GVSLMixed-Precision QuantizationMulti-Agent LLM Systems

  10. Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs

    May 5, 2026Faraz Kayani, Sarmad Kayani, Asad Ahmed +2Efficient Neural Network InferenceKnowledge Distillation

    Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

  11. Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection

    May 4, 2026Joydeep ChandraMixed-Precision QuantizationInformation Bottleneck

    Department of Computer Science & Technology, Tsinghua University, Beijing, China.

  12. AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification

    May 2, 2026Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1Efficient Neural Network InferencePlant Disease Classification

    AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam

  13. Efficient, VRAM-Constrained xLM Inference on Clients

    Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU AccelerationEfficient VLM Inference

    NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India

  14. At the Edge of the Heart: ULP FPGA-Based CNN for On-Device Cardiac Feature Extraction in Smart Health Sensors for Astronauts

    Apr 28, 2026Kazi Mohammad Abidur Rahman, Davis Rakhshan, Philipp Lütke +2Time Series ClassificationWearable Health Monitoring

    Smart Sensors Group, Hamburg University of Technology, Hamburg, Germany · Distributed Systems, Kiel University, Kiel, Germany · Networked Cyber-Physical Systems, Hamburg University of Technology, Hamburg, Germany

  15. PI-TTA: Physics-Informed Source-Free Test-Time Adaptation for Robust Human Activity Recognition on Mobile Devices

    Apr 28, 2026Changyu Li, Lu Wang, Ming Lei +4Test-Time AdaptationHuman Activity Recognition

    School of Computing and Information Technology, Great Bay University, Dongguan, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · University of Warwick, Coventry, U.K. +2

  16. Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment

    Apr 27, 2026Kaijun Zhou, Qiwei Chen, Da Peng +3AI Accelerator InferenceEdge Inference

    School of Computer Science, Shanghai Jiao Tong University

  17. GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

    Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM InferenceTraining Data Curation

    University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.

  18. On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcontroller

    Apr 24, 2026Jeremy EllisOn-Device TrainingTiny ML

    High School Robotics Educator, British Columbia, Canada

  19. Bridging the Training-Deployment Gap: Gated Encoding and Multi-Scale Refinement for Efficient Quantization-Aware Image Enhancement

    Apr 23, 2026Dat To-Thanh, Nghia Nguyen-Trong, Hoang Vo +2Image EnhancementOn-Device Inference

    University of Science, VNU-HCM, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, VNU-HCM, Vietnam +2

  20. KD-Judge: A Knowledge-Driven Automated Judge Framework for Functional Fitness Movements on Edge Devices

    Apr 21, 2026Shaibal Saha, Fan Li, Yunge Li +3On-Device Inference

    Department of Computer Science and Engineering, Oakland University, Rochester Hills, USA · Intelligent Data Management and Analytics LLC, USA

  21. Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

    Apr 20, 2026Afsara Benazir, Felix Xiaozhu LinAI Accelerator InferenceEnergy-Efficient ML

    University of Virginia Charlottesville, Virginia, USA

  22. Towards Real-Time ECG and EMG Modeling on μμNPUs

    Apr 20, 2026Josh Millar, Ashok Samraj Thangarajan, Soumyajit Chatterjee +1Efficient Neural Network InferenceElectrocardiography

    Imperial College London London, United Kingdom · Nokia Bell Labs London, United Kingdom · Brave Software & University of Cambridge London, United Kingdom

  23. The First Challenge on Mobile Real-World Image Super-Resolution at NTIRE 2026: Benchmark Results and Method Overview

    Apr 19, 2026Jiatong Li, Zheng Chen, Kai Liu +91Image Super-ResolutionOn-Device Inference

  24. MobileAgeNet: Lightweight Facial Age Estimation for Mobile Deployment

    Apr 18, 2026Arun Kumar, Aswathy Baiju, Radu Timofte +1On-Device Inference

    Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

  25. AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution

    Apr 17, 2026Yiwei Zhao, Yi Zheng, Huapeng Su +9Vision Foundation ModelsVision Foundation Model Adaptation

    Carnegie Mellon University · Meta

  26. Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference

    Apr 16, 2026Nenad Banfic, David Fan, Kunal Vaishnavi +5Efficient Neural Network InferenceStreaming ASR

    CoreAI, Microsoft

  27. LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment

    Apr 12, 2026Gregory MagarshakEfficient InferenceLLM Inference Acceleration

  28. FastSHADE: Fast Self-augmented Hierarchical Asymmetric Denoising for Efficient inference on mobile devices

    Apr 11, 2026Nikolay FalaleevImage DenoisingOn-Device Inference

    Fanis London, UK

  29. Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

    Apr 11, 2026Jiesong Chen, Jun You, Zhidan Liu +1Edge InferenceEfficient Neural Network Inference

    Department of Computer Science, City University of Hong Kong, Hong Kong, China · INTR Thrust, System Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

  30. DeFakeQ: Enabling Real-Time Deepfake Detection on Edge Devices via Adaptive Bidirectional Quantization

    Apr 10, 2026Xiangyu Li, Yujing Sun, Yuhang Zheng +2Adaptive QuantizationDeepfake Detection

    Digital Trust Centre, Nanyang Technological University · School of Information Science and Technology, ShanghaiTech University