On-Device Inference

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 175% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 117

All topics

PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs
May 7, 2026Rappy Saha, Jude Haris, Nicolas Bohm Agostini +2Efficient Neural Network Inference FPGA-Based Neural Network Acceleration
ExecuTorch -- A Unified PyTorch Solution to Run AI Models On-Device
May 5, 2026Mergen Nachin, Digant Desai, Sicheng Stephen Jia +36Heterogeneous Computing Edge ML
Meta · Work done while at Meta.
QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs
May 5, 2026Pratik Honavar, Tejpratap GVSLMixed-Precision Quantization Multi-Agent LLM Systems
Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs
May 5, 2026Faraz Kayani, Sarmad Kayani, Asad Ahmed +2Efficient Neural Network Inference Knowledge Distillation
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection
May 4, 2026Joydeep ChandraMixed-Precision Quantization Information Bottleneck
Department of Computer Science & Technology, Tsinghua University, Beijing, China.
AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification
May 2, 2026Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1Efficient Neural Network Inference Plant Disease Classification
AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
At the Edge of the Heart: ULP FPGA-Based CNN for On-Device Cardiac Feature Extraction in Smart Health Sensors for Astronauts
Apr 28, 2026Kazi Mohammad Abidur Rahman, Davis Rakhshan, Philipp Lütke +2Time Series Classification Wearable Health Monitoring
Smart Sensors Group, Hamburg University of Technology, Hamburg, Germany · Distributed Systems, Kiel University, Kiel, Germany · Networked Cyber-Physical Systems, Hamburg University of Technology, Hamburg, Germany
PI-TTA: Physics-Informed Source-Free Test-Time Adaptation for Robust Human Activity Recognition on Mobile Devices
Apr 28, 2026Changyu Li, Lu Wang, Ming Lei +4Test-Time Adaptation Human Activity Recognition
School of Computing and Information Technology, Great Bay University, Dongguan, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · University of Warwick, Coventry, U.K. +2
Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment
Apr 27, 2026Kaijun Zhou, Qiwei Chen, Da Peng +3AI Accelerator Inference Edge Inference
School of Computer Science, Shanghai Jiao Tong University
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM Inference Training Data Curation
University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.
On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcontroller
Apr 24, 2026Jeremy EllisOn-Device Training Tiny ML
High School Robotics Educator, British Columbia, Canada
Bridging the Training-Deployment Gap: Gated Encoding and Multi-Scale Refinement for Efficient Quantization-Aware Image Enhancement
Apr 23, 2026Dat To-Thanh, Nghia Nguyen-Trong, Hoang Vo +2Image Enhancement On-Device Inference
University of Science, VNU-HCM, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, VNU-HCM, Vietnam +2
KD-Judge: A Knowledge-Driven Automated Judge Framework for Functional Fitness Movements on Edge Devices
Apr 21, 2026Shaibal Saha, Fan Li, Yunge Li +3On-Device Inference
Department of Computer Science and Engineering, Oakland University, Rochester Hills, USA · Intelligent Data Management and Analytics LLC, USA
Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
Apr 20, 2026Afsara Benazir, Felix Xiaozhu LinAI Accelerator Inference Energy-Efficient ML
University of Virginia Charlottesville, Virginia, USA
Towards Real-Time ECG and EMG Modeling on $μ$ NPUs
Apr 20, 2026Josh Millar, Ashok Samraj Thangarajan, Soumyajit Chatterjee +1Efficient Neural Network Inference Electrocardiography
Imperial College London London, United Kingdom · Nokia Bell Labs London, United Kingdom · Brave Software & University of Cambridge London, United Kingdom
The First Challenge on Mobile Real-World Image Super-Resolution at NTIRE 2026: Benchmark Results and Method Overview
Apr 19, 2026Jiatong Li, Zheng Chen, Kai Liu +91Image Super-Resolution On-Device Inference
MobileAgeNet: Lightweight Facial Age Estimation for Mobile Deployment
Apr 18, 2026Arun Kumar, Aswathy Baiju, Radu Timofte +1On-Device Inference
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution
Apr 17, 2026Yiwei Zhao, Yi Zheng, Huapeng Su +9Vision Foundation Models Vision Foundation Model Adaptation
Carnegie Mellon University · Meta
Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference
Apr 16, 2026Nenad Banfic, David Fan, Kunal Vaishnavi +5Efficient Neural Network Inference Streaming ASR
CoreAI, Microsoft
Rapidly deploying on-device eye tracking by distilling visual foundation models
Apr 2, 2026Cheng Jiang, Jogendra Kundu, David Colmenares +4Gaze Estimation Vision Foundation Model Adaptation
Meta Reality Labs, USA
An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearing
Mar 3, 2026Riccardo Rota, Kiril Ratmanski, Jozef Coldenhoff +1Speech Enhancement On-Device Inference
Logitech Europe S.A., Switzerland · EPFL ( ´Ecole Polytechnique F´ed´erale de Lausanne), Switzerland
FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge Inference Mixture-of-Experts Inference
SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices
Jan 13, 2026Dongting Hu, Aarush Gupta, Magzhan Gabidolla +12Diffusion Transformer Diffusion Model Distillation
Snap Inc. · University of Melbourne · MBZUAI
An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile Devices
Jan 6, 2026Jeiyoon Park, Daehwan Lee, Changmin Yeo +2Machine Translation On-Device Inference
SOOP
Animal Re-Identification on Microcontrollers
Dec 9, 2025Yubo Chen, Di Zhao, Yun Sing Koh +1Efficient Neural Network Inference Animal Re-Identification
University of Auckland Auckland, New Zealand
Robust Wake-Up Word Detection by Two-stage Multi-resolution Ensembles
Oct 17, 2023Fernando López, Jordi Luque, Carlos Segura +1Keyword Spotting Ensemble Learning
Telefónica I+D, Digital Life Disruption Lab, Spain · Universidad Autónoma de Madrid · Telefónica I+D, Research, Spain

May 7, 2026, cs.ARJ/K move · Enter open · S save

PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs

Rappy Saha, Jude Haris, Nicolas Bohm Agostini, David Kaeli+1

Efficient Neural Network Inference FPGA-Based Neural Network Acceleration Neural Network Quantization On-Device Inference

Open paper AskPDF arXiv

Power-of-two (PoT) quantization significantly reduces the size of deep neural networks (DNNs) and replaces multiplications with bit-shift operations for inference. Prior work has shown that PoT-quantized DNNs can preserve accuracy for tasks such as image classification; however, their performance on resource-constrained edge devices remains insufficiently understood. While general-purpose edge CPUs and GPUs do not provide optimized backends for bit-shift operations, custom hardware accelerators can better exploit PoT quantization by implementing dedicated shift-based processing elements. However, deploying PoT-quantized models on such accelerators is challenging due to limited support in existing inference frameworks. In addition, the impact of different PoT quantization strategies on hardware design, performance, and energy efficiency during full inference has not been systematically explored. To address these challenges, we propose PoTAcc, an open-source end-to-end pipeline for accelerating and evaluating PoT-quantized DNNs on resource-constrained edge devices. PoTAcc enables seamless preparation and deployment of PoT-quantized models via TensorFlow Lite (TFLite) across heterogeneous platforms, including CPU-only systems and hybrid CPU-FPGA systems with custom accelerators. We design shift-based processing element (shift-PE) accelerators for three PoT quantization methods and implement them on two FPGA platforms. We evaluate accuracy, performance, energy efficiency, and resource utilization across a range of models, including CNNs and Transformer-based architectures. Results show that our CPU-accelerator design achieves up to 3.6x speedup and 78% energy reduction compared to CPU-only execution for PoT-quantized DNNs on PYNQ-Z2 and Kria boards. The code will be publicly released at https://github.com/gicLAB/PoTAcc

Momentum

Related topics

Latest papers 117